Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 08:16:30.063918 23429 test_util.cc:268] Using random seed: 946154751
2024-05-02T08:16:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:16:30Z Disabled control of system clock
I20240502 08:16:30.096746 23429 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2vVFYb/build/asan/bin/kudu
/tmp/dist-test-task2vVFYb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.126:43265
--webserver_interface=127.22.225.126
--webserver_port=0
--builtin_ntp_servers=127.22.225.84:40725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.126:43265,127.22.225.125:36189,127.22.225.124:41745
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240502 08:16:30.413693 23437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:30.414049 23437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:30.414191 23437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:30.421123 23437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:16:30.421247 23437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:30.421334 23437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:16:30.421447 23437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:16:30.430593 23437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.84:40725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.22.225.126:43265,127.22.225.125:36189,127.22.225.124:41745
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.126:43265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:30.431998 23437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:30.433919 23437 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:30.445127 23445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:30.445714 23443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:30.446369 23442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:30.447080 23437 server_base.cc:1034] running on GCE node
I20240502 08:16:30.447896 23437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:30.449335 23437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:30.450542 23437 hybrid_clock.cc:648] HybridClock initialized: now 1714637790450478 us; error 42 us; skew 500 ppm
I20240502 08:16:30.450855 23437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:30.458422 23437 webserver.cc:460] Webserver started at http://127.22.225.126:34035/ using document root <none> and password file <none>
I20240502 08:16:30.459175 23437 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:30.459313 23437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:30.459604 23437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:30.462316 23437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data/instance:
uuid: "bcc8a323e6b9429eaaf4e0c1698ad077"
format_stamp: "Formatted at 2024-05-02 08:16:30 on dist-test-slave-m44h"
I20240502 08:16:30.463029 23437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/wal/instance:
uuid: "bcc8a323e6b9429eaaf4e0c1698ad077"
format_stamp: "Formatted at 2024-05-02 08:16:30 on dist-test-slave-m44h"
I20240502 08:16:30.468102 23437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:16:30.471334 23451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:30.472491 23437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:30.472635 23437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data,/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/wal
uuid: "bcc8a323e6b9429eaaf4e0c1698ad077"
format_stamp: "Formatted at 2024-05-02 08:16:30 on dist-test-slave-m44h"
I20240502 08:16:30.472782 23437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:30.494277 23437 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:30.495214 23437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:30.495498 23437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:30.512974 23437 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.126:43265
I20240502 08:16:30.513000 23502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.126:43265 every 8 connection(s)
I20240502 08:16:30.514415 23437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data/info.pb
I20240502 08:16:30.518199 23503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:30.522310 23429 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2vVFYb/build/asan/bin/kudu as pid 23437
I20240502 08:16:30.522603 23429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/wal/instance
I20240502 08:16:30.525909 23429 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2vVFYb/build/asan/bin/kudu
/tmp/dist-test-task2vVFYb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.125:36189
--webserver_interface=127.22.225.125
--webserver_port=0
--builtin_ntp_servers=127.22.225.84:40725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.126:43265,127.22.225.125:36189,127.22.225.124:41745
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240502 08:16:30.525190 23503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:16:30.541144 23503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } has no permanent_uuid. Determining permanent_uuid...
W20240502 08:16:30.542194 23452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.225.125:36189: connect: Connection refused (error 111)
W20240502 08:16:30.543660 23503 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.225.125:36189: Network error: Client connection negotiation failed: client connection to 127.22.225.125:36189: connect: Connection refused (error 111)
I20240502 08:16:30.593014 23503 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } attempt: 1
W20240502 08:16:30.595005 23503 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.225.125:36189: Network error: Client connection negotiation failed: client connection to 127.22.225.125:36189: connect: Connection refused (error 111)
I20240502 08:16:30.663244 23503 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } attempt: 2
W20240502 08:16:30.665242 23503 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.225.125:36189: Network error: Client connection negotiation failed: client connection to 127.22.225.125:36189: connect: Connection refused (error 111)
I20240502 08:16:30.756580 23503 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } attempt: 3
W20240502 08:16:30.758332 23503 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.225.125:36189: Network error: Client connection negotiation failed: client connection to 127.22.225.125:36189: connect: Connection refused (error 111)
W20240502 08:16:30.847188 23507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:30.847610 23507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:30.847718 23507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:30.855124 23507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:16:30.855244 23507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:30.855283 23507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:16:30.855329 23507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:16:30.863588 23507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.84:40725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.22.225.126:43265,127.22.225.125:36189,127.22.225.124:41745
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.125:36189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.225.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:30.864938 23507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:30.866469 23507 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:30.875988 23513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:30.876852 23514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:30.877455 23516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:30.878165 23507 server_base.cc:1034] running on GCE node
I20240502 08:16:30.878821 23507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:30.879988 23507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:30.881284 23507 hybrid_clock.cc:648] HybridClock initialized: now 1714637790881227 us; error 33 us; skew 500 ppm
I20240502 08:16:30.881685 23507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:30.883949 23507 webserver.cc:460] Webserver started at http://127.22.225.125:41449/ using document root <none> and password file <none>
I20240502 08:16:30.884681 23507 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:30.884822 23507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:30.885140 23507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:30.888023 23507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/data/instance:
uuid: "84f3c07a31414c4f8fb61c44dc9f16e1"
format_stamp: "Formatted at 2024-05-02 08:16:30 on dist-test-slave-m44h"
I20240502 08:16:30.888888 23507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/wal/instance:
uuid: "84f3c07a31414c4f8fb61c44dc9f16e1"
format_stamp: "Formatted at 2024-05-02 08:16:30 on dist-test-slave-m44h"
I20240502 08:16:30.894593 23507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240502 08:16:30.898175 23522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:30.899516 23507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:16:30.899717 23507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/data,/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/wal
uuid: "84f3c07a31414c4f8fb61c44dc9f16e1"
format_stamp: "Formatted at 2024-05-02 08:16:30 on dist-test-slave-m44h"
I20240502 08:16:30.899921 23507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:30.901551 23503 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } attempt: 4
W20240502 08:16:30.903497 23503 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.225.125:36189: Network error: Client connection negotiation failed: client connection to 127.22.225.125:36189: connect: Connection refused (error 111)
I20240502 08:16:30.920513 23507 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:30.921495 23507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:30.921855 23507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:30.940897 23507 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.125:36189
I20240502 08:16:30.940936 23573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.125:36189 every 8 connection(s)
I20240502 08:16:30.942335 23507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/data/info.pb
I20240502 08:16:30.944371 23429 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2vVFYb/build/asan/bin/kudu as pid 23507
I20240502 08:16:30.944558 23429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/wal/instance
I20240502 08:16:30.945631 23429 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2vVFYb/build/asan/bin/kudu
/tmp/dist-test-task2vVFYb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.124:41745
--webserver_interface=127.22.225.124
--webserver_port=0
--builtin_ntp_servers=127.22.225.84:40725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.126:43265,127.22.225.125:36189,127.22.225.124:41745
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240502 08:16:30.946611 23574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:30.952600 23574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:16:30.965545 23574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:16:30.973832 23574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } has no permanent_uuid. Determining permanent_uuid...
W20240502 08:16:30.974715 23525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.225.124:41745: connect: Connection refused (error 111)
W20240502 08:16:30.976009 23574 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.225.124:41745: Network error: Client connection negotiation failed: client connection to 127.22.225.124:41745: connect: Connection refused (error 111)
I20240502 08:16:31.025311 23574 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } attempt: 1
W20240502 08:16:31.027123 23574 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.225.124:41745: Network error: Client connection negotiation failed: client connection to 127.22.225.124:41745: connect: Connection refused (error 111)
I20240502 08:16:31.095469 23574 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } attempt: 2
W20240502 08:16:31.097468 23574 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.225.124:41745: Network error: Client connection negotiation failed: client connection to 127.22.225.124:41745: connect: Connection refused (error 111)
I20240502 08:16:31.188746 23574 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } attempt: 3
W20240502 08:16:31.190593 23574 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.225.124:41745: Network error: Client connection negotiation failed: client connection to 127.22.225.124:41745: connect: Connection refused (error 111)
I20240502 08:16:31.202742 23503 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } attempt: 5
I20240502 08:16:31.209359 23503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } has no permanent_uuid. Determining permanent_uuid...
W20240502 08:16:31.211040 23503 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.225.124:41745: Network error: Client connection negotiation failed: client connection to 127.22.225.124:41745: connect: Connection refused (error 111)
W20240502 08:16:31.260867 23576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:31.261124 23576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:31.261209 23576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 08:16:31.262307 23503 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } attempt: 1
W20240502 08:16:31.263991 23503 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.225.124:41745: Network error: Client connection negotiation failed: client connection to 127.22.225.124:41745: connect: Connection refused (error 111)
W20240502 08:16:31.268361 23576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:16:31.268476 23576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:31.268532 23576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:16:31.268575 23576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:16:31.276604 23576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.84:40725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.22.225.126:43265,127.22.225.125:36189,127.22.225.124:41745
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.124:41745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.225.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:31.277855 23576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:31.279459 23576 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:31.290205 23587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:31.290696 23585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:31.291280 23584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:31.292330 23576 server_base.cc:1034] running on GCE node
I20240502 08:16:31.293066 23576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:31.294534 23576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:31.295753 23576 hybrid_clock.cc:648] HybridClock initialized: now 1714637791295684 us; error 54 us; skew 500 ppm
I20240502 08:16:31.296061 23576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:31.298336 23576 webserver.cc:460] Webserver started at http://127.22.225.124:34855/ using document root <none> and password file <none>
I20240502 08:16:31.299041 23576 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:31.299162 23576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:31.299438 23576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:31.302047 23576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-2/data/instance:
uuid: "f87df6a45ad94428a9aa6b2796a0fd25"
format_stamp: "Formatted at 2024-05-02 08:16:31 on dist-test-slave-m44h"
I20240502 08:16:31.302762 23576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-2/wal/instance:
uuid: "f87df6a45ad94428a9aa6b2796a0fd25"
format_stamp: "Formatted at 2024-05-02 08:16:31 on dist-test-slave-m44h"
I20240502 08:16:31.308233 23576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20240502 08:16:31.311499 23593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:31.312685 23576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:31.312841 23576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-2/data,/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-2/wal
uuid: "f87df6a45ad94428a9aa6b2796a0fd25"
format_stamp: "Formatted at 2024-05-02 08:16:31 on dist-test-slave-m44h"
I20240502 08:16:31.313012 23576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:31.332206 23503 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } attempt: 2
W20240502 08:16:31.333904 23503 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.225.124:41745: Network error: Client connection negotiation failed: client connection to 127.22.225.124:41745: connect: Connection refused (error 111)
I20240502 08:16:31.333842 23574 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } attempt: 4
I20240502 08:16:31.334148 23576 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:31.335228 23576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
W20240502 08:16:31.335387 23574 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.22.225.124:41745: Network error: Client connection negotiation failed: client connection to 127.22.225.124:41745: connect: Connection refused (error 111)
I20240502 08:16:31.335529 23576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:31.352830 23576 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.124:41745
I20240502 08:16:31.352866 23644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.124:41745 every 8 connection(s)
I20240502 08:16:31.354233 23576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-2/data/info.pb
I20240502 08:16:31.357839 23645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:31.361828 23429 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2vVFYb/build/asan/bin/kudu as pid 23576
I20240502 08:16:31.362030 23429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-2/wal/instance
I20240502 08:16:31.364949 23645 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:16:31.392359 23645 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:16:31.399137 23645 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } has no permanent_uuid. Determining permanent_uuid...
I20240502 08:16:31.408607 23645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Bootstrap starting.
I20240502 08:16:31.411515 23645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:31.412640 23645 log.cc:824] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:31.414997 23645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: No bootstrap required, opened a new log
I20240502 08:16:31.420483 23645 raft_consensus.cc:348] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:31.420755 23645 raft_consensus.cc:374] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:31.420869 23645 raft_consensus.cc:729] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f87df6a45ad94428a9aa6b2796a0fd25, State: Initialized, Role: FOLLOWER
I20240502 08:16:31.421547 23645 consensus_queue.cc:260] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:31.422421 23652 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } } }
I20240502 08:16:31.422703 23652 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:31.423395 23645 sys_catalog.cc:564] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [sys.catalog]: configured and running, proceeding with master startup.
W20240502 08:16:31.434828 23663 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 08:16:31.435006 23663 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 08:16:31.429805 23576 master_runner.cc:186] Error getting master registration for 127.22.225.126:43265: OK, instance_id { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" instance_seqno: 1714637790507995 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240502 08:16:31.436177 23576 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240502 08:16:31.440105 23503 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } attempt: 3
I20240502 08:16:31.451681 23503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077: Bootstrap starting.
I20240502 08:16:31.455762 23503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:31.457108 23503 log.cc:824] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:31.460692 23503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077: No bootstrap required, opened a new log
I20240502 08:16:31.469975 23503 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:31.470371 23503 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:31.470520 23503 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc8a323e6b9429eaaf4e0c1698ad077, State: Initialized, Role: FOLLOWER
I20240502 08:16:31.471272 23503 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:31.472493 23666 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } } }
I20240502 08:16:31.472906 23666 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:31.473745 23503 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [sys.catalog]: configured and running, proceeding with master startup.
W20240502 08:16:31.486294 23677 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 08:16:31.486536 23677 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 08:16:31.482731 23437 master_runner.cc:186] Error getting master registration for 127.22.225.125:36189: OK, instance_id { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" instance_seqno: 1714637790935910 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240502 08:16:31.490100 23437 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240502 08:16:31.498322 23457 tablet.cc:2369] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:16:31.587616 23599 tablet.cc:2369] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:31.634616 23574 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } attempt: 5
I20240502 08:16:31.645783 23574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1: Bootstrap starting.
I20240502 08:16:31.649883 23574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:31.651115 23574 log.cc:824] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:31.653781 23574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1: No bootstrap required, opened a new log
I20240502 08:16:31.662451 23574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:31.662791 23574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:31.662916 23574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f3c07a31414c4f8fb61c44dc9f16e1, State: Initialized, Role: FOLLOWER
I20240502 08:16:31.663597 23574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:31.664682 23680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } } }
I20240502 08:16:31.665096 23680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:31.665822 23574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:31.669766 23652 raft_consensus.cc:482] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:31.669984 23652 raft_consensus.cc:504] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:31.671545 23652 leader_election.cc:288] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcc8a323e6b9429eaaf4e0c1698ad077 (127.22.225.126:43265), 84f3c07a31414c4f8fb61c44dc9f16e1 (127.22.225.125:36189)
I20240502 08:16:31.672489 23478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" is_pre_election: true
I20240502 08:16:31.673012 23478 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f87df6a45ad94428a9aa6b2796a0fd25 in term 0.
I20240502 08:16:31.674067 23595 leader_election.cc:302] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcc8a323e6b9429eaaf4e0c1698ad077, f87df6a45ad94428a9aa6b2796a0fd25; no voters:
W20240502 08:16:31.674628 23528 tablet.cc:2369] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:31.675993 23549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" is_pre_election: true
I20240502 08:16:31.676417 23549 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f87df6a45ad94428a9aa6b2796a0fd25 in term 0.
I20240502 08:16:31.677196 23652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:31.677325 23652 raft_consensus.cc:482] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:31.677415 23652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:31.680279 23652 raft_consensus.cc:504] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:31.681315 23478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077"
I20240502 08:16:31.681517 23478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:31.684901 23478 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f87df6a45ad94428a9aa6b2796a0fd25 in term 1.
I20240502 08:16:31.685397 23595 leader_election.cc:302] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcc8a323e6b9429eaaf4e0c1698ad077, f87df6a45ad94428a9aa6b2796a0fd25; no voters:
I20240502 08:16:31.686154 23684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:31.687062 23549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1"
I20240502 08:16:31.687367 23549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:31.689725 23684 raft_consensus.cc:686] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [term 1 LEADER]: Becoming Leader. State: Replica: f87df6a45ad94428a9aa6b2796a0fd25, State: Running, Role: LEADER
I20240502 08:16:31.690596 23549 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f87df6a45ad94428a9aa6b2796a0fd25 in term 1.
I20240502 08:16:31.690372 23684 consensus_queue.cc:237] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:31.691314 23652 leader_election.cc:288] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [CANDIDATE]: Term 1 election: Requested vote from peers bcc8a323e6b9429eaaf4e0c1698ad077 (127.22.225.126:43265), 84f3c07a31414c4f8fb61c44dc9f16e1 (127.22.225.125:36189)
I20240502 08:16:31.704102 23685 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f87df6a45ad94428a9aa6b2796a0fd25. Latest consensus state: current_term: 1 leader_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } } }
I20240502 08:16:31.704505 23685 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:31.707142 23691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:16:31.710667 23691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:16:31.724210 23549 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 1 FOLLOWER]: Refusing update from remote peer f87df6a45ad94428a9aa6b2796a0fd25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:31.725453 23684 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:31.734601 23478 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [term 1 FOLLOWER]: Refusing update from remote peer f87df6a45ad94428a9aa6b2796a0fd25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:31.735491 23684 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:31.759306 23666 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f87df6a45ad94428a9aa6b2796a0fd25. Latest consensus state: current_term: 1 leader_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } } }
I20240502 08:16:31.759682 23666 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:31.761288 23680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f87df6a45ad94428a9aa6b2796a0fd25. Latest consensus state: current_term: 1 leader_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } } }
I20240502 08:16:31.761596 23680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:31.762593 23694 mvcc.cc:205] Tried to move back new op lower bound from 7023156394887118848 to 7023156394779136000. Current Snapshot: MvccSnapshot[applied={T|T < 7023156394887118848}]
I20240502 08:16:31.774998 23697 mvcc.cc:205] Tried to move back new op lower bound from 7023156394887118848 to 7023156394779136000. Current Snapshot: MvccSnapshot[applied={T|T < 7023156394887118848}]
I20240502 08:16:31.781908 23698 mvcc.cc:205] Tried to move back new op lower bound from 7023156394887118848 to 7023156394779136000. Current Snapshot: MvccSnapshot[applied={T|T < 7023156394887118848}]
I20240502 08:16:31.790972 23680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } } }
I20240502 08:16:31.791311 23680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:31.804016 23666 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } } }
I20240502 08:16:31.804309 23666 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:31.805516 23685 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } } }
I20240502 08:16:31.805799 23685 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:31.808061 23685 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } } }
I20240502 08:16:31.808290 23685 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:31.811321 23691 catalog_manager.cc:1347] Generated new cluster ID: d0e52a60b5c54ea9a5f0ea385bd7e9c9
I20240502 08:16:31.811426 23691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20240502 08:16:31.812634 23696 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 08:16:31.812846 23696 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 08:16:31.834851 23691 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:16:31.836534 23691 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:16:31.859376 23691 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Generated new TSK 0
I20240502 08:16:31.860396 23691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:16:31.876314 23429 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2vVFYb/build/asan/bin/kudu
/tmp/dist-test-task2vVFYb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.65:0
--local_ip_for_outbound_sockets=127.22.225.65
--webserver_interface=127.22.225.65
--webserver_port=0
--tserver_master_addrs=127.22.225.126:43265,127.22.225.125:36189,127.22.225.124:41745
--builtin_ntp_servers=127.22.225.84:40725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20240502 08:16:32.262739 23705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:32.263005 23705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:32.263127 23705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:32.270570 23705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:32.270795 23705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.65
I20240502 08:16:32.279659 23705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.84:40725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.225.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.22.225.126:43265,127.22.225.125:36189,127.22.225.124:41745
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.65
--log_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:32.280946 23705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:32.282765 23705 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:32.294615 23711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:32.295150 23713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:32.295878 23710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:32.296471 23705 server_base.cc:1034] running on GCE node
I20240502 08:16:32.297106 23705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:32.298079 23705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:32.299392 23705 hybrid_clock.cc:648] HybridClock initialized: now 1714637792299258 us; error 40 us; skew 500 ppm
I20240502 08:16:32.299760 23705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:32.301995 23705 webserver.cc:460] Webserver started at http://127.22.225.65:40507/ using document root <none> and password file <none>
I20240502 08:16:32.302721 23705 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:32.302857 23705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:32.303149 23705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:32.305950 23705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/data/instance:
uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
format_stamp: "Formatted at 2024-05-02 08:16:32 on dist-test-slave-m44h"
I20240502 08:16:32.306654 23705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/instance:
uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
format_stamp: "Formatted at 2024-05-02 08:16:32 on dist-test-slave-m44h"
I20240502 08:16:32.311875 23705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 08:16:32.315299 23719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:32.316581 23705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 08:16:32.316740 23705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/data,/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal
uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
format_stamp: "Formatted at 2024-05-02 08:16:32 on dist-test-slave-m44h"
I20240502 08:16:32.316920 23705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:32.358912 23724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f87df6a45ad94428a9aa6b2796a0fd25)
I20240502 08:16:32.359149 23724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:32.360652 23724 leader_election.cc:288] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcc8a323e6b9429eaaf4e0c1698ad077 (127.22.225.126:43265), f87df6a45ad94428a9aa6b2796a0fd25 (127.22.225.124:41745)
I20240502 08:16:32.361225 23478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" is_pre_election: true
I20240502 08:16:32.361294 23620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" is_pre_election: true
I20240502 08:16:32.361992 23525 leader_election.cc:302] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84f3c07a31414c4f8fb61c44dc9f16e1; no voters: bcc8a323e6b9429eaaf4e0c1698ad077, f87df6a45ad94428a9aa6b2796a0fd25
I20240502 08:16:32.362386 23724 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:32.373600 23705 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:32.374480 23705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:32.374785 23705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:32.376006 23705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:32.378343 23705 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:32.378424 23705 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:32.378572 23705 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:32.378630 23705 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:32.410179 23705 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.65:38709
I20240502 08:16:32.410212 23832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.65:38709 every 8 connection(s)
I20240502 08:16:32.411397 23705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/data/info.pb
I20240502 08:16:32.421695 23429 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2vVFYb/build/asan/bin/kudu as pid 23705
I20240502 08:16:32.421931 23429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/instance
I20240502 08:16:32.437510 23429 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2vVFYb/build/asan/bin/kudu
/tmp/dist-test-task2vVFYb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.66:0
--local_ip_for_outbound_sockets=127.22.225.66
--webserver_interface=127.22.225.66
--webserver_port=0
--tserver_master_addrs=127.22.225.126:43265,127.22.225.125:36189,127.22.225.124:41745
--builtin_ntp_servers=127.22.225.84:40725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240502 08:16:32.443956 23834 heartbeater.cc:347] Connected to a master server at 127.22.225.126:43265
I20240502 08:16:32.444355 23834 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:32.444384 23835 heartbeater.cc:347] Connected to a master server at 127.22.225.125:36189
I20240502 08:16:32.444572 23835 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:32.445079 23833 heartbeater.cc:347] Connected to a master server at 127.22.225.124:41745
I20240502 08:16:32.445216 23833 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:32.445715 23835 heartbeater.cc:511] Master 127.22.225.125:36189 requested a full tablet report, sending...
I20240502 08:16:32.445842 23833 heartbeater.cc:511] Master 127.22.225.124:41745 requested a full tablet report, sending...
I20240502 08:16:32.446096 23834 heartbeater.cc:511] Master 127.22.225.126:43265 requested a full tablet report, sending...
I20240502 08:16:32.447866 23610 ts_manager.cc:194] Registered new tserver with Master: a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:32.448099 23468 ts_manager.cc:194] Registered new tserver with Master: a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:32.448246 23539 ts_manager.cc:194] Registered new tserver with Master: a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:32.449928 23610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.225.65:44121
I20240502 08:16:32.489171 23677 catalog_manager.cc:1259] Loaded cluster ID: d0e52a60b5c54ea9a5f0ea385bd7e9c9
I20240502 08:16:32.489274 23677 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077: loading cluster ID for follower catalog manager: success
I20240502 08:16:32.492568 23677 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077: acquiring CA information for follower catalog manager: success
I20240502 08:16:32.494935 23677 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20240502 08:16:32.773272 23843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:32.773530 23843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:32.773641 23843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:32.780697 23843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:32.780894 23843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.66
I20240502 08:16:32.789535 23843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.84:40725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.225.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.22.225.126:43265,127.22.225.125:36189,127.22.225.124:41745
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.66
--log_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:32.790861 23843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:32.792496 23843 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:32.804605 23848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:32.805450 23851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:32.805971 23849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:32.807170 23843 server_base.cc:1034] running on GCE node
I20240502 08:16:32.807845 23843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:32.808794 23843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:32.810000 23843 hybrid_clock.cc:648] HybridClock initialized: now 1714637792809930 us; error 45 us; skew 500 ppm
I20240502 08:16:32.810281 23843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:32.812494 23843 webserver.cc:460] Webserver started at http://127.22.225.66:41203/ using document root <none> and password file <none>
I20240502 08:16:32.813257 23843 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:32.813405 23843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:32.813817 23843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:32.815351 23696 catalog_manager.cc:1259] Loaded cluster ID: d0e52a60b5c54ea9a5f0ea385bd7e9c9
I20240502 08:16:32.815449 23696 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1: loading cluster ID for follower catalog manager: success
I20240502 08:16:32.817435 23843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/data/instance:
uuid: "5b89a1f0485a44b7ac1710e2369a809b"
format_stamp: "Formatted at 2024-05-02 08:16:32 on dist-test-slave-m44h"
I20240502 08:16:32.818397 23843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/instance:
uuid: "5b89a1f0485a44b7ac1710e2369a809b"
format_stamp: "Formatted at 2024-05-02 08:16:32 on dist-test-slave-m44h"
I20240502 08:16:32.819010 23696 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1: acquiring CA information for follower catalog manager: success
I20240502 08:16:32.821374 23696 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 08:16:32.824458 23843 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240502 08:16:32.827915 23857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:32.829248 23843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:32.829419 23843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/data,/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal
uuid: "5b89a1f0485a44b7ac1710e2369a809b"
format_stamp: "Formatted at 2024-05-02 08:16:32 on dist-test-slave-m44h"
I20240502 08:16:32.829690 23843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:32.875283 23843 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:32.876098 23843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:32.876330 23843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:32.877189 23843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:32.878695 23843 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:32.878762 23843 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:32.878839 23843 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:32.878877 23843 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:32.910178 23843 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.66:44687
I20240502 08:16:32.910218 23969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.66:44687 every 8 connection(s)
I20240502 08:16:32.911422 23843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/data/info.pb
I20240502 08:16:32.921342 23429 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2vVFYb/build/asan/bin/kudu as pid 23843
I20240502 08:16:32.921602 23429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/instance
I20240502 08:16:32.926541 23429 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2vVFYb/build/asan/bin/kudu
/tmp/dist-test-task2vVFYb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.225.67:0
--local_ip_for_outbound_sockets=127.22.225.67
--webserver_interface=127.22.225.67
--webserver_port=0
--tserver_master_addrs=127.22.225.126:43265,127.22.225.125:36189,127.22.225.124:41745
--builtin_ntp_servers=127.22.225.84:40725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240502 08:16:32.928613 23971 heartbeater.cc:347] Connected to a master server at 127.22.225.126:43265
I20240502 08:16:32.928925 23971 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:32.929821 23971 heartbeater.cc:511] Master 127.22.225.126:43265 requested a full tablet report, sending...
I20240502 08:16:32.930574 23972 heartbeater.cc:347] Connected to a master server at 127.22.225.125:36189
I20240502 08:16:32.930722 23972 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:32.931288 23972 heartbeater.cc:511] Master 127.22.225.125:36189 requested a full tablet report, sending...
I20240502 08:16:32.931677 23468 ts_manager.cc:194] Registered new tserver with Master: 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:32.932356 23539 ts_manager.cc:194] Registered new tserver with Master: 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:32.933250 23970 heartbeater.cc:347] Connected to a master server at 127.22.225.124:41745
I20240502 08:16:32.933372 23970 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:32.933931 23970 heartbeater.cc:511] Master 127.22.225.124:41745 requested a full tablet report, sending...
I20240502 08:16:32.934743 23610 ts_manager.cc:194] Registered new tserver with Master: 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:32.935446 23610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.225.66:39077
I20240502 08:16:32.952818 23833 heartbeater.cc:503] Master 127.22.225.124:41745 was elected leader, sending a full tablet report...
I20240502 08:16:32.981410 23978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f87df6a45ad94428a9aa6b2796a0fd25)
I20240502 08:16:32.981623 23978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:32.982290 23978 leader_election.cc:288] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcc8a323e6b9429eaaf4e0c1698ad077 (127.22.225.126:43265), f87df6a45ad94428a9aa6b2796a0fd25 (127.22.225.124:41745)
I20240502 08:16:32.982796 23478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" is_pre_election: true
I20240502 08:16:32.982957 23620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" is_pre_election: true
I20240502 08:16:32.983553 23525 leader_election.cc:302] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84f3c07a31414c4f8fb61c44dc9f16e1; no voters: bcc8a323e6b9429eaaf4e0c1698ad077, f87df6a45ad94428a9aa6b2796a0fd25
I20240502 08:16:32.983892 23978 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:16:33.258543 23977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:33.258849 23977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:33.258999 23977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:33.266664 23977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:33.266903 23977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.225.67
I20240502 08:16:33.276095 23977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.84:40725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.225.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.225.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.22.225.126:43265,127.22.225.125:36189,127.22.225.124:41745
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.225.67
--log_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:33.277451 23977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:33.279152 23977 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:33.291438 23983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:33.292012 23984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:33.293627 23986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:33.293601 23977 server_base.cc:1034] running on GCE node
I20240502 08:16:33.294384 23977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:33.295325 23977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:33.296541 23977 hybrid_clock.cc:648] HybridClock initialized: now 1714637793296472 us; error 51 us; skew 500 ppm
I20240502 08:16:33.296877 23977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:33.299086 23977 webserver.cc:460] Webserver started at http://127.22.225.67:37401/ using document root <none> and password file <none>
I20240502 08:16:33.299775 23977 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:33.299872 23977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:33.300117 23977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:33.302738 23977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/data/instance:
uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
format_stamp: "Formatted at 2024-05-02 08:16:33 on dist-test-slave-m44h"
I20240502 08:16:33.303442 23977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/instance:
uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
format_stamp: "Formatted at 2024-05-02 08:16:33 on dist-test-slave-m44h"
I20240502 08:16:33.308512 23977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240502 08:16:33.312131 23992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:33.313460 23977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:33.313640 23977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/data,/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal
uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
format_stamp: "Formatted at 2024-05-02 08:16:33 on dist-test-slave-m44h"
I20240502 08:16:33.313822 23977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:33.335274 23977 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:33.336103 23977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:33.336411 23977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:33.337316 23977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:33.338793 23977 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:33.338860 23977 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:33.338951 23977 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:33.338990 23977 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:33.371315 23977 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.67:44163
I20240502 08:16:33.371351 24104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.67:44163 every 8 connection(s)
I20240502 08:16:33.372591 23977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/data/info.pb
I20240502 08:16:33.377383 23429 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2vVFYb/build/asan/bin/kudu as pid 23977
I20240502 08:16:33.377615 23429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/instance
I20240502 08:16:33.389624 24105 heartbeater.cc:347] Connected to a master server at 127.22.225.124:41745
I20240502 08:16:33.389665 24106 heartbeater.cc:347] Connected to a master server at 127.22.225.126:43265
I20240502 08:16:33.389994 24105 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:33.390137 24106 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:33.391194 24105 heartbeater.cc:511] Master 127.22.225.124:41745 requested a full tablet report, sending...
I20240502 08:16:33.391498 24106 heartbeater.cc:511] Master 127.22.225.126:43265 requested a full tablet report, sending...
I20240502 08:16:33.391638 24107 heartbeater.cc:347] Connected to a master server at 127.22.225.125:36189
I20240502 08:16:33.391781 24107 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:33.392308 24107 heartbeater.cc:511] Master 127.22.225.125:36189 requested a full tablet report, sending...
I20240502 08:16:33.392814 23610 ts_manager.cc:194] Registered new tserver with Master: 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:33.392863 23468 ts_manager.cc:194] Registered new tserver with Master: 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:33.393414 23539 ts_manager.cc:194] Registered new tserver with Master: 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:33.393918 23610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.225.67:46497
I20240502 08:16:33.406904 23429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 08:16:33.438364 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_82b1090ce1e84172878507ac4ab9d03f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:33.438822 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c586068a666245c896461672c2d612c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:33.438653 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_cedf5da841da47cd9be7fdabeca2da3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:33.439193 23970 heartbeater.cc:503] Master 127.22.225.124:41745 was elected leader, sending a full tablet report...
I20240502 08:16:33.439746 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_264f977f6ba34150ab42f907d7dc04b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:33.440361 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82b1090ce1e84172878507ac4ab9d03f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:33.440696 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_883c294a544b4cfea491a6ec6415d3bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:33.441340 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_883c294a544b4cfea491a6ec6415d3bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:33.442142 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c586068a666245c896461672c2d612c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:33.442929 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5bced16157c34d90afba862893547f03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:33.443539 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_95cf177d064b4d6f928867a30f628edf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:33.443748 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bced16157c34d90afba862893547f03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:33.444128 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95cf177d064b4d6f928867a30f628edf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:33.446045 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cedf5da841da47cd9be7fdabeca2da3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:33.440367 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_264f977f6ba34150ab42f907d7dc04b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:33.447432 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a9a17e8d192644c4bed536a377864f92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:33.448118 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9a17e8d192644c4bed536a377864f92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:33.438311 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_785c9ed197614bc78c888789b840a64c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:33.449486 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_785c9ed197614bc78c888789b840a64c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:33.449436 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c1a1a9803653466db65de16e28228059"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:33.450197 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1a1a9803653466db65de16e28228059 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:33.481217 23429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2vVFYb/build/asan/bin/kudu pid 23437
I20240502 08:16:33.482436 23498 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:33.489549 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_883c294a544b4cfea491a6ec6415d3bd" } new_table_name: "default.Table_ffa5fe5beacd4345b72e20da3bb7cb76" modify_external_catalogs: true
I20240502 08:16:33.510921 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5bced16157c34d90afba862893547f03" } new_table_name: "default.Table_bc71bb1f437e4476b7f3a26061a52db6" modify_external_catalogs: true
I20240502 08:16:33.511041 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cedf5da841da47cd9be7fdabeca2da3d" } modify_external_catalogs: true
I20240502 08:16:33.511276 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cedf5da841da47cd9be7fdabeca2da3d" } modify_external_catalogs: true
I20240502 08:16:33.512550 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_95cf177d064b4d6f928867a30f628edf" } new_table_name: "default.Table_938322b142834d4eb5ff155d2425e348" modify_external_catalogs: true
W20240502 08:16:33.514008 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c228645fe0b4bc89643178fff7b28a3 consensus state has no leader
I20240502 08:16:33.515216 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_82b1090ce1e84172878507ac4ab9d03f" } new_table_name: "default.Table_ff549ea350ef46a59aefb1e305429d98" modify_external_catalogs: true
I20240502 08:16:33.515928 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a9a17e8d192644c4bed536a377864f92" } modify_external_catalogs: true
I20240502 08:16:33.516131 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a9a17e8d192644c4bed536a377864f92" } modify_external_catalogs: true
I20240502 08:16:33.516371 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c1a1a9803653466db65de16e28228059" } modify_external_catalogs: true
I20240502 08:16:33.516557 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c1a1a9803653466db65de16e28228059" } modify_external_catalogs: true
I20240502 08:16:33.517292 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c586068a666245c896461672c2d612c3" } new_table_name: "default.Table_116d421c142443cebb54af5cca25ee16" modify_external_catalogs: true
I20240502 08:16:33.517652 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_785c9ed197614bc78c888789b840a64c" } modify_external_catalogs: true
I20240502 08:16:33.517833 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_785c9ed197614bc78c888789b840a64c" } modify_external_catalogs: true
I20240502 08:16:33.518901 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_264f977f6ba34150ab42f907d7dc04b6" } modify_external_catalogs: true
I20240502 08:16:33.519068 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_264f977f6ba34150ab42f907d7dc04b6" } modify_external_catalogs: true
W20240502 08:16:33.544247 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet abd595972fc144328c279a84ec7eeaa9 has no consensus state
I20240502 08:16:33.544790 23609 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 1486f7ab2bd1491a8ec54be804a9a68a
W20240502 08:16:33.545280 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67bc4942db6140a3b7984318a78f7ad3 has no consensus state
W20240502 08:16:33.545686 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0cf60a6da6f94525b02bdbb9e53192db has no consensus state
W20240502 08:16:33.545853 23595 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 6c228645fe0b4bc89643178fff7b28a3 (table default.Table_ffa5fe5beacd4345b72e20da3bb7cb76 [id=b24212eeb7784959b5fbcd68bcafa0d7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6c228645fe0b4bc89643178fff7b28a3 consensus state has no leader
I20240502 08:16:33.546073 23603 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 4c9d9306385243da85e65b2569b09ac2
I20240502 08:16:33.546715 23602 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet a05cab1e899b402787ea1f9f7bad57be
W20240502 08:16:33.547613 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 534e3bceaa5c4c77abaf288ea08287ff has no consensus state
I20240502 08:16:33.547894 23604 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 88009b815f624611aaea46a33e073468
I20240502 08:16:33.548719 23605 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 6a4e2563708e45d68ba7fe840bcff60b
I20240502 08:16:33.555563 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 6c228645fe0b4bc89643178fff7b28a3 (DEFAULT_TABLE table=default.Table_ffa5fe5beacd4345b72e20da3bb7cb76 [id=b24212eeb7784959b5fbcd68bcafa0d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.555879 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 6c228645fe0b4bc89643178fff7b28a3 (DEFAULT_TABLE table=default.Table_ffa5fe5beacd4345b72e20da3bb7cb76 [id=b24212eeb7784959b5fbcd68bcafa0d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.557235 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c228645fe0b4bc89643178fff7b28a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.557327 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c228645fe0b4bc89643178fff7b28a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.562458 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 534e3bceaa5c4c77abaf288ea08287ff (DEFAULT_TABLE table=default.Table_116d421c142443cebb54af5cca25ee16 [id=467a7f6db73641ca85a1898b251938fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.563066 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 534e3bceaa5c4c77abaf288ea08287ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.563755 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf60a6da6f94525b02bdbb9e53192db (DEFAULT_TABLE table=default.Table_ff549ea350ef46a59aefb1e305429d98 [id=40060891c6d44d559c37891803b42478]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.564368 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf60a6da6f94525b02bdbb9e53192db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.565994 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 67bc4942db6140a3b7984318a78f7ad3 (DEFAULT_TABLE table=default.Table_938322b142834d4eb5ff155d2425e348 [id=7338537cdd4249359d2af60930089412]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.566483 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67bc4942db6140a3b7984318a78f7ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.567736 23764 tablet_service.cc:1467] Processing CreateTablet for tablet abd595972fc144328c279a84ec7eeaa9 (DEFAULT_TABLE table=default.Table_bc71bb1f437e4476b7f3a26061a52db6 [id=1e590d80e3e34e1aac8a0c7908a9d813]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.568399 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abd595972fc144328c279a84ec7eeaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.569258 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 6c228645fe0b4bc89643178fff7b28a3 (DEFAULT_TABLE table=default.Table_ffa5fe5beacd4345b72e20da3bb7cb76 [id=b24212eeb7784959b5fbcd68bcafa0d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.570931 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c228645fe0b4bc89643178fff7b28a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.575430 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 534e3bceaa5c4c77abaf288ea08287ff (DEFAULT_TABLE table=default.Table_116d421c142443cebb54af5cca25ee16 [id=467a7f6db73641ca85a1898b251938fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.575840 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf60a6da6f94525b02bdbb9e53192db (DEFAULT_TABLE table=default.Table_ff549ea350ef46a59aefb1e305429d98 [id=40060891c6d44d559c37891803b42478]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.576006 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 534e3bceaa5c4c77abaf288ea08287ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.576433 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf60a6da6f94525b02bdbb9e53192db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.579222 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 534e3bceaa5c4c77abaf288ea08287ff (DEFAULT_TABLE table=default.Table_116d421c142443cebb54af5cca25ee16 [id=467a7f6db73641ca85a1898b251938fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.579809 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 534e3bceaa5c4c77abaf288ea08287ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.589169 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 67bc4942db6140a3b7984318a78f7ad3 (DEFAULT_TABLE table=default.Table_938322b142834d4eb5ff155d2425e348 [id=7338537cdd4249359d2af60930089412]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.589751 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67bc4942db6140a3b7984318a78f7ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.591198 23901 tablet_service.cc:1467] Processing CreateTablet for tablet abd595972fc144328c279a84ec7eeaa9 (DEFAULT_TABLE table=default.Table_bc71bb1f437e4476b7f3a26061a52db6 [id=1e590d80e3e34e1aac8a0c7908a9d813]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.591871 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abd595972fc144328c279a84ec7eeaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.599473 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 67bc4942db6140a3b7984318a78f7ad3 (DEFAULT_TABLE table=default.Table_938322b142834d4eb5ff155d2425e348 [id=7338537cdd4249359d2af60930089412]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.599972 24037 tablet_service.cc:1467] Processing CreateTablet for tablet abd595972fc144328c279a84ec7eeaa9 (DEFAULT_TABLE table=default.Table_bc71bb1f437e4476b7f3a26061a52db6 [id=1e590d80e3e34e1aac8a0c7908a9d813]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.600142 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67bc4942db6140a3b7984318a78f7ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.600462 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abd595972fc144328c279a84ec7eeaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.614069 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf60a6da6f94525b02bdbb9e53192db (DEFAULT_TABLE table=default.Table_ff549ea350ef46a59aefb1e305429d98 [id=40060891c6d44d559c37891803b42478]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.614789 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf60a6da6f94525b02bdbb9e53192db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.623651 24175 tablet_bootstrap.cc:492] T 67bc4942db6140a3b7984318a78f7ad3 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:33.625885 24174 tablet_bootstrap.cc:492] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:33.633808 24175 tablet_bootstrap.cc:654] T 67bc4942db6140a3b7984318a78f7ad3 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:33.635782 24175 log.cc:824] T 67bc4942db6140a3b7984318a78f7ad3 P 5b89a1f0485a44b7ac1710e2369a809b: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:33.638764 24174 tablet_bootstrap.cc:654] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:33.640179 24174 log.cc:824] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:33.643261 24173 tablet_bootstrap.cc:492] T 6c228645fe0b4bc89643178fff7b28a3 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:33.646028 23605 master_service.cc:946] ReplaceTablet: received request to replace tablet 0cf60a6da6f94525b02bdbb9e53192db from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:33.647023 24173 tablet_bootstrap.cc:654] T 6c228645fe0b4bc89643178fff7b28a3 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:33.648356 24173 log.cc:824] T 6c228645fe0b4bc89643178fff7b28a3 P 149ce744feab49b5aa5bcaa14bd6aba5: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:33.653142 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 0cf60a6da6f94525b02bdbb9e53192db
I20240502 08:16:33.654068 23605 catalog_manager.cc:6554] ReplaceTablet: tablet 0cf60a6da6f94525b02bdbb9e53192db deleted and replaced by tablet a2d10db950314f3db1bfa352b9d3a759
I20240502 08:16:33.655436 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf60a6da6f94525b02bdbb9e53192db with delete_type TABLET_DATA_DELETED (replaced by tablet a2d10db950314f3db1bfa352b9d3a759) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:33.656596 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf60a6da6f94525b02bdbb9e53192db with delete_type TABLET_DATA_DELETED (replaced by tablet a2d10db950314f3db1bfa352b9d3a759) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:33.657068 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf60a6da6f94525b02bdbb9e53192db with delete_type TABLET_DATA_DELETED (replaced by tablet a2d10db950314f3db1bfa352b9d3a759) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:33.657366 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet 6c228645fe0b4bc89643178fff7b28a3 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:33.657685 23607 master_service.cc:946] ReplaceTablet: received request to replace tablet 0cf60a6da6f94525b02bdbb9e53192db from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:33.658260 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet 6c228645fe0b4bc89643178fff7b28a3 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:33.662567 24174 tablet_bootstrap.cc:492] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:33.662883 24174 ts_tablet_manager.cc:1367] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.000s
I20240502 08:16:33.664294 23605 master_service.cc:946] ReplaceTablet: received request to replace tablet 67bc4942db6140a3b7984318a78f7ad3 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:33.665657 24175 tablet_bootstrap.cc:492] T 67bc4942db6140a3b7984318a78f7ad3 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:33.665943 24175 ts_tablet_manager.cc:1367] T 67bc4942db6140a3b7984318a78f7ad3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.008s
W20240502 08:16:33.667024 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 0cf60a6da6f94525b02bdbb9e53192db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cf60a6da6f94525b02bdbb9e53192db already in progress: creating tablet
W20240502 08:16:33.669553 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 0cf60a6da6f94525b02bdbb9e53192db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cf60a6da6f94525b02bdbb9e53192db already in progress: creating tablet
I20240502 08:16:33.672225 23902 tablet_service.cc:1467] Processing CreateTablet for tablet a2d10db950314f3db1bfa352b9d3a759 (DEFAULT_TABLE table=default.Table_ff549ea350ef46a59aefb1e305429d98 [id=40060891c6d44d559c37891803b42478]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:33.672441 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 0cf60a6da6f94525b02bdbb9e53192db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cf60a6da6f94525b02bdbb9e53192db already in progress: creating tablet
I20240502 08:16:33.672708 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2d10db950314f3db1bfa352b9d3a759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.673141 24174 raft_consensus.cc:348] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.673624 24174 raft_consensus.cc:374] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:33.673727 24174 raft_consensus.cc:729] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:33.674624 24174 consensus_queue.cc:260] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.675894 23764 tablet_service.cc:1467] Processing CreateTablet for tablet a2d10db950314f3db1bfa352b9d3a759 (DEFAULT_TABLE table=default.Table_ff549ea350ef46a59aefb1e305429d98 [id=40060891c6d44d559c37891803b42478]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.676369 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2d10db950314f3db1bfa352b9d3a759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.676715 24175 raft_consensus.cc:348] T 67bc4942db6140a3b7984318a78f7ad3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.677218 24175 raft_consensus.cc:374] T 67bc4942db6140a3b7984318a78f7ad3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:33.677356 24175 raft_consensus.cc:729] T 67bc4942db6140a3b7984318a78f7ad3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:33.678184 24175 consensus_queue.cc:260] T 67bc4942db6140a3b7984318a78f7ad3 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.681075 24040 tablet_service.cc:1467] Processing CreateTablet for tablet a2d10db950314f3db1bfa352b9d3a759 (DEFAULT_TABLE table=default.Table_ff549ea350ef46a59aefb1e305429d98 [id=40060891c6d44d559c37891803b42478]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.681679 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2d10db950314f3db1bfa352b9d3a759. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.684579 24173 tablet_bootstrap.cc:492] T 6c228645fe0b4bc89643178fff7b28a3 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:33.685055 24173 ts_tablet_manager.cc:1367] T 6c228645fe0b4bc89643178fff7b28a3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.042s user 0.016s sys 0.005s
I20240502 08:16:33.686506 23429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2vVFYb/build/asan/bin/kudu with pid 23437
I20240502 08:16:33.688970 24175 ts_tablet_manager.cc:1398] T 67bc4942db6140a3b7984318a78f7ad3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.023s user 0.014s sys 0.004s
I20240502 08:16:33.692304 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 6c228645fe0b4bc89643178fff7b28a3
I20240502 08:16:33.692832 23604 catalog_manager.cc:6554] ReplaceTablet: tablet 6c228645fe0b4bc89643178fff7b28a3 deleted and replaced by tablet 5c8a832a2e6844ff8481b1b66f30a3b2
I20240502 08:16:33.693526 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c228645fe0b4bc89643178fff7b28a3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c8a832a2e6844ff8481b1b66f30a3b2) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:33.694353 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 6c228645fe0b4bc89643178fff7b28a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c228645fe0b4bc89643178fff7b28a3 already in progress: creating tablet
I20240502 08:16:33.694473 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 67bc4942db6140a3b7984318a78f7ad3
I20240502 08:16:33.694960 23605 catalog_manager.cc:6554] ReplaceTablet: tablet 67bc4942db6140a3b7984318a78f7ad3 deleted and replaced by tablet 9c08a89a72894f0b96d7863fd71edfd0
I20240502 08:16:33.696460 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c228645fe0b4bc89643178fff7b28a3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c8a832a2e6844ff8481b1b66f30a3b2) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:33.696607 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 67bc4942db6140a3b7984318a78f7ad3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c08a89a72894f0b96d7863fd71edfd0) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:33.698421 24175 tablet_bootstrap.cc:492] T 0cf60a6da6f94525b02bdbb9e53192db P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:33.697513 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 67bc4942db6140a3b7984318a78f7ad3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67bc4942db6140a3b7984318a78f7ad3 already in progress: creating tablet
I20240502 08:16:33.700140 24175 tablet_bootstrap.cc:654] T 0cf60a6da6f94525b02bdbb9e53192db P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:33.702042 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet 67bc4942db6140a3b7984318a78f7ad3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c08a89a72894f0b96d7863fd71edfd0) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:33.703928 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_69151b7e39d14cc78bef4e8de2b2e109"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:33.704516 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69151b7e39d14cc78bef4e8de2b2e109 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:33.704829 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d32128de7b384f9cbc59df9a0782346d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:33.705361 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d32128de7b384f9cbc59df9a0782346d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:33.705786 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ff9e2689d66842458186e91ada519f05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:33.706228 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 67bc4942db6140a3b7984318a78f7ad3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c08a89a72894f0b96d7863fd71edfd0) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:33.706379 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff9e2689d66842458186e91ada519f05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:33.706573 24179 tablet_replica.cc:330] T 67bc4942db6140a3b7984318a78f7ad3 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:33.706792 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5485d4b9db4f416691aea5c4f94ba412"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:33.707149 24179 raft_consensus.cc:2229] T 67bc4942db6140a3b7984318a78f7ad3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
W20240502 08:16:33.707486 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5485d4b9db4f416691aea5c4f94ba412 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:33.707646 24179 raft_consensus.cc:2258] T 67bc4942db6140a3b7984318a78f7ad3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:33.708196 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b8ba42e1326143be948727bdcaa603f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:33.708827 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8ba42e1326143be948727bdcaa603f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:33.709759 24179 ts_tablet_manager.cc:1875] T 67bc4942db6140a3b7984318a78f7ad3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:33.710026 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c228645fe0b4bc89643178fff7b28a3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c8a832a2e6844ff8481b1b66f30a3b2) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:33.705823 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_031544f2dbe84f6e83cacfdfa409db06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:33.711023 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_031544f2dbe84f6e83cacfdfa409db06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:33.709797 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b01c340249ea483683e385a7fa964a51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:33.713234 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b01c340249ea483683e385a7fa964a51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:33.714022 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_4713741287e34e5d986ffb272ed6291d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:33.714470 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_545327c2458643f4956f4cc391f76a50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:33.714708 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4713741287e34e5d986ffb272ed6291d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:33.714967 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_545327c2458643f4956f4cc391f76a50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:33.718741 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 6c228645fe0b4bc89643178fff7b28a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c228645fe0b4bc89643178fff7b28a3 already in progress: creating tablet
W20240502 08:16:33.719252 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 67bc4942db6140a3b7984318a78f7ad3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67bc4942db6140a3b7984318a78f7ad3 already in progress: creating tablet
I20240502 08:16:33.725914 24179 ts_tablet_manager.cc:1888] T 67bc4942db6140a3b7984318a78f7ad3 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:33.726138 24179 log.cc:1196] T 67bc4942db6140a3b7984318a78f7ad3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/67bc4942db6140a3b7984318a78f7ad3
I20240502 08:16:33.726823 24179 ts_tablet_manager.cc:1909] T 67bc4942db6140a3b7984318a78f7ad3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:33.727342 24184 raft_consensus.cc:482] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:33.728403 24184 raft_consensus.cc:504] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.730594 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 67bc4942db6140a3b7984318a78f7ad3 (table default.Table_938322b142834d4eb5ff155d2425e348 [id=7338537cdd4249359d2af60930089412]) successfully deleted
W20240502 08:16:33.731396 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 6c228645fe0b4bc89643178fff7b28a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c228645fe0b4bc89643178fff7b28a3 already in progress: creating tablet
I20240502 08:16:33.734783 24174 ts_tablet_manager.cc:1398] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.072s user 0.016s sys 0.015s
I20240502 08:16:33.737155 24174 tablet_bootstrap.cc:492] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:33.738207 24184 leader_election.cc:288] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:33.739950 24175 tablet_bootstrap.cc:492] T 0cf60a6da6f94525b02bdbb9e53192db P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:33.740119 24175 ts_tablet_manager.cc:1367] T 0cf60a6da6f94525b02bdbb9e53192db P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.042s user 0.013s sys 0.025s
I20240502 08:16:33.740559 24174 tablet_bootstrap.cc:654] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:33.740927 24175 raft_consensus.cc:348] T 0cf60a6da6f94525b02bdbb9e53192db P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.741305 24175 raft_consensus.cc:374] T 0cf60a6da6f94525b02bdbb9e53192db P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:33.741451 24175 raft_consensus.cc:729] T 0cf60a6da6f94525b02bdbb9e53192db P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:33.741801 24175 consensus_queue.cc:260] T 0cf60a6da6f94525b02bdbb9e53192db P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.743170 24175 ts_tablet_manager.cc:1398] T 0cf60a6da6f94525b02bdbb9e53192db P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:33.744666 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 5c8a832a2e6844ff8481b1b66f30a3b2 (DEFAULT_TABLE table=default.Table_ffa5fe5beacd4345b72e20da3bb7cb76 [id=b24212eeb7784959b5fbcd68bcafa0d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.745254 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c8a832a2e6844ff8481b1b66f30a3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.745510 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 5c8a832a2e6844ff8481b1b66f30a3b2 (DEFAULT_TABLE table=default.Table_ffa5fe5beacd4345b72e20da3bb7cb76 [id=b24212eeb7784959b5fbcd68bcafa0d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.746059 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c8a832a2e6844ff8481b1b66f30a3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.749778 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 67bc4942db6140a3b7984318a78f7ad3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c08a89a72894f0b96d7863fd71edfd0) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:33.750553 23597 catalog_manager.cc:4797] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 67bc4942db6140a3b7984318a78f7ad3 because the tablet was not found. No further retry: Not found: Tablet not found: 67bc4942db6140a3b7984318a78f7ad3
I20240502 08:16:33.752218 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 5c8a832a2e6844ff8481b1b66f30a3b2 (DEFAULT_TABLE table=default.Table_ffa5fe5beacd4345b72e20da3bb7cb76 [id=b24212eeb7784959b5fbcd68bcafa0d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.752532 24175 tablet_bootstrap.cc:492] T abd595972fc144328c279a84ec7eeaa9 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:33.752642 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c8a832a2e6844ff8481b1b66f30a3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.756299 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c60e3fa5881347d09ba464e544d7cbf8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:33.756908 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c60e3fa5881347d09ba464e544d7cbf8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:33.757269 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bc71bb1f437e4476b7f3a26061a52db6" } new_table_name: "default.Table_2e0ee58bc7db40da88a1700e3830444d" modify_external_catalogs: true
I20240502 08:16:33.757359 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf60a6da6f94525b02bdbb9e53192db with delete_type TABLET_DATA_DELETED (replaced by tablet a2d10db950314f3db1bfa352b9d3a759) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:33.757654 24179 tablet_replica.cc:330] T 0cf60a6da6f94525b02bdbb9e53192db P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:33.757833 24179 raft_consensus.cc:2229] T 0cf60a6da6f94525b02bdbb9e53192db P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:33.757982 24179 raft_consensus.cc:2258] T 0cf60a6da6f94525b02bdbb9e53192db P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:33.758702 24179 ts_tablet_manager.cc:1875] T 0cf60a6da6f94525b02bdbb9e53192db P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:33.758747 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_938322b142834d4eb5ff155d2425e348" } new_table_name: "default.Table_d351b6717f35441cb5d9cf27f5c1087c" modify_external_catalogs: true
I20240502 08:16:33.761729 24173 raft_consensus.cc:348] T 6c228645fe0b4bc89643178fff7b28a3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.763375 24175 tablet_bootstrap.cc:654] T abd595972fc144328c279a84ec7eeaa9 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:33.765616 24173 raft_consensus.cc:374] T 6c228645fe0b4bc89643178fff7b28a3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:33.765542 24179 ts_tablet_manager.cc:1888] T 0cf60a6da6f94525b02bdbb9e53192db P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:33.765847 24179 log.cc:1196] T 0cf60a6da6f94525b02bdbb9e53192db P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/0cf60a6da6f94525b02bdbb9e53192db
I20240502 08:16:33.765834 24173 raft_consensus.cc:729] T 6c228645fe0b4bc89643178fff7b28a3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:33.766284 24179 ts_tablet_manager.cc:1909] T 0cf60a6da6f94525b02bdbb9e53192db P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:33.766947 24173 consensus_queue.cc:260] T 6c228645fe0b4bc89643178fff7b28a3 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.767597 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ffa5fe5beacd4345b72e20da3bb7cb76" } modify_external_catalogs: true
I20240502 08:16:33.767819 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ffa5fe5beacd4345b72e20da3bb7cb76" } modify_external_catalogs: true
I20240502 08:16:33.769500 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 0cf60a6da6f94525b02bdbb9e53192db (table default.Table_ff549ea350ef46a59aefb1e305429d98 [id=40060891c6d44d559c37891803b42478]) successfully deleted
I20240502 08:16:33.772056 24105 heartbeater.cc:503] Master 127.22.225.124:41745 was elected leader, sending a full tablet report...
I20240502 08:16:33.773515 24173 ts_tablet_manager.cc:1398] T 6c228645fe0b4bc89643178fff7b28a3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.088s user 0.001s sys 0.012s
W20240502 08:16:33.774762 23994 connection.cc:535] client connection to 127.22.225.126:43265 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:16:33.774938 24173 tablet_bootstrap.cc:492] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:33.775467 24106 heartbeater.cc:644] Failed to heartbeat to 127.22.225.126:43265 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:16:33.776664 24173 tablet_bootstrap.cc:654] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:33.776922 23859 connection.cc:535] client connection to 127.22.225.126:43265 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:16:33.777585 23971 heartbeater.cc:644] Failed to heartbeat to 127.22.225.126:43265 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:16:33.781850 24175 tablet_bootstrap.cc:492] T abd595972fc144328c279a84ec7eeaa9 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:33.782037 24175 ts_tablet_manager.cc:1367] T abd595972fc144328c279a84ec7eeaa9 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.001s
I20240502 08:16:33.782850 24175 raft_consensus.cc:348] T abd595972fc144328c279a84ec7eeaa9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.783123 24175 raft_consensus.cc:374] T abd595972fc144328c279a84ec7eeaa9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:33.783214 24175 raft_consensus.cc:729] T abd595972fc144328c279a84ec7eeaa9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:33.783484 24175 consensus_queue.cc:260] T abd595972fc144328c279a84ec7eeaa9 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:33.783799 23721 connection.cc:535] client connection to 127.22.225.126:43265 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:16:33.784747 23834 heartbeater.cc:644] Failed to heartbeat to 127.22.225.126:43265 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:16:33.787520 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd595972fc144328c279a84ec7eeaa9" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:33.789079 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd595972fc144328c279a84ec7eeaa9" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:33.789697 23925 raft_consensus.cc:2454] T abd595972fc144328c279a84ec7eeaa9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:33.789762 24175 ts_tablet_manager.cc:1398] T abd595972fc144328c279a84ec7eeaa9 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:16:33.790074 24175 tablet_bootstrap.cc:492] T 534e3bceaa5c4c77abaf288ea08287ff P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:33.790722 23723 leader_election.cc:302] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:33.791200 24184 raft_consensus.cc:2781] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:33.791353 24184 raft_consensus.cc:482] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:33.791455 24184 raft_consensus.cc:3037] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:33.791597 24175 tablet_bootstrap.cc:654] T 534e3bceaa5c4c77abaf288ea08287ff P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:33.791707 23723 leader_election.cc:341] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:33.792655 23595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.126:43265 (error 108)
I20240502 08:16:33.795166 24184 raft_consensus.cc:504] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.796020 24184 leader_election.cc:288] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:33.796764 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd595972fc144328c279a84ec7eeaa9" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:33.796983 23925 raft_consensus.cc:3037] T abd595972fc144328c279a84ec7eeaa9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:33.801201 23925 raft_consensus.cc:2454] T abd595972fc144328c279a84ec7eeaa9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:33.801716 23723 leader_election.cc:302] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:33.802044 24184 raft_consensus.cc:2781] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:33.802197 24184 raft_consensus.cc:686] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:33.802287 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd595972fc144328c279a84ec7eeaa9" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:33.802722 24184 consensus_queue.cc:237] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:33.802922 23723 leader_election.cc:341] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:33.806028 24109 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:16:33.776272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.126:43265 (local address 127.22.225.67:49161)
0502 08:16:33.776477 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0502 08:16:33.776495 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0502 08:16:33.776591 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:16:33.804701 (+ 28110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.126:43265: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":102}
W20240502 08:16:33.807613 24189 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:16:33.786238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.126:43265 (local address 127.22.225.65:41445)
0502 08:16:33.786651 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0502 08:16:33.786669 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0502 08:16:33.786769 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:16:33.806859 (+ 20090us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.126:43265: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":295}
W20240502 08:16:33.807787 24168 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:16:33.803704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.126:43265 (local address 127.0.0.1:59772)
0502 08:16:33.803891 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0502 08:16:33.803911 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0502 08:16:33.804028 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:16:33.806895 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.126:43265: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":42}
W20240502 08:16:33.808179 23595 consensus_peers.cc:485] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 -> Peer bcc8a323e6b9429eaaf4e0c1698ad077 (127.22.225.126:43265): Couldn't send request to peer bcc8a323e6b9429eaaf4e0c1698ad077. Status: Network error: Client connection negotiation failed: client connection to 127.22.225.126:43265: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:33.808357 24174 tablet_bootstrap.cc:492] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:33.808573 24174 ts_tablet_manager.cc:1367] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.072s user 0.007s sys 0.002s
I20240502 08:16:33.809798 24174 raft_consensus.cc:348] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.810053 24174 raft_consensus.cc:374] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:33.810175 24174 raft_consensus.cc:729] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:33.810456 24174 consensus_queue.cc:260] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.811878 24174 ts_tablet_manager.cc:1398] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:33.812212 24174 tablet_bootstrap.cc:492] T 534e3bceaa5c4c77abaf288ea08287ff P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:33.814450 24174 tablet_bootstrap.cc:654] T 534e3bceaa5c4c77abaf288ea08287ff P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:33.815500 24192 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:16:33.778245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.126:43265 (local address 127.22.225.66:56165)
0502 08:16:33.813513 (+ 35268us) negotiation.cc:107] Waiting for socket to connect
0502 08:16:33.814017 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.126:43265: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":35139,"thread_start_us":35326,"threads_started":1}
I20240502 08:16:33.819674 24182 raft_consensus.cc:482] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:33.819824 24182 raft_consensus.cc:504] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.820629 24182 leader_election.cc:288] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:33.822006 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67bc4942db6140a3b7984318a78f7ad3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:33.822623 23723 leader_election.cc:341] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Not found: Tablet not found: 67bc4942db6140a3b7984318a78f7ad3
W20240502 08:16:33.823029 23723 leader_election.cc:341] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:33.823060 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ff549ea350ef46a59aefb1e305429d98" } new_table_name: "default.Table_0112e7ea83b84f04a2e6038f2179b7c6" modify_external_catalogs: true
I20240502 08:16:33.823158 23723 leader_election.cc:302] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:33.823407 24184 raft_consensus.cc:2726] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:33.825037 24173 tablet_bootstrap.cc:492] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:33.825228 24173 ts_tablet_manager.cc:1367] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.018s
I20240502 08:16:33.825733 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_116d421c142443cebb54af5cca25ee16" } new_table_name: "default.Table_232703fab6964c29987c2f5096c5e118" modify_external_catalogs: true
I20240502 08:16:33.826555 24174 tablet_bootstrap.cc:492] T 534e3bceaa5c4c77abaf288ea08287ff P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:33.826764 24174 ts_tablet_manager.cc:1367] T 534e3bceaa5c4c77abaf288ea08287ff P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240502 08:16:33.827649 24174 raft_consensus.cc:348] T 534e3bceaa5c4c77abaf288ea08287ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.827944 24174 raft_consensus.cc:374] T 534e3bceaa5c4c77abaf288ea08287ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:33.828083 24174 raft_consensus.cc:729] T 534e3bceaa5c4c77abaf288ea08287ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:33.828426 24174 consensus_queue.cc:260] T 534e3bceaa5c4c77abaf288ea08287ff P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.829057 24174 ts_tablet_manager.cc:1398] T 534e3bceaa5c4c77abaf288ea08287ff P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:33.829954 24174 tablet_bootstrap.cc:492] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:33.830431 24173 raft_consensus.cc:348] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.832345 24174 tablet_bootstrap.cc:654] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:33.832485 24173 raft_consensus.cc:374] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:33.832656 24173 raft_consensus.cc:729] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:33.833007 24173 consensus_queue.cc:260] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.833822 24173 ts_tablet_manager.cc:1398] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:16:33.835882 24173 tablet_bootstrap.cc:492] T abd595972fc144328c279a84ec7eeaa9 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:33.836400 24175 tablet_bootstrap.cc:492] T 534e3bceaa5c4c77abaf288ea08287ff P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:33.836585 24175 ts_tablet_manager.cc:1367] T 534e3bceaa5c4c77abaf288ea08287ff P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.047s user 0.018s sys 0.025s
I20240502 08:16:33.837643 24175 raft_consensus.cc:348] T 534e3bceaa5c4c77abaf288ea08287ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.837821 24173 tablet_bootstrap.cc:654] T abd595972fc144328c279a84ec7eeaa9 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:33.837873 24175 raft_consensus.cc:374] T 534e3bceaa5c4c77abaf288ea08287ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:33.837975 24175 raft_consensus.cc:729] T 534e3bceaa5c4c77abaf288ea08287ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:33.838272 24175 consensus_queue.cc:260] T 534e3bceaa5c4c77abaf288ea08287ff P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.839195 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d32128de7b384f9cbc59df9a0782346d" } modify_external_catalogs: true
I20240502 08:16:33.839422 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d32128de7b384f9cbc59df9a0782346d" } modify_external_catalogs: true
I20240502 08:16:33.840526 24175 ts_tablet_manager.cc:1398] T 534e3bceaa5c4c77abaf288ea08287ff P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:16:33.841799 24175 tablet_bootstrap.cc:492] T 6c228645fe0b4bc89643178fff7b28a3 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:33.843502 24175 tablet_bootstrap.cc:654] T 6c228645fe0b4bc89643178fff7b28a3 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:33.847548 24174 tablet_bootstrap.cc:492] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:33.847826 24174 ts_tablet_manager.cc:1367] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.007s
I20240502 08:16:33.848830 24174 raft_consensus.cc:348] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.849160 24174 raft_consensus.cc:374] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:33.849308 24174 raft_consensus.cc:729] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:33.849663 24174 consensus_queue.cc:260] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.852416 24174 ts_tablet_manager.cc:1398] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:16:33.853863 24174 tablet_bootstrap.cc:492] T 6c228645fe0b4bc89643178fff7b28a3 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:33.855674 23602 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet abd595972fc144328c279a84ec7eeaa9 (table default.Table_bc71bb1f437e4476b7f3a26061a52db6 [id=1e590d80e3e34e1aac8a0c7908a9d813]). Expected version 1 got 0
I20240502 08:16:33.856004 24174 tablet_bootstrap.cc:654] T 6c228645fe0b4bc89643178fff7b28a3 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:33.856243 23606 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet abd595972fc144328c279a84ec7eeaa9 (table default.Table_bc71bb1f437e4476b7f3a26061a52db6 [id=1e590d80e3e34e1aac8a0c7908a9d813]). Expected version 1 got 0
I20240502 08:16:33.857054 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 9c08a89a72894f0b96d7863fd71edfd0 (DEFAULT_TABLE table=default.Table_938322b142834d4eb5ff155d2425e348 [id=7338537cdd4249359d2af60930089412]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.857545 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c08a89a72894f0b96d7863fd71edfd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.858292 24173 tablet_bootstrap.cc:492] T abd595972fc144328c279a84ec7eeaa9 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:33.858513 24173 ts_tablet_manager.cc:1367] T abd595972fc144328c279a84ec7eeaa9 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.005s
I20240502 08:16:33.859552 24173 raft_consensus.cc:348] T abd595972fc144328c279a84ec7eeaa9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.859860 24173 raft_consensus.cc:374] T abd595972fc144328c279a84ec7eeaa9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:33.859943 24173 raft_consensus.cc:729] T abd595972fc144328c279a84ec7eeaa9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:33.860277 24173 consensus_queue.cc:260] T abd595972fc144328c279a84ec7eeaa9 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.861476 24173 ts_tablet_manager.cc:1398] T abd595972fc144328c279a84ec7eeaa9 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 08:16:33.861893 24173 tablet_bootstrap.cc:492] T 67bc4942db6140a3b7984318a78f7ad3 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:33.864220 24173 tablet_bootstrap.cc:654] T 67bc4942db6140a3b7984318a78f7ad3 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:33.866091 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 9c08a89a72894f0b96d7863fd71edfd0 (DEFAULT_TABLE table=default.Table_938322b142834d4eb5ff155d2425e348 [id=7338537cdd4249359d2af60930089412]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.866562 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c08a89a72894f0b96d7863fd71edfd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:33.873761 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 9c08a89a72894f0b96d7863fd71edfd0 (DEFAULT_TABLE table=default.Table_938322b142834d4eb5ff155d2425e348 [id=7338537cdd4249359d2af60930089412]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:33.874325 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c08a89a72894f0b96d7863fd71edfd0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:33.875310 24108 tablet.cc:2369] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:33.876595 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_69151b7e39d14cc78bef4e8de2b2e109" } modify_external_catalogs: true
I20240502 08:16:33.876832 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_69151b7e39d14cc78bef4e8de2b2e109" } modify_external_catalogs: true
I20240502 08:16:33.879642 24175 tablet_bootstrap.cc:492] T 6c228645fe0b4bc89643178fff7b28a3 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:33.879885 24175 ts_tablet_manager.cc:1367] T 6c228645fe0b4bc89643178fff7b28a3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.024s
I20240502 08:16:33.880937 24175 raft_consensus.cc:348] T 6c228645fe0b4bc89643178fff7b28a3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.881248 24175 raft_consensus.cc:374] T 6c228645fe0b4bc89643178fff7b28a3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:33.881343 24175 raft_consensus.cc:729] T 6c228645fe0b4bc89643178fff7b28a3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:33.881767 24175 consensus_queue.cc:260] T 6c228645fe0b4bc89643178fff7b28a3 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.884052 24175 ts_tablet_manager.cc:1398] T 6c228645fe0b4bc89643178fff7b28a3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:16:33.887293 24173 tablet_bootstrap.cc:492] T 67bc4942db6140a3b7984318a78f7ad3 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:33.887495 24173 ts_tablet_manager.cc:1367] T 67bc4942db6140a3b7984318a78f7ad3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.008s
I20240502 08:16:33.888525 24173 raft_consensus.cc:348] T 67bc4942db6140a3b7984318a78f7ad3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.888790 23607 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 3d30ee6d04134002868d672bc696ce35
I20240502 08:16:33.888796 24173 raft_consensus.cc:374] T 67bc4942db6140a3b7984318a78f7ad3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:33.888906 24173 raft_consensus.cc:729] T 67bc4942db6140a3b7984318a78f7ad3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:33.889214 24173 consensus_queue.cc:260] T 67bc4942db6140a3b7984318a78f7ad3 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.891022 24173 ts_tablet_manager.cc:1398] T 67bc4942db6140a3b7984318a78f7ad3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:33.891410 24173 tablet_bootstrap.cc:492] T 0cf60a6da6f94525b02bdbb9e53192db P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:33.894481 24173 tablet_bootstrap.cc:654] T 0cf60a6da6f94525b02bdbb9e53192db P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:33.895852 24175 tablet_bootstrap.cc:492] T a2d10db950314f3db1bfa352b9d3a759 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:33.897943 24175 tablet_bootstrap.cc:654] T a2d10db950314f3db1bfa352b9d3a759 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:33.898975 24174 tablet_bootstrap.cc:492] T 6c228645fe0b4bc89643178fff7b28a3 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:33.899173 24174 ts_tablet_manager.cc:1367] T 6c228645fe0b4bc89643178fff7b28a3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.045s user 0.024s sys 0.005s
I20240502 08:16:33.899116 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ff9e2689d66842458186e91ada519f05" } new_table_name: "default.Table_2047aed3b97c4f4283bd8f6e80aeb92e" modify_external_catalogs: true
I20240502 08:16:33.902254 24174 raft_consensus.cc:348] T 6c228645fe0b4bc89643178fff7b28a3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.902493 24174 raft_consensus.cc:374] T 6c228645fe0b4bc89643178fff7b28a3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:33.902572 24174 raft_consensus.cc:729] T 6c228645fe0b4bc89643178fff7b28a3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:33.902900 24174 consensus_queue.cc:260] T 6c228645fe0b4bc89643178fff7b28a3 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.903455 24174 ts_tablet_manager.cc:1398] T 6c228645fe0b4bc89643178fff7b28a3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240502 08:16:33.903802 24174 tablet_bootstrap.cc:492] T 5c8a832a2e6844ff8481b1b66f30a3b2 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:33.905652 24174 tablet_bootstrap.cc:654] T 5c8a832a2e6844ff8481b1b66f30a3b2 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:33.912511 23609 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 534e3bceaa5c4c77abaf288ea08287ff (table default.Table_116d421c142443cebb54af5cca25ee16 [id=467a7f6db73641ca85a1898b251938fb]). Expected version 1 got 0
I20240502 08:16:33.912856 23609 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet abd595972fc144328c279a84ec7eeaa9 (table default.Table_bc71bb1f437e4476b7f3a26061a52db6 [id=1e590d80e3e34e1aac8a0c7908a9d813]). Expected version 1 got 0
W20240502 08:16:33.913417 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet abd595972fc144328c279a84ec7eeaa9 consensus state has no leader
I20240502 08:16:33.913813 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5485d4b9db4f416691aea5c4f94ba412" } modify_external_catalogs: true
I20240502 08:16:33.914006 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5485d4b9db4f416691aea5c4f94ba412" } modify_external_catalogs: true
W20240502 08:16:33.914528 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2d10db950314f3db1bfa352b9d3a759 consensus state has no leader
I20240502 08:16:33.914867 24174 tablet_bootstrap.cc:492] T 5c8a832a2e6844ff8481b1b66f30a3b2 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:33.914872 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_031544f2dbe84f6e83cacfdfa409db06" } modify_external_catalogs: true
I20240502 08:16:33.915017 24174 ts_tablet_manager.cc:1367] T 5c8a832a2e6844ff8481b1b66f30a3b2 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 08:16:33.915051 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_031544f2dbe84f6e83cacfdfa409db06" } modify_external_catalogs: true
W20240502 08:16:33.915128 23973 tablet.cc:2369] T 534e3bceaa5c4c77abaf288ea08287ff P 5b89a1f0485a44b7ac1710e2369a809b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:33.915649 24174 raft_consensus.cc:348] T 5c8a832a2e6844ff8481b1b66f30a3b2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.915844 24174 raft_consensus.cc:374] T 5c8a832a2e6844ff8481b1b66f30a3b2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:33.915936 24174 raft_consensus.cc:729] T 5c8a832a2e6844ff8481b1b66f30a3b2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
W20240502 08:16:33.916256 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 534e3bceaa5c4c77abaf288ea08287ff consensus state has no leader
W20240502 08:16:33.916218 23836 tablet.cc:2369] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:33.916517 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet abd595972fc144328c279a84ec7eeaa9 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:33.917261 24184 raft_consensus.cc:482] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:33.916214 24174 consensus_queue.cc:260] T 5c8a832a2e6844ff8481b1b66f30a3b2 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.917395 24184 raft_consensus.cc:504] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:33.917727 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c08a89a72894f0b96d7863fd71edfd0 consensus state has no leader
I20240502 08:16:33.917924 23603 master_service.cc:946] ReplaceTablet: received request to replace tablet 534e3bceaa5c4c77abaf288ea08287ff from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:33.918100 24184 leader_election.cc:288] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:33.919028 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf60a6da6f94525b02bdbb9e53192db" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:33.919191 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet abd595972fc144328c279a84ec7eeaa9 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:33.919266 24174 ts_tablet_manager.cc:1398] T 5c8a832a2e6844ff8481b1b66f30a3b2 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:16:33.919540 24174 tablet_bootstrap.cc:492] T a2d10db950314f3db1bfa352b9d3a759 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:33.919675 23606 master_service.cc:946] ReplaceTablet: received request to replace tablet abd595972fc144328c279a84ec7eeaa9 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:33.920652 24190 raft_consensus.cc:482] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:33.920851 24190 raft_consensus.cc:504] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.921187 24174 tablet_bootstrap.cc:654] T a2d10db950314f3db1bfa352b9d3a759 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:33.922339 23723 leader_election.cc:341] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Not found: Tablet not found: 0cf60a6da6f94525b02bdbb9e53192db
I20240502 08:16:33.922474 24190 leader_election.cc:288] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:33.923161 23723 leader_election.cc:341] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:33.923324 23723 leader_election.cc:302] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:33.926836 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 5c8a832a2e6844ff8481b1b66f30a3b2
I20240502 08:16:33.926899 24184 raft_consensus.cc:2726] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:33.927882 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c8a832a2e6844ff8481b1b66f30a3b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:33 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:33.929028 24180 tablet_replica.cc:330] T 5c8a832a2e6844ff8481b1b66f30a3b2 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:33.929401 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c8a832a2e6844ff8481b1b66f30a3b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:33 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:33.929544 24180 raft_consensus.cc:2229] T 5c8a832a2e6844ff8481b1b66f30a3b2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:33.929944 24180 raft_consensus.cc:2258] T 5c8a832a2e6844ff8481b1b66f30a3b2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:33.929998 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c8a832a2e6844ff8481b1b66f30a3b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:33 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:33.930696 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 5c8a832a2e6844ff8481b1b66f30a3b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c8a832a2e6844ff8481b1b66f30a3b2 already in progress: creating tablet
W20240502 08:16:33.930934 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 5c8a832a2e6844ff8481b1b66f30a3b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c8a832a2e6844ff8481b1b66f30a3b2 already in progress: creating tablet
I20240502 08:16:33.932026 24180 ts_tablet_manager.cc:1875] T 5c8a832a2e6844ff8481b1b66f30a3b2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:33.935082 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf60a6da6f94525b02bdbb9e53192db with delete_type TABLET_DATA_DELETED (replaced by tablet a2d10db950314f3db1bfa352b9d3a759) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:33.935150 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c8a832a2e6844ff8481b1b66f30a3b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:33 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:33.935107 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet 67bc4942db6140a3b7984318a78f7ad3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c08a89a72894f0b96d7863fd71edfd0) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:33.935098 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534e3bceaa5c4c77abaf288ea08287ff" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:33.935487 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c228645fe0b4bc89643178fff7b28a3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c8a832a2e6844ff8481b1b66f30a3b2) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:33.935523 23788 raft_consensus.cc:2454] T 534e3bceaa5c4c77abaf288ea08287ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:33.935699 24181 tablet_replica.cc:330] T 67bc4942db6140a3b7984318a78f7ad3 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
W20240502 08:16:33.935735 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 0cf60a6da6f94525b02bdbb9e53192db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cf60a6da6f94525b02bdbb9e53192db already in progress: creating tablet
I20240502 08:16:33.936095 24181 raft_consensus.cc:2229] T 67bc4942db6140a3b7984318a78f7ad3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:33.936113 23993 leader_election.cc:302] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:33.936470 24181 raft_consensus.cc:2258] T 67bc4942db6140a3b7984318a78f7ad3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:33.936769 24190 raft_consensus.cc:2781] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:33.936915 24190 raft_consensus.cc:482] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240502 08:16:33.936982 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 5c8a832a2e6844ff8481b1b66f30a3b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c8a832a2e6844ff8481b1b66f30a3b2 already in progress: creating tablet
I20240502 08:16:33.937012 24190 raft_consensus.cc:3037] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:33.937839 24174 tablet_bootstrap.cc:492] T a2d10db950314f3db1bfa352b9d3a759 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:33.937996 24174 ts_tablet_manager.cc:1367] T a2d10db950314f3db1bfa352b9d3a759 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20240502 08:16:33.938109 24181 ts_tablet_manager.cc:1875] T 67bc4942db6140a3b7984318a78f7ad3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:33.939136 24174 raft_consensus.cc:348] T a2d10db950314f3db1bfa352b9d3a759 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.939522 24174 raft_consensus.cc:374] T a2d10db950314f3db1bfa352b9d3a759 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:33.939591 24174 raft_consensus.cc:729] T a2d10db950314f3db1bfa352b9d3a759 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:33.939855 24174 consensus_queue.cc:260] T a2d10db950314f3db1bfa352b9d3a759 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.939893 24190 raft_consensus.cc:504] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.941406 24174 ts_tablet_manager.cc:1398] T a2d10db950314f3db1bfa352b9d3a759 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:16:33.941709 24174 tablet_bootstrap.cc:492] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:33.943540 24174 tablet_bootstrap.cc:654] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:33.946151 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bce9392a8ad64004ba6ce9a236976cd6 has no consensus state
I20240502 08:16:33.948287 23608 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 8dd89f956021430ab5af68e98e0d3b20
I20240502 08:16:33.948539 24181 ts_tablet_manager.cc:1888] T 67bc4942db6140a3b7984318a78f7ad3 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:33.948606 23608 master_service.cc:946] ReplaceTablet: received request to replace tablet 6c228645fe0b4bc89643178fff7b28a3 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:33.948720 24181 log.cc:1196] T 67bc4942db6140a3b7984318a78f7ad3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/67bc4942db6140a3b7984318a78f7ad3
I20240502 08:16:33.948822 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 534e3bceaa5c4c77abaf288ea08287ff
I20240502 08:16:33.949182 24181 ts_tablet_manager.cc:1909] T 67bc4942db6140a3b7984318a78f7ad3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:33.949314 23603 catalog_manager.cc:6554] ReplaceTablet: tablet 534e3bceaa5c4c77abaf288ea08287ff deleted and replaced by tablet 8791834597254e7c9f730e03568c0748
I20240502 08:16:33.949702 24180 ts_tablet_manager.cc:1888] T 5c8a832a2e6844ff8481b1b66f30a3b2 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:33.949908 24180 log.cc:1196] T 5c8a832a2e6844ff8481b1b66f30a3b2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/5c8a832a2e6844ff8481b1b66f30a3b2
I20240502 08:16:33.950135 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet 534e3bceaa5c4c77abaf288ea08287ff with delete_type TABLET_DATA_DELETED (replaced by tablet 8791834597254e7c9f730e03568c0748) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:33.950361 24180 ts_tablet_manager.cc:1909] T 5c8a832a2e6844ff8481b1b66f30a3b2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:33.950541 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet 534e3bceaa5c4c77abaf288ea08287ff with delete_type TABLET_DATA_DELETED (replaced by tablet 8791834597254e7c9f730e03568c0748) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:33.951047 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet 534e3bceaa5c4c77abaf288ea08287ff with delete_type TABLET_DATA_DELETED (replaced by tablet 8791834597254e7c9f730e03568c0748) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:33.951334 24179 tablet_replica.cc:330] T 534e3bceaa5c4c77abaf288ea08287ff P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:33.951493 24179 raft_consensus.cc:2229] T 534e3bceaa5c4c77abaf288ea08287ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:33.951654 24179 raft_consensus.cc:2258] T 534e3bceaa5c4c77abaf288ea08287ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:33.952096 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 5c8a832a2e6844ff8481b1b66f30a3b2 (table default.Table_ffa5fe5beacd4345b72e20da3bb7cb76 [id=b24212eeb7784959b5fbcd68bcafa0d7]) successfully deleted
I20240502 08:16:33.952533 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet abd595972fc144328c279a84ec7eeaa9
I20240502 08:16:33.952679 24179 ts_tablet_manager.cc:1875] T 534e3bceaa5c4c77abaf288ea08287ff P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:33.952951 23604 catalog_manager.cc:6554] ReplaceTablet: tablet abd595972fc144328c279a84ec7eeaa9 deleted and replaced by tablet ec4028a8e66442bfabac9e587aed00bb
I20240502 08:16:33.956094 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet abd595972fc144328c279a84ec7eeaa9 with delete_type TABLET_DATA_DELETED (replaced by tablet ec4028a8e66442bfabac9e587aed00bb) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:33.956941 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet abd595972fc144328c279a84ec7eeaa9 with delete_type TABLET_DATA_DELETED (replaced by tablet ec4028a8e66442bfabac9e587aed00bb) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:33.957526 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet abd595972fc144328c279a84ec7eeaa9 with delete_type TABLET_DATA_DELETED (replaced by tablet ec4028a8e66442bfabac9e587aed00bb) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:33.958621 24179 ts_tablet_manager.cc:1888] T 534e3bceaa5c4c77abaf288ea08287ff P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:33.958698 24174 tablet_bootstrap.cc:492] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:33.958741 24179 log.cc:1196] T 534e3bceaa5c4c77abaf288ea08287ff P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/534e3bceaa5c4c77abaf288ea08287ff
I20240502 08:16:33.958856 24174 ts_tablet_manager.cc:1367] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.014s
I20240502 08:16:33.959089 24179 ts_tablet_manager.cc:1909] T 534e3bceaa5c4c77abaf288ea08287ff P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:33.959630 24174 raft_consensus.cc:348] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:33.959846 24174 raft_consensus.cc:374] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:33.959933 24174 raft_consensus.cc:729] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:33.959993 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 534e3bceaa5c4c77abaf288ea08287ff (table default.Table_232703fab6964c29987c2f5096c5e118 [id=467a7f6db73641ca85a1898b251938fb]) successfully deleted
I20240502 08:16:33.960286 24179 tablet_replica.cc:330] T abd595972fc144328c279a84ec7eeaa9 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:33.960191 24174 consensus_queue.cc:260] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:33.960460 24179 raft_consensus.cc:2229] T abd595972fc144328c279a84ec7eeaa9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:33.960613 24179 raft_consensus.cc:2258] T abd595972fc144328c279a84ec7eeaa9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:33.961258 24179 ts_tablet_manager.cc:1875] T abd595972fc144328c279a84ec7eeaa9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:33.963688 24174 ts_tablet_manager.cc:1398] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:16:33.963985 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_017d876815e94b87b8c60e535aa06ad5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:33.964608 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_017d876815e94b87b8c60e535aa06ad5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:33.966221 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534e3bceaa5c4c77abaf288ea08287ff" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:33.966403 23788 raft_consensus.cc:3037] T 534e3bceaa5c4c77abaf288ea08287ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:33.966660 24190 leader_election.cc:288] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:33.967924 24173 tablet_bootstrap.cc:492] T 0cf60a6da6f94525b02bdbb9e53192db P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:33.968101 24173 ts_tablet_manager.cc:1367] T 0cf60a6da6f94525b02bdbb9e53192db P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.077s user 0.004s sys 0.003s
I20240502 08:16:33.967973 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_95acc6e36a0b448eb3fd6d5df86e2de4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:33.968453 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95acc6e36a0b448eb3fd6d5df86e2de4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:33.969292 23788 raft_consensus.cc:2454] T 534e3bceaa5c4c77abaf288ea08287ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:33.969204 24173 raft_consensus.cc:348] T 0cf60a6da6f94525b02bdbb9e53192db P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.969730 23993 leader_election.cc:302] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:33.970026 24190 raft_consensus.cc:2781] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:33.970525 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 67bc4942db6140a3b7984318a78f7ad3 (table default.Table_d351b6717f35441cb5d9cf27f5c1087c [id=7338537cdd4249359d2af60930089412]) successfully deleted
I20240502 08:16:33.970965 24181 tablet_replica.cc:330] T 6c228645fe0b4bc89643178fff7b28a3 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:33.971220 24181 raft_consensus.cc:2229] T 6c228645fe0b4bc89643178fff7b28a3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:33.971415 24181 raft_consensus.cc:2258] T 6c228645fe0b4bc89643178fff7b28a3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:33.969436 24173 raft_consensus.cc:374] T 0cf60a6da6f94525b02bdbb9e53192db P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:33.971675 24173 raft_consensus.cc:729] T 0cf60a6da6f94525b02bdbb9e53192db P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:33.971783 24180 tablet_replica.cc:330] T 534e3bceaa5c4c77abaf288ea08287ff P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:33.972013 24180 raft_consensus.cc:2229] T 534e3bceaa5c4c77abaf288ea08287ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:33.972184 24180 raft_consensus.cc:2258] T 534e3bceaa5c4c77abaf288ea08287ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:33.972373 23610 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 8fc6ef91aca24cba91b8521cf467c78d
I20240502 08:16:33.972213 24173 consensus_queue.cc:260] T 0cf60a6da6f94525b02bdbb9e53192db P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.972651 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2c03990935e343a8bfca754e83a25ccf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:33.973104 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c03990935e343a8bfca754e83a25ccf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:33.973073 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_658ef73f4b084866b43dd4bfc8b337c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:33.973531 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_658ef73f4b084866b43dd4bfc8b337c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:33.974073 24173 ts_tablet_manager.cc:1398] T 0cf60a6da6f94525b02bdbb9e53192db P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:16:33.976003 24173 tablet_bootstrap.cc:492] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:33.977481 24173 tablet_bootstrap.cc:654] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:33.977979 24181 ts_tablet_manager.cc:1875] T 6c228645fe0b4bc89643178fff7b28a3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:33.984383 24173 tablet_bootstrap.cc:492] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:33.985589 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c9fdfaaf57304983858c5afcf7b28ef6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:33.986197 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9fdfaaf57304983858c5afcf7b28ef6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:33.986423 24190 raft_consensus.cc:686] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:33.986855 24190 consensus_queue.cc:237] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.989079 24179 ts_tablet_manager.cc:1888] T abd595972fc144328c279a84ec7eeaa9 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:33.989244 24179 log.cc:1196] T abd595972fc144328c279a84ec7eeaa9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/abd595972fc144328c279a84ec7eeaa9
I20240502 08:16:33.992759 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet 67bc4942db6140a3b7984318a78f7ad3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c08a89a72894f0b96d7863fd71edfd0) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:33.992795 24172 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } } }
I20240502 08:16:33.993036 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet 534e3bceaa5c4c77abaf288ea08287ff with delete_type TABLET_DATA_DELETED (replaced by tablet 8791834597254e7c9f730e03568c0748) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:33.993098 24172 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:33.993270 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c228645fe0b4bc89643178fff7b28a3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c8a832a2e6844ff8481b1b66f30a3b2) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:33.993427 24173 ts_tablet_manager.cc:1367] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240502 08:16:33.994189 24180 ts_tablet_manager.cc:1875] T 534e3bceaa5c4c77abaf288ea08287ff P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:33.994961 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf60a6da6f94525b02bdbb9e53192db with delete_type TABLET_DATA_DELETED (replaced by tablet a2d10db950314f3db1bfa352b9d3a759) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:33.996243 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet 67bc4942db6140a3b7984318a78f7ad3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c08a89a72894f0b96d7863fd71edfd0) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:33.996407 24179 ts_tablet_manager.cc:1909] T abd595972fc144328c279a84ec7eeaa9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:33.996464 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c8a832a2e6844ff8481b1b66f30a3b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:33 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:33.996661 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c228645fe0b4bc89643178fff7b28a3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c8a832a2e6844ff8481b1b66f30a3b2) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:33.997467 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet abd595972fc144328c279a84ec7eeaa9 (table default.Table_2e0ee58bc7db40da88a1700e3830444d [id=1e590d80e3e34e1aac8a0c7908a9d813]) successfully deleted
W20240502 08:16:33.997633 23996 leader_election.cc:341] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Not found: Tablet not found: 534e3bceaa5c4c77abaf288ea08287ff
W20240502 08:16:33.998239 23996 leader_election.cc:341] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Not found: Tablet not found: 534e3bceaa5c4c77abaf288ea08287ff
I20240502 08:16:33.998358 23609 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet a2d10db950314f3db1bfa352b9d3a759 (table default.Table_0112e7ea83b84f04a2e6038f2179b7c6 [id=40060891c6d44d559c37891803b42478]). Expected version 2 got 0
I20240502 08:16:33.998538 23609 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 9c08a89a72894f0b96d7863fd71edfd0 (table default.Table_d351b6717f35441cb5d9cf27f5c1087c [id=7338537cdd4249359d2af60930089412]). Expected version 2 got 0
I20240502 08:16:33.999044 24173 raft_consensus.cc:348] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:33.999225 24173 raft_consensus.cc:374] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:33.999297 24173 raft_consensus.cc:729] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:33.999771 24173 consensus_queue.cc:260] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:34.000257 24173 ts_tablet_manager.cc:1398] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240502 08:16:34.000531 24173 tablet_bootstrap.cc:492] T a2d10db950314f3db1bfa352b9d3a759 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:34.001518 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 359eee8b86af469b94ab5e5323b7972c
I20240502 08:16:34.002257 24175 tablet_bootstrap.cc:492] T a2d10db950314f3db1bfa352b9d3a759 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:34.002321 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 359eee8b86af469b94ab5e5323b7972c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:33 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.002431 24175 ts_tablet_manager.cc:1367] T a2d10db950314f3db1bfa352b9d3a759 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.107s user 0.014s sys 0.006s
I20240502 08:16:34.002609 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 359eee8b86af469b94ab5e5323b7972c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:33 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.003087 24175 raft_consensus.cc:348] T a2d10db950314f3db1bfa352b9d3a759 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:34.003268 24175 raft_consensus.cc:374] T a2d10db950314f3db1bfa352b9d3a759 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.003337 24175 raft_consensus.cc:729] T a2d10db950314f3db1bfa352b9d3a759 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.003561 24175 consensus_queue.cc:260] T a2d10db950314f3db1bfa352b9d3a759 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:34.003669 23905 tablet_service.cc:1467] Processing CreateTablet for tablet bf7678dbc7e641dfa259d36101d0dd5a (DEFAULT_TABLE table=default.Table_c60e3fa5881347d09ba464e544d7cbf8 [id=fac14487dd8840d0be3dcce182527117]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.004093 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf7678dbc7e641dfa259d36101d0dd5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.005133 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 315263b66700471b982072b335eb70c4 (DEFAULT_TABLE table=default.Table_4713741287e34e5d986ffb272ed6291d [id=38e955d6bd644d24a7c45172ab60282b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.005288 23767 tablet_service.cc:1467] Processing CreateTablet for tablet bf7678dbc7e641dfa259d36101d0dd5a (DEFAULT_TABLE table=default.Table_c60e3fa5881347d09ba464e544d7cbf8 [id=fac14487dd8840d0be3dcce182527117]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.005475 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 315263b66700471b982072b335eb70c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.006304 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 315263b66700471b982072b335eb70c4 (DEFAULT_TABLE table=default.Table_4713741287e34e5d986ffb272ed6291d [id=38e955d6bd644d24a7c45172ab60282b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.006526 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet 359eee8b86af469b94ab5e5323b7972c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:33 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.006705 24173 tablet_bootstrap.cc:654] T a2d10db950314f3db1bfa352b9d3a759 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.008255 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf7678dbc7e641dfa259d36101d0dd5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.008765 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 315263b66700471b982072b335eb70c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.008924 24040 tablet_service.cc:1467] Processing CreateTablet for tablet bf7678dbc7e641dfa259d36101d0dd5a (DEFAULT_TABLE table=default.Table_c60e3fa5881347d09ba464e544d7cbf8 [id=fac14487dd8840d0be3dcce182527117]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.009320 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf7678dbc7e641dfa259d36101d0dd5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.009421 23764 tablet_service.cc:1467] Processing CreateTablet for tablet c9f33b62a4da4634b5d1bc40bdd39b15 (DEFAULT_TABLE table=default.Table_b01c340249ea483683e385a7fa964a51 [id=ee0e1cf16cb34aa98699970438cc36ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.009794 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f33b62a4da4634b5d1bc40bdd39b15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.012410 23904 tablet_service.cc:1467] Processing CreateTablet for tablet c9f33b62a4da4634b5d1bc40bdd39b15 (DEFAULT_TABLE table=default.Table_b01c340249ea483683e385a7fa964a51 [id=ee0e1cf16cb34aa98699970438cc36ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.012718 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f33b62a4da4634b5d1bc40bdd39b15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.013852 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 315263b66700471b982072b335eb70c4 (DEFAULT_TABLE table=default.Table_4713741287e34e5d986ffb272ed6291d [id=38e955d6bd644d24a7c45172ab60282b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.014225 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 315263b66700471b982072b335eb70c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.016623 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 840aba1388ae4ff78fbb869d87dc1bf1 (DEFAULT_TABLE table=default.Table_b8ba42e1326143be948727bdcaa603f5 [id=20e5e7a21d9f4439a3c3723a0fdfd264]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.016932 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840aba1388ae4ff78fbb869d87dc1bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.017802 24039 tablet_service.cc:1467] Processing CreateTablet for tablet c9f33b62a4da4634b5d1bc40bdd39b15 (DEFAULT_TABLE table=default.Table_b01c340249ea483683e385a7fa964a51 [id=ee0e1cf16cb34aa98699970438cc36ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.017855 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 840aba1388ae4ff78fbb869d87dc1bf1 (DEFAULT_TABLE table=default.Table_b8ba42e1326143be948727bdcaa603f5 [id=20e5e7a21d9f4439a3c3723a0fdfd264]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.018126 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f33b62a4da4634b5d1bc40bdd39b15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.018193 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840aba1388ae4ff78fbb869d87dc1bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.022562 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 359eee8b86af469b94ab5e5323b7972c (DEFAULT_TABLE table=default.Table_5485d4b9db4f416691aea5c4f94ba412 [id=1fedc678a02342729dda973cb255bf14]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.022884 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 359eee8b86af469b94ab5e5323b7972c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.026098 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_4737e2cfe1834ff9abb44d634c3b2f26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.026638 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4737e2cfe1834ff9abb44d634c3b2f26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.027680 23604 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet a2d10db950314f3db1bfa352b9d3a759 (table default.Table_0112e7ea83b84f04a2e6038f2179b7c6 [id=40060891c6d44d559c37891803b42478]). Expected version 2 got 0
I20240502 08:16:34.028936 24175 ts_tablet_manager.cc:1398] T a2d10db950314f3db1bfa352b9d3a759 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240502 08:16:34.029829 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_39d3e9a8fea548abadf2eef7723684f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.030342 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39d3e9a8fea548abadf2eef7723684f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.030432 24180 ts_tablet_manager.cc:1888] T 534e3bceaa5c4c77abaf288ea08287ff P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.030452 23605 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet a2d10db950314f3db1bfa352b9d3a759 (table default.Table_0112e7ea83b84f04a2e6038f2179b7c6 [id=40060891c6d44d559c37891803b42478]). Expected version 2 got 0
I20240502 08:16:34.030581 24180 log.cc:1196] T 534e3bceaa5c4c77abaf288ea08287ff P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/534e3bceaa5c4c77abaf288ea08287ff
I20240502 08:16:34.030936 24180 ts_tablet_manager.cc:1909] T 534e3bceaa5c4c77abaf288ea08287ff P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:34.031896 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 534e3bceaa5c4c77abaf288ea08287ff (table default.Table_232703fab6964c29987c2f5096c5e118 [id=467a7f6db73641ca85a1898b251938fb]) successfully deleted
I20240502 08:16:34.032447 24173 tablet_bootstrap.cc:492] T a2d10db950314f3db1bfa352b9d3a759 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:34.032614 24173 ts_tablet_manager.cc:1367] T a2d10db950314f3db1bfa352b9d3a759 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.001s
I20240502 08:16:34.033372 24173 raft_consensus.cc:348] T a2d10db950314f3db1bfa352b9d3a759 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:34.033598 24173 raft_consensus.cc:374] T a2d10db950314f3db1bfa352b9d3a759 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.033674 24173 raft_consensus.cc:729] T a2d10db950314f3db1bfa352b9d3a759 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.034480 24175 tablet_bootstrap.cc:492] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:34.034869 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 840aba1388ae4ff78fbb869d87dc1bf1 (DEFAULT_TABLE table=default.Table_b8ba42e1326143be948727bdcaa603f5 [id=20e5e7a21d9f4439a3c3723a0fdfd264]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.036682 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e39b1ebf80484278bec29e25b57ba585"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:34.036945 24180 tablet_replica.cc:330] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:34.037063 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840aba1388ae4ff78fbb869d87dc1bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.037122 24180 raft_consensus.cc:2229] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
W20240502 08:16:34.037144 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e39b1ebf80484278bec29e25b57ba585 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.037494 24180 pending_rounds.cc:62] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:16:34.037650 24180 pending_rounds.cc:68] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156403420774400 op_type: NO_OP noop_request { }
I20240502 08:16:34.037850 24180 raft_consensus.cc:2866] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:34.037993 24180 raft_consensus.cc:2258] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.038863 24175 tablet_bootstrap.cc:654] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.039202 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6d48e2ca21a2409fae7c972ec53adbf5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.039685 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d48e2ca21a2409fae7c972ec53adbf5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.040302 24180 ts_tablet_manager.cc:1875] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:34.040572 23597 catalog_manager.cc:4797] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 534e3bceaa5c4c77abaf288ea08287ff because the tablet was not found. No further retry: Not found: Tablet not found: 534e3bceaa5c4c77abaf288ea08287ff
I20240502 08:16:34.040877 24179 tablet_replica.cc:330] T 6c228645fe0b4bc89643178fff7b28a3 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:34.041041 24179 raft_consensus.cc:2229] T 6c228645fe0b4bc89643178fff7b28a3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.041184 24179 raft_consensus.cc:2258] T 6c228645fe0b4bc89643178fff7b28a3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.041813 24179 ts_tablet_manager.cc:1875] T 6c228645fe0b4bc89643178fff7b28a3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.042376 24174 tablet_bootstrap.cc:492] T c9f33b62a4da4634b5d1bc40bdd39b15 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:34.045949 24181 ts_tablet_manager.cc:1888] T 6c228645fe0b4bc89643178fff7b28a3 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.046080 24181 log.cc:1196] T 6c228645fe0b4bc89643178fff7b28a3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/6c228645fe0b4bc89643178fff7b28a3
I20240502 08:16:34.047008 24174 tablet_bootstrap.cc:654] T c9f33b62a4da4634b5d1bc40bdd39b15 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.035408 24173 consensus_queue.cc:260] T a2d10db950314f3db1bfa352b9d3a759 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:34.048506 24173 ts_tablet_manager.cc:1398] T a2d10db950314f3db1bfa352b9d3a759 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240502 08:16:34.050009 24181 ts_tablet_manager.cc:1909] T 6c228645fe0b4bc89643178fff7b28a3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:34.050253 24173 tablet_bootstrap.cc:492] T 9c08a89a72894f0b96d7863fd71edfd0 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:34.051234 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 6c228645fe0b4bc89643178fff7b28a3 (table default.Table_ffa5fe5beacd4345b72e20da3bb7cb76 [id=b24212eeb7784959b5fbcd68bcafa0d7]) successfully deleted
I20240502 08:16:34.051621 24181 tablet_replica.cc:330] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:34.052852 24173 tablet_bootstrap.cc:654] T 9c08a89a72894f0b96d7863fd71edfd0 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.053997 24180 ts_tablet_manager.cc:1888] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:34.054155 24180 log.cc:1196] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/abd595972fc144328c279a84ec7eeaa9
I20240502 08:16:34.054553 24180 ts_tablet_manager.cc:1909] T abd595972fc144328c279a84ec7eeaa9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:34.055475 24179 ts_tablet_manager.cc:1888] T 6c228645fe0b4bc89643178fff7b28a3 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.055621 24179 log.cc:1196] T 6c228645fe0b4bc89643178fff7b28a3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/6c228645fe0b4bc89643178fff7b28a3
I20240502 08:16:34.055634 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet abd595972fc144328c279a84ec7eeaa9 (table default.Table_2e0ee58bc7db40da88a1700e3830444d [id=1e590d80e3e34e1aac8a0c7908a9d813]) successfully deleted
I20240502 08:16:34.056039 24179 ts_tablet_manager.cc:1909] T 6c228645fe0b4bc89643178fff7b28a3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:34.056753 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_afaca5b9d2774c0baff53ca022b9c4dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:34.057193 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 6c228645fe0b4bc89643178fff7b28a3 (table default.Table_ffa5fe5beacd4345b72e20da3bb7cb76 [id=b24212eeb7784959b5fbcd68bcafa0d7]) successfully deleted
W20240502 08:16:34.057756 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afaca5b9d2774c0baff53ca022b9c4dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.058310 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 359eee8b86af469b94ab5e5323b7972c (DEFAULT_TABLE table=default.Table_5485d4b9db4f416691aea5c4f94ba412 [id=1fedc678a02342729dda973cb255bf14]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.058672 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 359eee8b86af469b94ab5e5323b7972c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.058710 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 68bdb761ff36426c96949e97279e2756 (DEFAULT_TABLE table=default.Table_545327c2458643f4956f4cc391f76a50 [id=eca4080e8ebf46e9b58c85ee1d8675ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.059039 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68bdb761ff36426c96949e97279e2756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.059010 23905 tablet_service.cc:1467] Processing CreateTablet for tablet bce9392a8ad64004ba6ce9a236976cd6 (DEFAULT_TABLE table=default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.059360 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce9392a8ad64004ba6ce9a236976cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.061847 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 68bdb761ff36426c96949e97279e2756 (DEFAULT_TABLE table=default.Table_545327c2458643f4956f4cc391f76a50 [id=eca4080e8ebf46e9b58c85ee1d8675ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.062184 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68bdb761ff36426c96949e97279e2756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.062876 24174 tablet_bootstrap.cc:492] T c9f33b62a4da4634b5d1bc40bdd39b15 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:34.063097 24174 ts_tablet_manager.cc:1367] T c9f33b62a4da4634b5d1bc40bdd39b15 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.003s
I20240502 08:16:34.063892 24174 raft_consensus.cc:348] T c9f33b62a4da4634b5d1bc40bdd39b15 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.064312 24174 raft_consensus.cc:374] T c9f33b62a4da4634b5d1bc40bdd39b15 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.064430 24174 raft_consensus.cc:729] T c9f33b62a4da4634b5d1bc40bdd39b15 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.065169 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 68bdb761ff36426c96949e97279e2756 (DEFAULT_TABLE table=default.Table_545327c2458643f4956f4cc391f76a50 [id=eca4080e8ebf46e9b58c85ee1d8675ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.065506 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68bdb761ff36426c96949e97279e2756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.064711 24174 consensus_queue.cc:260] T c9f33b62a4da4634b5d1bc40bdd39b15 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.066720 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 359eee8b86af469b94ab5e5323b7972c (DEFAULT_TABLE table=default.Table_5485d4b9db4f416691aea5c4f94ba412 [id=1fedc678a02342729dda973cb255bf14]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.067095 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 359eee8b86af469b94ab5e5323b7972c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.070919 24038 tablet_service.cc:1467] Processing CreateTablet for tablet bce9392a8ad64004ba6ce9a236976cd6 (DEFAULT_TABLE table=default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.071312 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce9392a8ad64004ba6ce9a236976cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.072984 24175 tablet_bootstrap.cc:492] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:34.073143 24175 ts_tablet_manager.cc:1367] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.001s
I20240502 08:16:34.073052 24180 tablet_replica.cc:330] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:34.073868 24180 raft_consensus.cc:2229] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.074059 24180 raft_consensus.cc:2258] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.074172 24175 raft_consensus.cc:348] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:34.074435 24175 raft_consensus.cc:374] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.074517 24175 raft_consensus.cc:729] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.074822 24175 consensus_queue.cc:260] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:34.076300 24180 ts_tablet_manager.cc:1875] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.079849 23767 tablet_service.cc:1467] Processing CreateTablet for tablet bce9392a8ad64004ba6ce9a236976cd6 (DEFAULT_TABLE table=default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.080410 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce9392a8ad64004ba6ce9a236976cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.081593 24175 ts_tablet_manager.cc:1398] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 08:16:34.083348 24174 ts_tablet_manager.cc:1398] T c9f33b62a4da4634b5d1bc40bdd39b15 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240502 08:16:34.086143 24175 tablet_bootstrap.cc:492] T 9c08a89a72894f0b96d7863fd71edfd0 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:34.086517 24174 tablet_bootstrap.cc:492] T bf7678dbc7e641dfa259d36101d0dd5a P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:34.087680 24175 tablet_bootstrap.cc:654] T 9c08a89a72894f0b96d7863fd71edfd0 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.088645 24181 raft_consensus.cc:2229] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:34.090497 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_017d876815e94b87b8c60e535aa06ad5" } modify_external_catalogs: true
I20240502 08:16:34.090780 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_017d876815e94b87b8c60e535aa06ad5" } modify_external_catalogs: true
I20240502 08:16:34.091847 24180 ts_tablet_manager.cc:1888] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.091976 24180 log.cc:1196] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/0cf60a6da6f94525b02bdbb9e53192db
I20240502 08:16:34.092154 24174 tablet_bootstrap.cc:654] T bf7678dbc7e641dfa259d36101d0dd5a P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.092409 24180 ts_tablet_manager.cc:1909] T 0cf60a6da6f94525b02bdbb9e53192db P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
W20240502 08:16:34.092551 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 359eee8b86af469b94ab5e5323b7972c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 359eee8b86af469b94ab5e5323b7972c already in progress: creating tablet
I20240502 08:16:34.093468 24180 tablet_replica.cc:330] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:34.093683 24180 raft_consensus.cc:2229] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.093901 24180 raft_consensus.cc:2258] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.094697 24180 ts_tablet_manager.cc:1875] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.095438 24181 pending_rounds.cc:62] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5: Trying to abort 1 pending ops.
I20240502 08:16:34.095606 24181 pending_rounds.cc:68] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156404175642624 op_type: NO_OP noop_request { }
I20240502 08:16:34.095841 24181 raft_consensus.cc:2866] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:34.093498 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 0cf60a6da6f94525b02bdbb9e53192db (table default.Table_0112e7ea83b84f04a2e6038f2179b7c6 [id=40060891c6d44d559c37891803b42478]) successfully deleted
I20240502 08:16:34.095961 24181 raft_consensus.cc:2258] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.097174 24181 ts_tablet_manager.cc:1875] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.097772 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c8a832a2e6844ff8481b1b66f30a3b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:33 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.098524 24179 tablet_replica.cc:330] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:34.098709 24179 raft_consensus.cc:2229] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.098843 24179 raft_consensus.cc:2258] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.098978 24173 tablet_bootstrap.cc:492] T 9c08a89a72894f0b96d7863fd71edfd0 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:34.099149 24173 ts_tablet_manager.cc:1367] T 9c08a89a72894f0b96d7863fd71edfd0 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.001s
I20240502 08:16:34.099645 24175 tablet_bootstrap.cc:492] T 9c08a89a72894f0b96d7863fd71edfd0 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:34.099809 24175 ts_tablet_manager.cc:1367] T 9c08a89a72894f0b96d7863fd71edfd0 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20240502 08:16:34.099916 24173 raft_consensus.cc:348] T 9c08a89a72894f0b96d7863fd71edfd0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:34.100131 24173 raft_consensus.cc:374] T 9c08a89a72894f0b96d7863fd71edfd0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.100248 24173 raft_consensus.cc:729] T 9c08a89a72894f0b96d7863fd71edfd0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.100502 24173 consensus_queue.cc:260] T 9c08a89a72894f0b96d7863fd71edfd0 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:34.100802 24175 raft_consensus.cc:348] T 9c08a89a72894f0b96d7863fd71edfd0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:34.101050 24175 raft_consensus.cc:374] T 9c08a89a72894f0b96d7863fd71edfd0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.101097 24173 ts_tablet_manager.cc:1398] T 9c08a89a72894f0b96d7863fd71edfd0 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:34.101236 24175 raft_consensus.cc:729] T 9c08a89a72894f0b96d7863fd71edfd0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.101382 24173 tablet_bootstrap.cc:492] T c9f33b62a4da4634b5d1bc40bdd39b15 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:34.101595 24175 consensus_queue.cc:260] T 9c08a89a72894f0b96d7863fd71edfd0 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:34.102085 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf60a6da6f94525b02bdbb9e53192db with delete_type TABLET_DATA_DELETED (replaced by tablet a2d10db950314f3db1bfa352b9d3a759) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.103000 24173 tablet_bootstrap.cc:654] T c9f33b62a4da4634b5d1bc40bdd39b15 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.104844 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet a2d10db950314f3db1bfa352b9d3a759 (table default.Table_0112e7ea83b84f04a2e6038f2179b7c6 [id=40060891c6d44d559c37891803b42478]). Expected version 2 got 0
I20240502 08:16:34.105929 24179 ts_tablet_manager.cc:1875] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.107451 23604 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 9c08a89a72894f0b96d7863fd71edfd0 (table default.Table_d351b6717f35441cb5d9cf27f5c1087c [id=7338537cdd4249359d2af60930089412]). Expected version 2 got 0
I20240502 08:16:34.110102 24180 ts_tablet_manager.cc:1888] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.110265 24180 log.cc:1196] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/67bc4942db6140a3b7984318a78f7ad3
I20240502 08:16:34.110538 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 9c08a89a72894f0b96d7863fd71edfd0 (table default.Table_d351b6717f35441cb5d9cf27f5c1087c [id=7338537cdd4249359d2af60930089412]). Expected version 2 got 0
I20240502 08:16:34.110627 24180 ts_tablet_manager.cc:1909] T 67bc4942db6140a3b7984318a78f7ad3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:34.111150 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 8dc8158255194b868af917c33c44f0f5
I20240502 08:16:34.111574 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 67bc4942db6140a3b7984318a78f7ad3 (table default.Table_d351b6717f35441cb5d9cf27f5c1087c [id=7338537cdd4249359d2af60930089412]) successfully deleted
W20240502 08:16:34.111912 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 5c8a832a2e6844ff8481b1b66f30a3b2 because the tablet was not found. No further retry: Not found: Tablet not found: 5c8a832a2e6844ff8481b1b66f30a3b2
I20240502 08:16:34.111958 24175 ts_tablet_manager.cc:1398] T 9c08a89a72894f0b96d7863fd71edfd0 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240502 08:16:34.112175 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dc8158255194b868af917c33c44f0f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.112542 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dc8158255194b868af917c33c44f0f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.112875 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dc8158255194b868af917c33c44f0f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.113329 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 9c08a89a72894f0b96d7863fd71edfd0 (table default.Table_d351b6717f35441cb5d9cf27f5c1087c [id=7338537cdd4249359d2af60930089412]). Expected version 2 got 0
I20240502 08:16:34.114012 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 74aaea4b05e84f2ab31a0439f82016c1 (DEFAULT_TABLE table=default.Table_39d3e9a8fea548abadf2eef7723684f2 [id=495bc1caa3e44d56abf4269f906ef74d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.114575 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74aaea4b05e84f2ab31a0439f82016c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.115404 24179 ts_tablet_manager.cc:1888] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.115489 24180 tablet_replica.cc:330] T 6c228645fe0b4bc89643178fff7b28a3 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:34.115556 24179 log.cc:1196] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/5c8a832a2e6844ff8481b1b66f30a3b2
I20240502 08:16:34.115689 24180 raft_consensus.cc:2229] T 6c228645fe0b4bc89643178fff7b28a3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.115751 24175 tablet_bootstrap.cc:492] T c9f33b62a4da4634b5d1bc40bdd39b15 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:34.115895 24180 raft_consensus.cc:2258] T 6c228645fe0b4bc89643178fff7b28a3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.115962 24179 ts_tablet_manager.cc:1909] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:34.116469 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c8a832a2e6844ff8481b1b66f30a3b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:33 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.117396 24175 tablet_bootstrap.cc:654] T c9f33b62a4da4634b5d1bc40bdd39b15 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.117405 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 5c8a832a2e6844ff8481b1b66f30a3b2 (table default.Table_ffa5fe5beacd4345b72e20da3bb7cb76 [id=b24212eeb7784959b5fbcd68bcafa0d7]) successfully deleted
W20240502 08:16:34.118141 23597 catalog_manager.cc:4797] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 8dc8158255194b868af917c33c44f0f5 because the tablet was not found. No further retry: Not found: Tablet not found: 8dc8158255194b868af917c33c44f0f5
I20240502 08:16:34.118237 24180 ts_tablet_manager.cc:1875] T 6c228645fe0b4bc89643178fff7b28a3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.120404 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 74aaea4b05e84f2ab31a0439f82016c1 (DEFAULT_TABLE table=default.Table_39d3e9a8fea548abadf2eef7723684f2 [id=495bc1caa3e44d56abf4269f906ef74d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.120962 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74aaea4b05e84f2ab31a0439f82016c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.121986 24181 ts_tablet_manager.cc:1888] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:34.122177 24181 log.cc:1196] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/534e3bceaa5c4c77abaf288ea08287ff
I20240502 08:16:34.122723 24181 ts_tablet_manager.cc:1909] T 534e3bceaa5c4c77abaf288ea08287ff P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:34.126065 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 050c16739efb4037a4adc78a2ffd3cba (DEFAULT_TABLE table=default.Table_e39b1ebf80484278bec29e25b57ba585 [id=ba62a673079d424992daa0f6b0c5d6b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.126132 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e39b1ebf80484278bec29e25b57ba585" } new_table_name: "default.Table_e0bf76fa2b2d451eb0102e9897669b6f" modify_external_catalogs: true
I20240502 08:16:34.126355 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 74aaea4b05e84f2ab31a0439f82016c1 (DEFAULT_TABLE table=default.Table_39d3e9a8fea548abadf2eef7723684f2 [id=495bc1caa3e44d56abf4269f906ef74d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.126430 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 050c16739efb4037a4adc78a2ffd3cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.126725 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74aaea4b05e84f2ab31a0439f82016c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.127435 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_545327c2458643f4956f4cc391f76a50" } new_table_name: "default.Table_50b02617337c4151bdc0ba38abd68144" modify_external_catalogs: true
I20240502 08:16:34.128588 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b8ba42e1326143be948727bdcaa603f5" } new_table_name: "default.Table_1d5afdec101c4b57864bc6d8eeefd20f" modify_external_catalogs: true
I20240502 08:16:34.130566 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4737e2cfe1834ff9abb44d634c3b2f26" } new_table_name: "default.Table_f44fe9285e9e4ee5968523bbc6adece8" modify_external_catalogs: true
I20240502 08:16:34.132838 24181 tablet_replica.cc:330] T abd595972fc144328c279a84ec7eeaa9 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:34.132814 24173 tablet_bootstrap.cc:492] T c9f33b62a4da4634b5d1bc40bdd39b15 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:34.133062 24181 raft_consensus.cc:2229] T abd595972fc144328c279a84ec7eeaa9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.133066 24173 ts_tablet_manager.cc:1367] T c9f33b62a4da4634b5d1bc40bdd39b15 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.000s
I20240502 08:16:34.133236 24181 raft_consensus.cc:2258] T abd595972fc144328c279a84ec7eeaa9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.133955 24181 ts_tablet_manager.cc:1875] T abd595972fc144328c279a84ec7eeaa9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.134567 24173 raft_consensus.cc:348] T c9f33b62a4da4634b5d1bc40bdd39b15 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.134811 24173 raft_consensus.cc:374] T c9f33b62a4da4634b5d1bc40bdd39b15 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.134918 24173 raft_consensus.cc:729] T c9f33b62a4da4634b5d1bc40bdd39b15 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.135200 24173 consensus_queue.cc:260] T c9f33b62a4da4634b5d1bc40bdd39b15 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.136159 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0112e7ea83b84f04a2e6038f2179b7c6" } modify_external_catalogs: true
I20240502 08:16:34.136310 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0112e7ea83b84f04a2e6038f2179b7c6" } modify_external_catalogs: true
I20240502 08:16:34.137483 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 534e3bceaa5c4c77abaf288ea08287ff (table default.Table_232703fab6964c29987c2f5096c5e118 [id=467a7f6db73641ca85a1898b251938fb]) successfully deleted
I20240502 08:16:34.138654 24173 ts_tablet_manager.cc:1398] T c9f33b62a4da4634b5d1bc40bdd39b15 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20240502 08:16:34.138895 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 68bdb761ff36426c96949e97279e2756 consensus state has no leader
I20240502 08:16:34.138931 24173 tablet_bootstrap.cc:492] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:34.140348 24173 tablet_bootstrap.cc:654] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:34.140933 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 050c16739efb4037a4adc78a2ffd3cba consensus state has no leader
W20240502 08:16:34.143115 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 840aba1388ae4ff78fbb869d87dc1bf1 consensus state has no leader
W20240502 08:16:34.143429 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5def9306dfba4d019d3acece81df2f86 consensus state has no leader
I20240502 08:16:34.144049 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 050c16739efb4037a4adc78a2ffd3cba (DEFAULT_TABLE table=default.Table_e39b1ebf80484278bec29e25b57ba585 [id=ba62a673079d424992daa0f6b0c5d6b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.144500 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 050c16739efb4037a4adc78a2ffd3cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.145494 24180 ts_tablet_manager.cc:1888] T 6c228645fe0b4bc89643178fff7b28a3 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.149019 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet a2d10db950314f3db1bfa352b9d3a759
I20240502 08:16:34.149397 24180 log.cc:1196] T 6c228645fe0b4bc89643178fff7b28a3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/6c228645fe0b4bc89643178fff7b28a3
I20240502 08:16:34.149534 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f44fe9285e9e4ee5968523bbc6adece8" } new_table_name: "default.Table_b142a1494f9942d6a7dd855cfbbc38f7" modify_external_catalogs: true
I20240502 08:16:34.149817 24180 ts_tablet_manager.cc:1909] T 6c228645fe0b4bc89643178fff7b28a3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:34.149992 23903 tablet_service.cc:1514] Processing DeleteTablet for tablet a2d10db950314f3db1bfa352b9d3a759 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.150516 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet a2d10db950314f3db1bfa352b9d3a759 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.150697 24179 tablet_replica.cc:330] T a2d10db950314f3db1bfa352b9d3a759 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:34.150826 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 6c228645fe0b4bc89643178fff7b28a3 (table default.Table_ffa5fe5beacd4345b72e20da3bb7cb76 [id=b24212eeb7784959b5fbcd68bcafa0d7]) successfully deleted
I20240502 08:16:34.150883 24179 raft_consensus.cc:2229] T a2d10db950314f3db1bfa352b9d3a759 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.151126 24179 raft_consensus.cc:2258] T a2d10db950314f3db1bfa352b9d3a759 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.151577 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet a2d10db950314f3db1bfa352b9d3a759 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.152789 24181 ts_tablet_manager.cc:1888] T abd595972fc144328c279a84ec7eeaa9 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.152906 24181 log.cc:1196] T abd595972fc144328c279a84ec7eeaa9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/abd595972fc144328c279a84ec7eeaa9
I20240502 08:16:34.153630 24179 ts_tablet_manager.cc:1875] T a2d10db950314f3db1bfa352b9d3a759 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:34.154196 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 359eee8b86af469b94ab5e5323b7972c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 359eee8b86af469b94ab5e5323b7972c already in progress: creating tablet
I20240502 08:16:34.154734 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 050c16739efb4037a4adc78a2ffd3cba (DEFAULT_TABLE table=default.Table_e0bf76fa2b2d451eb0102e9897669b6f [id=ba62a673079d424992daa0f6b0c5d6b9]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:34.155225 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 8dc8158255194b868af917c33c44f0f5 because the tablet was not found. No further retry: Not found: Tablet not found: 8dc8158255194b868af917c33c44f0f5
I20240502 08:16:34.155611 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 050c16739efb4037a4adc78a2ffd3cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.155661 24180 tablet_replica.cc:330] T a2d10db950314f3db1bfa352b9d3a759 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:34.155827 24180 raft_consensus.cc:2229] T a2d10db950314f3db1bfa352b9d3a759 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.155978 24180 raft_consensus.cc:2258] T a2d10db950314f3db1bfa352b9d3a759 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.157845 24175 tablet_bootstrap.cc:492] T c9f33b62a4da4634b5d1bc40bdd39b15 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:34.157948 24180 ts_tablet_manager.cc:1875] T a2d10db950314f3db1bfa352b9d3a759 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.158004 24175 ts_tablet_manager.cc:1367] T c9f33b62a4da4634b5d1bc40bdd39b15 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.042s user 0.000s sys 0.018s
I20240502 08:16:34.159325 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1d5afdec101c4b57864bc6d8eeefd20f" } modify_external_catalogs: true
I20240502 08:16:34.159440 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet c9f33b62a4da4634b5d1bc40bdd39b15 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:34.159464 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet bf7678dbc7e641dfa259d36101d0dd5a from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:34.159523 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1d5afdec101c4b57864bc6d8eeefd20f" } modify_external_catalogs: true
I20240502 08:16:34.161260 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 5def9306dfba4d019d3acece81df2f86 (DEFAULT_TABLE table=default.Table_f44fe9285e9e4ee5968523bbc6adece8 [id=b42e15a2350f4c9a8097ccb0cf1aaddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.161661 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5def9306dfba4d019d3acece81df2f86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.162930 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 5def9306dfba4d019d3acece81df2f86 (DEFAULT_TABLE table=default.Table_f44fe9285e9e4ee5968523bbc6adece8 [id=b42e15a2350f4c9a8097ccb0cf1aaddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.163358 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5def9306dfba4d019d3acece81df2f86. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:34.164069 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5def9306dfba4d019d3acece81df2f86 consensus state has no leader
I20240502 08:16:34.165494 24190 raft_consensus.cc:482] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:34.165733 24190 raft_consensus.cc:504] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:34.166393 24190 leader_election.cc:288] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:34.167387 23996 leader_election.cc:341] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Not found: Tablet not found: 5c8a832a2e6844ff8481b1b66f30a3b2
W20240502 08:16:34.167565 23993 leader_election.cc:341] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 5c8a832a2e6844ff8481b1b66f30a3b2
I20240502 08:16:34.167681 23993 leader_election.cc:302] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:34.168100 24190 raft_consensus.cc:2726] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:34.171811 24180 ts_tablet_manager.cc:1888] T a2d10db950314f3db1bfa352b9d3a759 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.171953 24180 log.cc:1196] T a2d10db950314f3db1bfa352b9d3a759 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/a2d10db950314f3db1bfa352b9d3a759
I20240502 08:16:34.172365 24180 ts_tablet_manager.cc:1909] T a2d10db950314f3db1bfa352b9d3a759 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:34.173242 24179 ts_tablet_manager.cc:1888] T a2d10db950314f3db1bfa352b9d3a759 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.173557 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_f8286fd5bd9f4a1ca39afc1e8b57eae2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.174221 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8286fd5bd9f4a1ca39afc1e8b57eae2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.175976 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet bf7678dbc7e641dfa259d36101d0dd5a
I20240502 08:16:34.176101 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet c9f33b62a4da4634b5d1bc40bdd39b15
I20240502 08:16:34.176155 24181 ts_tablet_manager.cc:1909] T abd595972fc144328c279a84ec7eeaa9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:34.176402 23601 catalog_manager.cc:6554] ReplaceTablet: tablet bf7678dbc7e641dfa259d36101d0dd5a deleted and replaced by tablet 476569457c3949c8a74007fcff8756f4
I20240502 08:16:34.176528 23609 catalog_manager.cc:6554] ReplaceTablet: tablet c9f33b62a4da4634b5d1bc40bdd39b15 deleted and replaced by tablet e17eec8f2c1a4190945b434794635b52
I20240502 08:16:34.177445 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 840aba1388ae4ff78fbb869d87dc1bf1
I20240502 08:16:34.177587 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet bf7678dbc7e641dfa259d36101d0dd5a with delete_type TABLET_DATA_DELETED (replaced by tablet 476569457c3949c8a74007fcff8756f4) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.177688 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet c9f33b62a4da4634b5d1bc40bdd39b15 with delete_type TABLET_DATA_DELETED (replaced by tablet e17eec8f2c1a4190945b434794635b52) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.178431 24181 tablet_replica.cc:330] T 0cf60a6da6f94525b02bdbb9e53192db P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:34.178522 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet abd595972fc144328c279a84ec7eeaa9 (table default.Table_2e0ee58bc7db40da88a1700e3830444d [id=1e590d80e3e34e1aac8a0c7908a9d813]) successfully deleted
I20240502 08:16:34.178642 24181 raft_consensus.cc:2229] T 0cf60a6da6f94525b02bdbb9e53192db P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240502 08:16:34.178742 23597 catalog_manager.cc:4797] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 67bc4942db6140a3b7984318a78f7ad3 because the tablet was not found. No further retry: Not found: Tablet not found: 67bc4942db6140a3b7984318a78f7ad3
I20240502 08:16:34.178798 24181 raft_consensus.cc:2258] T 0cf60a6da6f94525b02bdbb9e53192db P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.179508 23765 tablet_service.cc:1467] Processing CreateTablet for tablet e1cf5f66859d416baad2f767060197c8 (DEFAULT_TABLE table=default.Table_c9fdfaaf57304983858c5afcf7b28ef6 [id=c08bf22592064123bb9cab28935d81a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.179862 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1cf5f66859d416baad2f767060197c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.175740 24175 raft_consensus.cc:348] T c9f33b62a4da4634b5d1bc40bdd39b15 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.177322 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet c9f33b62a4da4634b5d1bc40bdd39b15 with delete_type TABLET_DATA_DELETED (replaced by tablet e17eec8f2c1a4190945b434794635b52) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.177172 23903 tablet_service.cc:1514] Processing DeleteTablet for tablet bf7678dbc7e641dfa259d36101d0dd5a with delete_type TABLET_DATA_DELETED (replaced by tablet 476569457c3949c8a74007fcff8756f4) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.180160 23902 tablet_service.cc:1467] Processing CreateTablet for tablet e1cf5f66859d416baad2f767060197c8 (DEFAULT_TABLE table=default.Table_c9fdfaaf57304983858c5afcf7b28ef6 [id=c08bf22592064123bb9cab28935d81a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.180370 24175 raft_consensus.cc:374] T c9f33b62a4da4634b5d1bc40bdd39b15 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.180449 24175 raft_consensus.cc:729] T c9f33b62a4da4634b5d1bc40bdd39b15 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.180541 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1cf5f66859d416baad2f767060197c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.180830 24175 consensus_queue.cc:260] T c9f33b62a4da4634b5d1bc40bdd39b15 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.181372 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 8dc8158255194b868af917c33c44f0f5 (DEFAULT_TABLE table=default.Table_017d876815e94b87b8c60e535aa06ad5 [id=307769d00cda4d679da90a24bd461dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.182199 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dc8158255194b868af917c33c44f0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.183367 24175 ts_tablet_manager.cc:1398] T c9f33b62a4da4634b5d1bc40bdd39b15 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.025s user 0.000s sys 0.002s
I20240502 08:16:34.184571 24175 tablet_bootstrap.cc:492] T 315263b66700471b982072b335eb70c4 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:34.184947 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 840aba1388ae4ff78fbb869d87dc1bf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.185276 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet 840aba1388ae4ff78fbb869d87dc1bf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.179888 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 6c0a4b216c7e474db1992a299ff8eeb4 (DEFAULT_TABLE table=default.Table_2c03990935e343a8bfca754e83a25ccf [id=a9cdc35f95ec47d9a41d375133749c88]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.185602 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c0a4b216c7e474db1992a299ff8eeb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.185642 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 6c0a4b216c7e474db1992a299ff8eeb4 (DEFAULT_TABLE table=default.Table_2c03990935e343a8bfca754e83a25ccf [id=a9cdc35f95ec47d9a41d375133749c88]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.185951 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c0a4b216c7e474db1992a299ff8eeb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.186187 24175 tablet_bootstrap.cc:654] T 315263b66700471b982072b335eb70c4 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.187233 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet bf7678dbc7e641dfa259d36101d0dd5a with delete_type TABLET_DATA_DELETED (replaced by tablet 476569457c3949c8a74007fcff8756f4) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.187841 24174 tablet_bootstrap.cc:492] T bf7678dbc7e641dfa259d36101d0dd5a P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:34.187980 24174 ts_tablet_manager.cc:1367] T bf7678dbc7e641dfa259d36101d0dd5a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.102s user 0.004s sys 0.013s
I20240502 08:16:34.188759 24174 raft_consensus.cc:348] T bf7678dbc7e641dfa259d36101d0dd5a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.188987 24174 raft_consensus.cc:374] T bf7678dbc7e641dfa259d36101d0dd5a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.189079 24174 raft_consensus.cc:729] T bf7678dbc7e641dfa259d36101d0dd5a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.189491 24038 tablet_service.cc:1467] Processing CreateTablet for tablet e1cf5f66859d416baad2f767060197c8 (DEFAULT_TABLE table=default.Table_c9fdfaaf57304983858c5afcf7b28ef6 [id=c08bf22592064123bb9cab28935d81a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.189535 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 5def9306dfba4d019d3acece81df2f86 (DEFAULT_TABLE table=default.Table_f44fe9285e9e4ee5968523bbc6adece8 [id=b42e15a2350f4c9a8097ccb0cf1aaddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.189864 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 6c0a4b216c7e474db1992a299ff8eeb4 (DEFAULT_TABLE table=default.Table_2c03990935e343a8bfca754e83a25ccf [id=a9cdc35f95ec47d9a41d375133749c88]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.189953 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1cf5f66859d416baad2f767060197c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.189867 24174 consensus_queue.cc:260] T bf7678dbc7e641dfa259d36101d0dd5a P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.189918 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 7a10f86a0f3f4385bef39eda14d4fa9f (DEFAULT_TABLE table=default.Table_95acc6e36a0b448eb3fd6d5df86e2de4 [id=1328b5d30ef24f98bf00c4429383ce92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.190320 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c0a4b216c7e474db1992a299ff8eeb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.190407 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a10f86a0f3f4385bef39eda14d4fa9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.190389 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 8dc8158255194b868af917c33c44f0f5 (DEFAULT_TABLE table=default.Table_017d876815e94b87b8c60e535aa06ad5 [id=307769d00cda4d679da90a24bd461dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.190830 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dc8158255194b868af917c33c44f0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.191560 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 840aba1388ae4ff78fbb869d87dc1bf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.191653 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5def9306dfba4d019d3acece81df2f86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.193392 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 7a10f86a0f3f4385bef39eda14d4fa9f (DEFAULT_TABLE table=default.Table_95acc6e36a0b448eb3fd6d5df86e2de4 [id=1328b5d30ef24f98bf00c4429383ce92]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:34.194056 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 359eee8b86af469b94ab5e5323b7972c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 359eee8b86af469b94ab5e5323b7972c already in progress: creating tablet
I20240502 08:16:34.194401 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet 359eee8b86af469b94ab5e5323b7972c from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:34.194587 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a10f86a0f3f4385bef39eda14d4fa9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.197717 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet c9f33b62a4da4634b5d1bc40bdd39b15 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:34.197821 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet 359eee8b86af469b94ab5e5323b7972c from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:34.198499 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b2dc80505cd440f591086162a662e92d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.199081 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2dc80505cd440f591086162a662e92d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.202167 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6c583799e352452a87de6faa19d88490"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.202685 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c583799e352452a87de6faa19d88490 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.202601 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a0ca472d0e4748e6995f3b55e306e483"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.203074 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0ca472d0e4748e6995f3b55e306e483 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.206724 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c60e3fa5881347d09ba464e544d7cbf8" } modify_external_catalogs: true
I20240502 08:16:34.206836 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 8dc8158255194b868af917c33c44f0f5 (DEFAULT_TABLE table=default.Table_017d876815e94b87b8c60e535aa06ad5 [id=307769d00cda4d679da90a24bd461dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.206908 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c60e3fa5881347d09ba464e544d7cbf8" } modify_external_catalogs: true
I20240502 08:16:34.207336 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dc8158255194b868af917c33c44f0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.180186 24179 log.cc:1196] T a2d10db950314f3db1bfa352b9d3a759 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/a2d10db950314f3db1bfa352b9d3a759
I20240502 08:16:34.212495 24179 ts_tablet_manager.cc:1909] T a2d10db950314f3db1bfa352b9d3a759 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:34.213589 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet a2d10db950314f3db1bfa352b9d3a759 (table default.Table_0112e7ea83b84f04a2e6038f2179b7c6 [id=40060891c6d44d559c37891803b42478]) successfully deleted
I20240502 08:16:34.213728 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_68bf91a14ffa42cdbb10a1d7ca56cb08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:34.214030 24179 tablet_replica.cc:330] T c9f33b62a4da4634b5d1bc40bdd39b15 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:34.214232 24179 raft_consensus.cc:2229] T c9f33b62a4da4634b5d1bc40bdd39b15 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
W20240502 08:16:34.214334 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68bf91a14ffa42cdbb10a1d7ca56cb08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.214386 24179 raft_consensus.cc:2258] T c9f33b62a4da4634b5d1bc40bdd39b15 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.215391 24181 ts_tablet_manager.cc:1875] T 0cf60a6da6f94525b02bdbb9e53192db P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.217476 24179 ts_tablet_manager.cc:1875] T c9f33b62a4da4634b5d1bc40bdd39b15 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.220021 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6d48e2ca21a2409fae7c972ec53adbf5" } modify_external_catalogs: true
I20240502 08:16:34.220341 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6d48e2ca21a2409fae7c972ec53adbf5" } modify_external_catalogs: true
I20240502 08:16:34.224400 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet c9f33b62a4da4634b5d1bc40bdd39b15 with delete_type TABLET_DATA_DELETED (replaced by tablet e17eec8f2c1a4190945b434794635b52) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.224799 23604 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 476569457c3949c8a74007fcff8756f4
I20240502 08:16:34.226838 24180 tablet_replica.cc:330] T c9f33b62a4da4634b5d1bc40bdd39b15 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:34.227087 24180 raft_consensus.cc:2229] T c9f33b62a4da4634b5d1bc40bdd39b15 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.227396 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 4ce3e95d5ea7435593b7cfe5a670a76a
I20240502 08:16:34.227512 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet a2d10db950314f3db1bfa352b9d3a759 (table default.Table_0112e7ea83b84f04a2e6038f2179b7c6 [id=40060891c6d44d559c37891803b42478]) successfully deleted
I20240502 08:16:34.227810 24180 raft_consensus.cc:2258] T c9f33b62a4da4634b5d1bc40bdd39b15 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:34.227843 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 840aba1388ae4ff78fbb869d87dc1bf1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 840aba1388ae4ff78fbb869d87dc1bf1 already in progress: creating tablet
I20240502 08:16:34.227927 24174 ts_tablet_manager.cc:1398] T bf7678dbc7e641dfa259d36101d0dd5a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.040s user 0.001s sys 0.001s
I20240502 08:16:34.228765 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce3e95d5ea7435593b7cfe5a670a76a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.229351 24175 tablet_bootstrap.cc:492] T 315263b66700471b982072b335eb70c4 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:34.229379 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce3e95d5ea7435593b7cfe5a670a76a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.229537 24175 ts_tablet_manager.cc:1367] T 315263b66700471b982072b335eb70c4 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.014s
I20240502 08:16:34.229700 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 7a10f86a0f3f4385bef39eda14d4fa9f (DEFAULT_TABLE table=default.Table_95acc6e36a0b448eb3fd6d5df86e2de4 [id=1328b5d30ef24f98bf00c4429383ce92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.229993 24180 ts_tablet_manager.cc:1875] T c9f33b62a4da4634b5d1bc40bdd39b15 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.230095 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a10f86a0f3f4385bef39eda14d4fa9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.230443 24175 raft_consensus.cc:348] T 315263b66700471b982072b335eb70c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.230659 24175 raft_consensus.cc:374] T 315263b66700471b982072b335eb70c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.230751 24175 raft_consensus.cc:729] T 315263b66700471b982072b335eb70c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.230969 24175 consensus_queue.cc:260] T 315263b66700471b982072b335eb70c4 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.231066 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce3e95d5ea7435593b7cfe5a670a76a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.231467 24175 ts_tablet_manager.cc:1398] T 315263b66700471b982072b335eb70c4 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:34.232185 24175 tablet_bootstrap.cc:492] T 840aba1388ae4ff78fbb869d87dc1bf1 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:34.233791 24175 tablet_bootstrap.cc:654] T 840aba1388ae4ff78fbb869d87dc1bf1 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.241973 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet bf7678dbc7e641dfa259d36101d0dd5a with delete_type TABLET_DATA_DELETED (replaced by tablet 476569457c3949c8a74007fcff8756f4) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.242473 24179 ts_tablet_manager.cc:1888] T c9f33b62a4da4634b5d1bc40bdd39b15 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.242878 24174 tablet_bootstrap.cc:492] T 315263b66700471b982072b335eb70c4 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:34.242892 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e077b6d407694d3eaa7f0c34e3f8b917"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.243485 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e077b6d407694d3eaa7f0c34e3f8b917 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.244215 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 175adf0ecb504e28a3d39f001f977839 (DEFAULT_TABLE table=default.Table_658ef73f4b084866b43dd4bfc8b337c6 [id=a6a5f4c65807479392c01ac5c955c959]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.244364 24181 ts_tablet_manager.cc:1888] T 0cf60a6da6f94525b02bdbb9e53192db P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.244524 24181 log.cc:1196] T 0cf60a6da6f94525b02bdbb9e53192db P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/0cf60a6da6f94525b02bdbb9e53192db
I20240502 08:16:34.244937 24181 ts_tablet_manager.cc:1909] T 0cf60a6da6f94525b02bdbb9e53192db P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:34.246065 24179 log.cc:1196] T c9f33b62a4da4634b5d1bc40bdd39b15 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/c9f33b62a4da4634b5d1bc40bdd39b15
W20240502 08:16:34.246315 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 8dc8158255194b868af917c33c44f0f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dc8158255194b868af917c33c44f0f5 already in progress: creating tablet
I20240502 08:16:34.246460 24179 ts_tablet_manager.cc:1909] T c9f33b62a4da4634b5d1bc40bdd39b15 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:34.246642 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175adf0ecb504e28a3d39f001f977839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.248034 24173 tablet_bootstrap.cc:492] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:34.248198 24173 ts_tablet_manager.cc:1367] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.109s user 0.002s sys 0.006s
I20240502 08:16:34.249161 24173 raft_consensus.cc:348] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.249408 24173 raft_consensus.cc:374] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.249498 24173 raft_consensus.cc:729] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.249655 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet c9f33b62a4da4634b5d1bc40bdd39b15 (table default.Table_b01c340249ea483683e385a7fa964a51 [id=ee0e1cf16cb34aa98699970438cc36ae]) successfully deleted
I20240502 08:16:34.249781 24173 consensus_queue.cc:260] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
W20240502 08:16:34.250742 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 840aba1388ae4ff78fbb869d87dc1bf1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 840aba1388ae4ff78fbb869d87dc1bf1 already in progress: creating tablet
I20240502 08:16:34.251046 24181 tablet_replica.cc:330] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:34.251220 24181 raft_consensus.cc:2229] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240502 08:16:34.251644 23597 catalog_manager.cc:4797] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 4ce3e95d5ea7435593b7cfe5a670a76a because the tablet was not found. No further retry: Not found: Tablet not found: 4ce3e95d5ea7435593b7cfe5a670a76a
I20240502 08:16:34.252014 24174 tablet_bootstrap.cc:654] T 315263b66700471b982072b335eb70c4 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.253669 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 8791834597254e7c9f730e03568c0748 (DEFAULT_TABLE table=default.Table_232703fab6964c29987c2f5096c5e118 [id=467a7f6db73641ca85a1898b251938fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.254067 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8791834597254e7c9f730e03568c0748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.254644 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 175adf0ecb504e28a3d39f001f977839 (DEFAULT_TABLE table=default.Table_658ef73f4b084866b43dd4bfc8b337c6 [id=a6a5f4c65807479392c01ac5c955c959]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.255008 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175adf0ecb504e28a3d39f001f977839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.257658 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 8791834597254e7c9f730e03568c0748 (DEFAULT_TABLE table=default.Table_232703fab6964c29987c2f5096c5e118 [id=467a7f6db73641ca85a1898b251938fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.258111 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8791834597254e7c9f730e03568c0748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.261354 24175 tablet_bootstrap.cc:492] T 840aba1388ae4ff78fbb869d87dc1bf1 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:34.261510 24175 ts_tablet_manager.cc:1367] T 840aba1388ae4ff78fbb869d87dc1bf1 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.019s
I20240502 08:16:34.261739 24174 tablet_bootstrap.cc:492] T 315263b66700471b982072b335eb70c4 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:34.261941 24174 ts_tablet_manager.cc:1367] T 315263b66700471b982072b335eb70c4 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.005s
I20240502 08:16:34.262256 24175 raft_consensus.cc:348] T 840aba1388ae4ff78fbb869d87dc1bf1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.262488 24175 raft_consensus.cc:374] T 840aba1388ae4ff78fbb869d87dc1bf1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.262588 24175 raft_consensus.cc:729] T 840aba1388ae4ff78fbb869d87dc1bf1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.262979 24174 raft_consensus.cc:348] T 315263b66700471b982072b335eb70c4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.263056 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 175adf0ecb504e28a3d39f001f977839 (DEFAULT_TABLE table=default.Table_658ef73f4b084866b43dd4bfc8b337c6 [id=a6a5f4c65807479392c01ac5c955c959]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.263274 24174 raft_consensus.cc:374] T 315263b66700471b982072b335eb70c4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.263389 24174 raft_consensus.cc:729] T 315263b66700471b982072b335eb70c4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.263427 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 175adf0ecb504e28a3d39f001f977839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.263723 24174 consensus_queue.cc:260] T 315263b66700471b982072b335eb70c4 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.265223 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_623eac7101f94d959fb83b6f51d8fe0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.266009 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_623eac7101f94d959fb83b6f51d8fe0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.267293 24175 consensus_queue.cc:260] T 840aba1388ae4ff78fbb869d87dc1bf1 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.267510 24174 ts_tablet_manager.cc:1398] T 315263b66700471b982072b335eb70c4 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:16:34.267390 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0c1824385d9c4fca83aebae44d01483c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:34.267849 24181 raft_consensus.cc:2258] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:34.267881 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c1824385d9c4fca83aebae44d01483c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.269279 24174 tablet_bootstrap.cc:492] T 359eee8b86af469b94ab5e5323b7972c P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:34.270021 24181 ts_tablet_manager.cc:1875] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.270754 24174 tablet_bootstrap.cc:654] T 359eee8b86af469b94ab5e5323b7972c P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.272562 24173 ts_tablet_manager.cc:1398] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240502 08:16:34.273586 24175 ts_tablet_manager.cc:1398] T 840aba1388ae4ff78fbb869d87dc1bf1 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 08:16:34.273942 24175 tablet_bootstrap.cc:492] T 359eee8b86af469b94ab5e5323b7972c P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:34.274358 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5d8bead3c17041909cc72b6a7537570c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.274957 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d8bead3c17041909cc72b6a7537570c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.275617 24175 tablet_bootstrap.cc:654] T 359eee8b86af469b94ab5e5323b7972c P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.277489 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 840aba1388ae4ff78fbb869d87dc1bf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.277781 24179 tablet_replica.cc:330] T 840aba1388ae4ff78fbb869d87dc1bf1 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:34.277969 24179 raft_consensus.cc:2229] T 840aba1388ae4ff78fbb869d87dc1bf1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.278152 24179 raft_consensus.cc:2258] T 840aba1388ae4ff78fbb869d87dc1bf1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.278960 24179 ts_tablet_manager.cc:1875] T 840aba1388ae4ff78fbb869d87dc1bf1 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.286044 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 8791834597254e7c9f730e03568c0748 (DEFAULT_TABLE table=default.Table_232703fab6964c29987c2f5096c5e118 [id=467a7f6db73641ca85a1898b251938fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.286548 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8791834597254e7c9f730e03568c0748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.287912 24179 ts_tablet_manager.cc:1888] T 840aba1388ae4ff78fbb869d87dc1bf1 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.288103 24179 log.cc:1196] T 840aba1388ae4ff78fbb869d87dc1bf1 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/840aba1388ae4ff78fbb869d87dc1bf1
I20240502 08:16:34.289016 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_51bd1babb80d4d00afe457d7a5d68b4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.289687 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51bd1babb80d4d00afe457d7a5d68b4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.289835 24181 ts_tablet_manager.cc:1888] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.289985 24181 log.cc:1196] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/5c8a832a2e6844ff8481b1b66f30a3b2
I20240502 08:16:34.290261 24179 ts_tablet_manager.cc:1909] T 840aba1388ae4ff78fbb869d87dc1bf1 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:34.290416 24181 ts_tablet_manager.cc:1909] T 5c8a832a2e6844ff8481b1b66f30a3b2 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:34.291419 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 5c8a832a2e6844ff8481b1b66f30a3b2 (table default.Table_ffa5fe5beacd4345b72e20da3bb7cb76 [id=b24212eeb7784959b5fbcd68bcafa0d7]) successfully deleted
I20240502 08:16:34.292007 23899 tablet_service.cc:1467] Processing CreateTablet for tablet ec4028a8e66442bfabac9e587aed00bb (DEFAULT_TABLE table=default.Table_2e0ee58bc7db40da88a1700e3830444d [id=1e590d80e3e34e1aac8a0c7908a9d813]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.292419 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec4028a8e66442bfabac9e587aed00bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.292474 24173 tablet_bootstrap.cc:492] T bf7678dbc7e641dfa259d36101d0dd5a P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:34.293382 24181 tablet_replica.cc:330] T a2d10db950314f3db1bfa352b9d3a759 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:34.293596 24181 raft_consensus.cc:2229] T a2d10db950314f3db1bfa352b9d3a759 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.293756 24181 raft_consensus.cc:2258] T a2d10db950314f3db1bfa352b9d3a759 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.294258 23765 tablet_service.cc:1467] Processing CreateTablet for tablet ec4028a8e66442bfabac9e587aed00bb (DEFAULT_TABLE table=default.Table_2e0ee58bc7db40da88a1700e3830444d [id=1e590d80e3e34e1aac8a0c7908a9d813]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.294726 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec4028a8e66442bfabac9e587aed00bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.295267 24035 tablet_service.cc:1467] Processing CreateTablet for tablet ec4028a8e66442bfabac9e587aed00bb (DEFAULT_TABLE table=default.Table_2e0ee58bc7db40da88a1700e3830444d [id=1e590d80e3e34e1aac8a0c7908a9d813]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.295621 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec4028a8e66442bfabac9e587aed00bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.301533 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 840aba1388ae4ff78fbb869d87dc1bf1 (table default.Table_1d5afdec101c4b57864bc6d8eeefd20f [id=20e5e7a21d9f4439a3c3723a0fdfd264]) successfully deleted
I20240502 08:16:34.303781 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce3e95d5ea7435593b7cfe5a670a76a (DEFAULT_TABLE table=default.Table_6d48e2ca21a2409fae7c972ec53adbf5 [id=d06bffc628cd4e07a8651559cbe38e09]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.303956 24180 ts_tablet_manager.cc:1888] T c9f33b62a4da4634b5d1bc40bdd39b15 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.303965 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce3e95d5ea7435593b7cfe5a670a76a (DEFAULT_TABLE table=default.Table_6d48e2ca21a2409fae7c972ec53adbf5 [id=d06bffc628cd4e07a8651559cbe38e09]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.304104 24180 log.cc:1196] T c9f33b62a4da4634b5d1bc40bdd39b15 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/c9f33b62a4da4634b5d1bc40bdd39b15
I20240502 08:16:34.304177 24181 ts_tablet_manager.cc:1875] T a2d10db950314f3db1bfa352b9d3a759 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.304351 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ce3e95d5ea7435593b7cfe5a670a76a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.304523 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ce3e95d5ea7435593b7cfe5a670a76a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.304613 24180 ts_tablet_manager.cc:1909] T c9f33b62a4da4634b5d1bc40bdd39b15 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:34.305763 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet c9f33b62a4da4634b5d1bc40bdd39b15 (table default.Table_b01c340249ea483683e385a7fa964a51 [id=ee0e1cf16cb34aa98699970438cc36ae]) successfully deleted
W20240502 08:16:34.306553 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 4ce3e95d5ea7435593b7cfe5a670a76a because the tablet was not found. No further retry: Not found: Tablet not found: 4ce3e95d5ea7435593b7cfe5a670a76a
I20240502 08:16:34.306861 24180 tablet_replica.cc:330] T bf7678dbc7e641dfa259d36101d0dd5a P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:34.307065 24180 raft_consensus.cc:2229] T bf7678dbc7e641dfa259d36101d0dd5a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.307247 24180 raft_consensus.cc:2258] T bf7678dbc7e641dfa259d36101d0dd5a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.307943 24180 ts_tablet_manager.cc:1875] T bf7678dbc7e641dfa259d36101d0dd5a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.309015 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce3e95d5ea7435593b7cfe5a670a76a (DEFAULT_TABLE table=default.Table_6d48e2ca21a2409fae7c972ec53adbf5 [id=d06bffc628cd4e07a8651559cbe38e09]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.303649 24173 tablet_bootstrap.cc:654] T bf7678dbc7e641dfa259d36101d0dd5a P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.309407 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ce3e95d5ea7435593b7cfe5a670a76a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.312166 24174 tablet_bootstrap.cc:492] T 359eee8b86af469b94ab5e5323b7972c P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:34.312317 24174 ts_tablet_manager.cc:1367] T 359eee8b86af469b94ab5e5323b7972c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.006s
I20240502 08:16:34.312932 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0c1824385d9c4fca83aebae44d01483c" } modify_external_catalogs: true
I20240502 08:16:34.313108 24174 raft_consensus.cc:348] T 359eee8b86af469b94ab5e5323b7972c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.313198 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0c1824385d9c4fca83aebae44d01483c" } modify_external_catalogs: true
I20240502 08:16:34.313333 24174 raft_consensus.cc:374] T 359eee8b86af469b94ab5e5323b7972c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.313429 24174 raft_consensus.cc:729] T 359eee8b86af469b94ab5e5323b7972c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.313714 24174 consensus_queue.cc:260] T 359eee8b86af469b94ab5e5323b7972c P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.315196 24181 ts_tablet_manager.cc:1888] T a2d10db950314f3db1bfa352b9d3a759 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.315347 24181 log.cc:1196] T a2d10db950314f3db1bfa352b9d3a759 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/a2d10db950314f3db1bfa352b9d3a759
I20240502 08:16:34.315697 24181 ts_tablet_manager.cc:1909] T a2d10db950314f3db1bfa352b9d3a759 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:34.319247 24173 tablet_bootstrap.cc:492] T bf7678dbc7e641dfa259d36101d0dd5a P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:34.319566 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet a2d10db950314f3db1bfa352b9d3a759 (table default.Table_0112e7ea83b84f04a2e6038f2179b7c6 [id=40060891c6d44d559c37891803b42478]) successfully deleted
I20240502 08:16:34.319844 24173 ts_tablet_manager.cc:1367] T bf7678dbc7e641dfa259d36101d0dd5a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.001s
I20240502 08:16:34.320330 23604 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet f3f1ede6a35545169c9769ed2dbcd5ca
I20240502 08:16:34.321159 24173 raft_consensus.cc:348] T bf7678dbc7e641dfa259d36101d0dd5a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.321502 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet 359eee8b86af469b94ab5e5323b7972c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:33 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.321475 24181 tablet_replica.cc:330] T c9f33b62a4da4634b5d1bc40bdd39b15 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:34.321866 24173 raft_consensus.cc:374] T bf7678dbc7e641dfa259d36101d0dd5a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.321928 24181 raft_consensus.cc:2229] T c9f33b62a4da4634b5d1bc40bdd39b15 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.321956 24174 ts_tablet_manager.cc:1398] T 359eee8b86af469b94ab5e5323b7972c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 08:16:34.322052 24173 raft_consensus.cc:729] T bf7678dbc7e641dfa259d36101d0dd5a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.322121 24181 raft_consensus.cc:2258] T c9f33b62a4da4634b5d1bc40bdd39b15 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.322412 24173 consensus_queue.cc:260] T bf7678dbc7e641dfa259d36101d0dd5a P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.322829 24181 ts_tablet_manager.cc:1875] T c9f33b62a4da4634b5d1bc40bdd39b15 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.323006 24173 ts_tablet_manager.cc:1398] T bf7678dbc7e641dfa259d36101d0dd5a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:16:34.324054 24173 tablet_bootstrap.cc:492] T 840aba1388ae4ff78fbb869d87dc1bf1 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:34.325404 24174 tablet_bootstrap.cc:492] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:34.325903 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet bf7678dbc7e641dfa259d36101d0dd5a with delete_type TABLET_DATA_DELETED (replaced by tablet 476569457c3949c8a74007fcff8756f4) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.326181 24173 tablet_bootstrap.cc:654] T 840aba1388ae4ff78fbb869d87dc1bf1 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.326916 24174 tablet_bootstrap.cc:654] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.334293 24173 tablet_bootstrap.cc:492] T 840aba1388ae4ff78fbb869d87dc1bf1 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:34.334524 24173 ts_tablet_manager.cc:1367] T 840aba1388ae4ff78fbb869d87dc1bf1 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 08:16:34.335430 24173 raft_consensus.cc:348] T 840aba1388ae4ff78fbb869d87dc1bf1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.335747 24173 raft_consensus.cc:374] T 840aba1388ae4ff78fbb869d87dc1bf1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.335877 24173 raft_consensus.cc:729] T 840aba1388ae4ff78fbb869d87dc1bf1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.336151 24173 consensus_queue.cc:260] T 840aba1388ae4ff78fbb869d87dc1bf1 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.336246 24180 ts_tablet_manager.cc:1888] T bf7678dbc7e641dfa259d36101d0dd5a P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.336465 24180 log.cc:1196] T bf7678dbc7e641dfa259d36101d0dd5a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/bf7678dbc7e641dfa259d36101d0dd5a
I20240502 08:16:34.336897 24180 ts_tablet_manager.cc:1909] T bf7678dbc7e641dfa259d36101d0dd5a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:34.337220 24181 ts_tablet_manager.cc:1888] T c9f33b62a4da4634b5d1bc40bdd39b15 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.337328 24181 log.cc:1196] T c9f33b62a4da4634b5d1bc40bdd39b15 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/c9f33b62a4da4634b5d1bc40bdd39b15
I20240502 08:16:34.337684 24181 ts_tablet_manager.cc:1909] T c9f33b62a4da4634b5d1bc40bdd39b15 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:34.339044 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet c9f33b62a4da4634b5d1bc40bdd39b15 (table default.Table_b01c340249ea483683e385a7fa964a51 [id=ee0e1cf16cb34aa98699970438cc36ae]) successfully deleted
I20240502 08:16:34.339270 24175 tablet_bootstrap.cc:492] T 359eee8b86af469b94ab5e5323b7972c P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:34.339401 24175 ts_tablet_manager.cc:1367] T 359eee8b86af469b94ab5e5323b7972c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.007s
W20240502 08:16:34.339673 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 840aba1388ae4ff78fbb869d87dc1bf1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 840aba1388ae4ff78fbb869d87dc1bf1 already in progress: creating tablet
I20240502 08:16:34.340103 24175 raft_consensus.cc:348] T 359eee8b86af469b94ab5e5323b7972c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.340309 24175 raft_consensus.cc:374] T 359eee8b86af469b94ab5e5323b7972c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.340390 24175 raft_consensus.cc:729] T 359eee8b86af469b94ab5e5323b7972c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.340626 24175 consensus_queue.cc:260] T 359eee8b86af469b94ab5e5323b7972c P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.340891 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2c03990935e343a8bfca754e83a25ccf" } new_table_name: "default.Table_1f432841712848378bc24e8050ac57c7" modify_external_catalogs: true
I20240502 08:16:34.341121 24175 ts_tablet_manager.cc:1398] T 359eee8b86af469b94ab5e5323b7972c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:34.341394 24175 tablet_bootstrap.cc:492] T bf7678dbc7e641dfa259d36101d0dd5a P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:34.343433 24175 tablet_bootstrap.cc:654] T bf7678dbc7e641dfa259d36101d0dd5a P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.345381 24190 raft_consensus.cc:482] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:34.345549 24190 raft_consensus.cc:504] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.346149 24190 leader_election.cc:288] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
W20240502 08:16:34.340579 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 4ce3e95d5ea7435593b7cfe5a670a76a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ce3e95d5ea7435593b7cfe5a670a76a already in progress: creating tablet
I20240502 08:16:34.346721 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315263b66700471b982072b335eb70c4" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:34.346925 23924 raft_consensus.cc:2454] T 315263b66700471b982072b335eb70c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:34.347354 23996 leader_election.cc:302] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:34.347793 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 359eee8b86af469b94ab5e5323b7972c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:33 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.347852 24190 raft_consensus.cc:2781] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:34.347929 24190 raft_consensus.cc:482] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:34.348053 24179 tablet_replica.cc:330] T 359eee8b86af469b94ab5e5323b7972c P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:34.348054 24190 raft_consensus.cc:3037] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:34.348225 24179 raft_consensus.cc:2229] T 359eee8b86af469b94ab5e5323b7972c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.348371 24179 raft_consensus.cc:2258] T 359eee8b86af469b94ab5e5323b7972c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.350407 24190 raft_consensus.cc:504] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.350553 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b142a1494f9942d6a7dd855cfbbc38f7" } new_table_name: "default.Table_45e94198b75a4c62afc0f6f10d8c79f3" modify_external_catalogs: true
I20240502 08:16:34.351296 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315263b66700471b982072b335eb70c4" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:34.351449 23924 raft_consensus.cc:3037] T 315263b66700471b982072b335eb70c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:34.352691 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b2dc80505cd440f591086162a662e92d" } new_table_name: "default.Table_065c3fa8292242d59b5fbebbfa233fdc" modify_external_catalogs: true
I20240502 08:16:34.354640 23924 raft_consensus.cc:2454] T 315263b66700471b982072b335eb70c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:34.355053 23996 leader_election.cc:302] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:34.355350 24211 raft_consensus.cc:2781] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:34.355445 24211 raft_consensus.cc:686] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:34.355720 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c9fdfaaf57304983858c5afcf7b28ef6" } new_table_name: "default.Table_bc6b3019e44547cda285ac0021565619" modify_external_catalogs: true
I20240502 08:16:34.355683 24211 consensus_queue.cc:237] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.357789 24190 leader_election.cc:288] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
W20240502 08:16:34.360296 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c0a4b216c7e474db1992a299ff8eeb4 consensus state has no leader
I20240502 08:16:34.361541 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a0ca472d0e4748e6995f3b55e306e483" } modify_external_catalogs: true
I20240502 08:16:34.361637 24179 ts_tablet_manager.cc:1875] T 359eee8b86af469b94ab5e5323b7972c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.361788 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a0ca472d0e4748e6995f3b55e306e483" } modify_external_catalogs: true
I20240502 08:16:34.365598 24173 ts_tablet_manager.cc:1398] T 840aba1388ae4ff78fbb869d87dc1bf1 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
W20240502 08:16:34.365952 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5def9306dfba4d019d3acece81df2f86 consensus state has no leader
I20240502 08:16:34.366820 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_658ef73f4b084866b43dd4bfc8b337c6" } new_table_name: "default.Table_9530003aa5b841b683920da217d58576" modify_external_catalogs: true
W20240502 08:16:34.369881 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f2ff50fa357448494434d1487ce83bc has no consensus state
W20240502 08:16:34.371763 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e1cf5f66859d416baad2f767060197c8 consensus state has no leader
I20240502 08:16:34.371956 24179 ts_tablet_manager.cc:1888] T 359eee8b86af469b94ab5e5323b7972c P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.372105 24179 log.cc:1196] T 359eee8b86af469b94ab5e5323b7972c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/359eee8b86af469b94ab5e5323b7972c
I20240502 08:16:34.372402 24179 ts_tablet_manager.cc:1909] T 359eee8b86af469b94ab5e5323b7972c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:34.377300 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315263b66700471b982072b335eb70c4" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:34.377540 23787 raft_consensus.cc:3037] T 315263b66700471b982072b335eb70c4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:34.340806 24181 tablet_replica.cc:330] T bf7678dbc7e641dfa259d36101d0dd5a P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:34.378839 24181 raft_consensus.cc:2229] T bf7678dbc7e641dfa259d36101d0dd5a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.378935 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 359eee8b86af469b94ab5e5323b7972c (table default.Table_5485d4b9db4f416691aea5c4f94ba412 [id=1fedc678a02342729dda973cb255bf14]) successfully deleted
I20240502 08:16:34.379002 24181 raft_consensus.cc:2258] T bf7678dbc7e641dfa259d36101d0dd5a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:34.379462 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 175adf0ecb504e28a3d39f001f977839 consensus state has no leader
I20240502 08:16:34.378696 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315263b66700471b982072b335eb70c4" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:34.379840 24181 ts_tablet_manager.cc:1875] T bf7678dbc7e641dfa259d36101d0dd5a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.381233 24173 tablet_bootstrap.cc:492] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:34.385394 24173 tablet_bootstrap.cc:654] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.387704 23787 raft_consensus.cc:2454] T 315263b66700471b982072b335eb70c4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:34.388504 24181 ts_tablet_manager.cc:1888] T bf7678dbc7e641dfa259d36101d0dd5a P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.388643 24181 log.cc:1196] T bf7678dbc7e641dfa259d36101d0dd5a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/bf7678dbc7e641dfa259d36101d0dd5a
I20240502 08:16:34.389017 24181 ts_tablet_manager.cc:1909] T bf7678dbc7e641dfa259d36101d0dd5a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:34.389660 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet bf7678dbc7e641dfa259d36101d0dd5a (table default.Table_c60e3fa5881347d09ba464e544d7cbf8 [id=fac14487dd8840d0be3dcce182527117]) successfully deleted
I20240502 08:16:34.390134 24174 tablet_bootstrap.cc:492] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:34.390309 24174 ts_tablet_manager.cc:1367] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.065s user 0.028s sys 0.023s
I20240502 08:16:34.391086 24174 raft_consensus.cc:348] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.391317 24174 raft_consensus.cc:374] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.391402 24174 raft_consensus.cc:729] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.391378 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet bf7678dbc7e641dfa259d36101d0dd5a (table default.Table_c60e3fa5881347d09ba464e544d7cbf8 [id=fac14487dd8840d0be3dcce182527117]) successfully deleted
I20240502 08:16:34.391705 24180 tablet_replica.cc:330] T 359eee8b86af469b94ab5e5323b7972c P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:34.391916 24180 raft_consensus.cc:2229] T 359eee8b86af469b94ab5e5323b7972c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.392081 24180 raft_consensus.cc:2258] T 359eee8b86af469b94ab5e5323b7972c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.392772 24180 ts_tablet_manager.cc:1875] T 359eee8b86af469b94ab5e5323b7972c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.393669 24174 consensus_queue.cc:260] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.394229 24174 ts_tablet_manager.cc:1398] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:34.394415 24173 tablet_bootstrap.cc:492] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:34.394575 24173 ts_tablet_manager.cc:1367] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 08:16:34.394603 24174 tablet_bootstrap.cc:492] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:34.395254 24173 raft_consensus.cc:348] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.395448 24173 raft_consensus.cc:374] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.395499 24173 raft_consensus.cc:729] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.396176 24174 tablet_bootstrap.cc:654] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.397732 24173 consensus_queue.cc:260] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.398556 24173 ts_tablet_manager.cc:1398] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:34.399868 24175 tablet_bootstrap.cc:492] T bf7678dbc7e641dfa259d36101d0dd5a P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:34.400045 24175 ts_tablet_manager.cc:1367] T bf7678dbc7e641dfa259d36101d0dd5a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.005s
I20240502 08:16:34.400734 24175 raft_consensus.cc:348] T bf7678dbc7e641dfa259d36101d0dd5a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.400952 24175 raft_consensus.cc:374] T bf7678dbc7e641dfa259d36101d0dd5a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.401019 24175 raft_consensus.cc:729] T bf7678dbc7e641dfa259d36101d0dd5a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.401278 24175 consensus_queue.cc:260] T bf7678dbc7e641dfa259d36101d0dd5a P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.405195 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet bf7678dbc7e641dfa259d36101d0dd5a with delete_type TABLET_DATA_DELETED (replaced by tablet 476569457c3949c8a74007fcff8756f4) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.405437 23607 catalog_manager.cc:5484] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67). New cstate: current_term: 1 leader_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:34.407456 23899 tablet_service.cc:1467] Processing CreateTablet for tablet c1c2a988a3524c8fb7410e8508bd1abc (DEFAULT_TABLE table=default.Table_51bd1babb80d4d00afe457d7a5d68b4b [id=31ddf074089243a89bab5928860c625e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.407860 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c2a988a3524c8fb7410e8508bd1abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.408879 23601 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 68bdb761ff36426c96949e97279e2756 (table default.Table_50b02617337c4151bdc0ba38abd68144 [id=eca4080e8ebf46e9b58c85ee1d8675ca]). Expected version 1 got 0
I20240502 08:16:34.409436 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 3052510d77f344bf87decb9c2e5ec0bd (DEFAULT_TABLE table=default.Table_623eac7101f94d959fb83b6f51d8fe0a [id=54454efbe9394170822f237dc8e6115e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.409865 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3052510d77f344bf87decb9c2e5ec0bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.410998 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 3052510d77f344bf87decb9c2e5ec0bd (DEFAULT_TABLE table=default.Table_623eac7101f94d959fb83b6f51d8fe0a [id=54454efbe9394170822f237dc8e6115e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.411367 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3052510d77f344bf87decb9c2e5ec0bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.413851 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 26c9913f1ca146fe84bc07ca1b4315c5 (DEFAULT_TABLE table=default.Table_f8286fd5bd9f4a1ca39afc1e8b57eae2 [id=93515920607e433ca1e2b5573176dc0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.414245 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c9913f1ca146fe84bc07ca1b4315c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.414870 23764 tablet_service.cc:1467] Processing CreateTablet for tablet c1c2a988a3524c8fb7410e8508bd1abc (DEFAULT_TABLE table=default.Table_51bd1babb80d4d00afe457d7a5d68b4b [id=31ddf074089243a89bab5928860c625e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.414865 23767 tablet_service.cc:1467] Processing CreateTablet for tablet e17eec8f2c1a4190945b434794635b52 (DEFAULT_TABLE table=default.Table_b01c340249ea483683e385a7fa964a51 [id=ee0e1cf16cb34aa98699970438cc36ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.415300 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c2a988a3524c8fb7410e8508bd1abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.415633 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17eec8f2c1a4190945b434794635b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.417171 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6c583799e352452a87de6faa19d88490" } modify_external_catalogs: true
I20240502 08:16:34.417354 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6c583799e352452a87de6faa19d88490" } modify_external_catalogs: true
I20240502 08:16:34.417829 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2ff50fa357448494434d1487ce83bc (DEFAULT_TABLE table=default.Table_065c3fa8292242d59b5fbebbfa233fdc [id=efa730295c3c4bb69e64ca3fa324abb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.418150 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2ff50fa357448494434d1487ce83bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.418481 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet 5def9306dfba4d019d3acece81df2f86 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:34.418656 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet 315263b66700471b982072b335eb70c4 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:34.419638 24173 tablet_bootstrap.cc:492] T bce9392a8ad64004ba6ce9a236976cd6 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:34.420612 23606 master_service.cc:946] ReplaceTablet: received request to replace tablet e1cf5f66859d416baad2f767060197c8 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:34.421149 24173 tablet_bootstrap.cc:654] T bce9392a8ad64004ba6ce9a236976cd6 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.423256 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 9628134656eb4be1b3e1419e5c1accbe (DEFAULT_TABLE table=default.Table_e077b6d407694d3eaa7f0c34e3f8b917 [id=9f2f530c5f91424d91ec92a42602b781]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.423450 24039 tablet_service.cc:1467] Processing CreateTablet for tablet c1c2a988a3524c8fb7410e8508bd1abc (DEFAULT_TABLE table=default.Table_51bd1babb80d4d00afe457d7a5d68b4b [id=31ddf074089243a89bab5928860c625e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.423774 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9628134656eb4be1b3e1419e5c1accbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.423822 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c2a988a3524c8fb7410e8508bd1abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.423686 23762 tablet_service.cc:1467] Processing CreateTablet for tablet db80a159f14e485a971fd8bf58a21c4a (DEFAULT_TABLE table=default.Table_afaca5b9d2774c0baff53ca022b9c4dd [id=47682c26a85842c7b8991bf3828a9336]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.425069 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db80a159f14e485a971fd8bf58a21c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.426256 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7e2b03f5ed46441d9c62c30319eab83b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:34.426330 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 774487be9bc644cab4e420c3d33b3cad (DEFAULT_TABLE table=default.Table_a0ca472d0e4748e6995f3b55e306e483 [id=70dab926ba6c4a1e8b4cf29853ba5755]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.426294 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1956c23bde184dd4ba01ad0231165eb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.426786 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e2b03f5ed46441d9c62c30319eab83b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.426831 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 774487be9bc644cab4e420c3d33b3cad. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:34.426831 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1956c23bde184dd4ba01ad0231165eb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:34.430330 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet bf7678dbc7e641dfa259d36101d0dd5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf7678dbc7e641dfa259d36101d0dd5a already in progress: creating tablet
I20240502 08:16:34.430536 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_3ad20764d6174afbbf3c36e64babdb15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.431039 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ad20764d6174afbbf3c36e64babdb15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.431696 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 26c9913f1ca146fe84bc07ca1b4315c5 (DEFAULT_TABLE table=default.Table_f8286fd5bd9f4a1ca39afc1e8b57eae2 [id=93515920607e433ca1e2b5573176dc0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.432109 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c9913f1ca146fe84bc07ca1b4315c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.432262 23904 tablet_service.cc:1467] Processing CreateTablet for tablet e17eec8f2c1a4190945b434794635b52 (DEFAULT_TABLE table=default.Table_b01c340249ea483683e385a7fa964a51 [id=ee0e1cf16cb34aa98699970438cc36ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.432610 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17eec8f2c1a4190945b434794635b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.435742 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 3052510d77f344bf87decb9c2e5ec0bd (DEFAULT_TABLE table=default.Table_623eac7101f94d959fb83b6f51d8fe0a [id=54454efbe9394170822f237dc8e6115e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.436146 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 9628134656eb4be1b3e1419e5c1accbe (DEFAULT_TABLE table=default.Table_e077b6d407694d3eaa7f0c34e3f8b917 [id=9f2f530c5f91424d91ec92a42602b781]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.436285 24038 tablet_service.cc:1467] Processing CreateTablet for tablet e17eec8f2c1a4190945b434794635b52 (DEFAULT_TABLE table=default.Table_b01c340249ea483683e385a7fa964a51 [id=ee0e1cf16cb34aa98699970438cc36ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.436151 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3052510d77f344bf87decb9c2e5ec0bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.436697 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17eec8f2c1a4190945b434794635b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.436911 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9628134656eb4be1b3e1419e5c1accbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.437170 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 26c9913f1ca146fe84bc07ca1b4315c5 (DEFAULT_TABLE table=default.Table_f8286fd5bd9f4a1ca39afc1e8b57eae2 [id=93515920607e433ca1e2b5573176dc0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.436844 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2ff50fa357448494434d1487ce83bc (DEFAULT_TABLE table=default.Table_065c3fa8292242d59b5fbebbfa233fdc [id=efa730295c3c4bb69e64ca3fa324abb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.437477 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c9913f1ca146fe84bc07ca1b4315c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.438936 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2ff50fa357448494434d1487ce83bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.442124 24034 tablet_service.cc:1467] Processing CreateTablet for tablet db80a159f14e485a971fd8bf58a21c4a (DEFAULT_TABLE table=default.Table_afaca5b9d2774c0baff53ca022b9c4dd [id=47682c26a85842c7b8991bf3828a9336]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.442798 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db80a159f14e485a971fd8bf58a21c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.443544 24173 tablet_bootstrap.cc:492] T bce9392a8ad64004ba6ce9a236976cd6 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:34.443686 24173 ts_tablet_manager.cc:1367] T bce9392a8ad64004ba6ce9a236976cd6 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.003s
I20240502 08:16:34.444540 24173 raft_consensus.cc:348] T bce9392a8ad64004ba6ce9a236976cd6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.444765 24173 raft_consensus.cc:374] T bce9392a8ad64004ba6ce9a236976cd6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.444859 24173 raft_consensus.cc:729] T bce9392a8ad64004ba6ce9a236976cd6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.445163 24173 consensus_queue.cc:260] T bce9392a8ad64004ba6ce9a236976cd6 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.446081 24173 ts_tablet_manager.cc:1398] T bce9392a8ad64004ba6ce9a236976cd6 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:34.446429 24173 tablet_bootstrap.cc:492] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:34.448169 24173 tablet_bootstrap.cc:654] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.449772 23605 master_service.cc:946] ReplaceTablet: received request to replace tablet bce9392a8ad64004ba6ce9a236976cd6 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:34.453845 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 768167b977c14b53a7537879e349d354 (DEFAULT_TABLE table=default.Table_5d8bead3c17041909cc72b6a7537570c [id=ab1669d2a12f4187809987ed41fa0379]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.454437 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768167b977c14b53a7537879e349d354. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.454982 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 05cbbf9614094afaa6f7cee0af597242 (DEFAULT_TABLE table=default.Table_6c583799e352452a87de6faa19d88490 [id=80701c77b9b04253bba1d4438e6f2a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.455104 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 71c0c2f3def643dfa99ff4efe65f6acf (DEFAULT_TABLE table=default.Table_68bf91a14ffa42cdbb10a1d7ca56cb08 [id=59d945a73daa4918ba17063eae0b9b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.455035 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 768167b977c14b53a7537879e349d354 (DEFAULT_TABLE table=default.Table_5d8bead3c17041909cc72b6a7537570c [id=ab1669d2a12f4187809987ed41fa0379]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.455502 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05cbbf9614094afaa6f7cee0af597242. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.457058 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 774487be9bc644cab4e420c3d33b3cad (DEFAULT_TABLE table=default.Table_a0ca472d0e4748e6995f3b55e306e483 [id=70dab926ba6c4a1e8b4cf29853ba5755]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.457412 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 774487be9bc644cab4e420c3d33b3cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.458745 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 71c0c2f3def643dfa99ff4efe65f6acf (DEFAULT_TABLE table=default.Table_68bf91a14ffa42cdbb10a1d7ca56cb08 [id=59d945a73daa4918ba17063eae0b9b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.459254 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c0c2f3def643dfa99ff4efe65f6acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.461349 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 05cbbf9614094afaa6f7cee0af597242
I20240502 08:16:34.461673 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 05cbbf9614094afaa6f7cee0af597242 (DEFAULT_TABLE table=default.Table_6c583799e352452a87de6faa19d88490 [id=80701c77b9b04253bba1d4438e6f2a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.461800 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768167b977c14b53a7537879e349d354. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.462077 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_53754b76a22f4f7ca0aebc5b98d31887"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:34.462342 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05cbbf9614094afaa6f7cee0af597242. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:34.462582 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53754b76a22f4f7ca0aebc5b98d31887 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.463727 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 774487be9bc644cab4e420c3d33b3cad
I20240502 08:16:34.464342 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1f432841712848378bc24e8050ac57c7" } modify_external_catalogs: true
I20240502 08:16:34.464524 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1f432841712848378bc24e8050ac57c7" } modify_external_catalogs: true
I20240502 08:16:34.465873 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 5def9306dfba4d019d3acece81df2f86
I20240502 08:16:34.465904 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c0c2f3def643dfa99ff4efe65f6acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.466257 23602 catalog_manager.cc:6554] ReplaceTablet: tablet 5def9306dfba4d019d3acece81df2f86 deleted and replaced by tablet 223c92ae8f63402f8643e03d914e8e7e
I20240502 08:16:34.466490 23903 tablet_service.cc:1467] Processing CreateTablet for tablet db80a159f14e485a971fd8bf58a21c4a (DEFAULT_TABLE table=default.Table_afaca5b9d2774c0baff53ca022b9c4dd [id=47682c26a85842c7b8991bf3828a9336]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.466614 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d8c99815c26041c882419d42c6c32d90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:34.466883 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db80a159f14e485a971fd8bf58a21c4a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:34.467026 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8c99815c26041c882419d42c6c32d90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.431696 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 9628134656eb4be1b3e1419e5c1accbe (DEFAULT_TABLE table=default.Table_e077b6d407694d3eaa7f0c34e3f8b917 [id=9f2f530c5f91424d91ec92a42602b781]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.468350 23607 master_service.cc:946] ReplaceTablet: received request to replace tablet e1cf5f66859d416baad2f767060197c8 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:34.469130 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 774487be9bc644cab4e420c3d33b3cad (DEFAULT_TABLE table=default.Table_a0ca472d0e4748e6995f3b55e306e483 [id=70dab926ba6c4a1e8b4cf29853ba5755]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.469141 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2ff50fa357448494434d1487ce83bc (DEFAULT_TABLE table=default.Table_065c3fa8292242d59b5fbebbfa233fdc [id=efa730295c3c4bb69e64ca3fa324abb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.470228 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet 5def9306dfba4d019d3acece81df2f86 with delete_type TABLET_DATA_DELETED (replaced by tablet 223c92ae8f63402f8643e03d914e8e7e) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.470261 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet 774487be9bc644cab4e420c3d33b3cad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.470520 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 05cbbf9614094afaa6f7cee0af597242 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.470759 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet 5def9306dfba4d019d3acece81df2f86 with delete_type TABLET_DATA_DELETED (replaced by tablet 223c92ae8f63402f8643e03d914e8e7e) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.470978 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 774487be9bc644cab4e420c3d33b3cad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.471150 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 05cbbf9614094afaa6f7cee0af597242 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.476230 24174 tablet_bootstrap.cc:492] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:34.476450 24174 ts_tablet_manager.cc:1367] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.082s user 0.003s sys 0.007s
I20240502 08:16:34.477154 24174 raft_consensus.cc:348] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.477312 24174 raft_consensus.cc:374] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.477787 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e1cf5f66859d416baad2f767060197c8
I20240502 08:16:34.478245 23606 catalog_manager.cc:6554] ReplaceTablet: tablet e1cf5f66859d416baad2f767060197c8 deleted and replaced by tablet a15ff3120b1943a69ddb609d0575ae1a
I20240502 08:16:34.477397 24174 raft_consensus.cc:729] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.478551 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_39d3e9a8fea548abadf2eef7723684f2" } modify_external_catalogs: true
I20240502 08:16:34.478499 24174 consensus_queue.cc:260] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.478747 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_39d3e9a8fea548abadf2eef7723684f2" } modify_external_catalogs: true
I20240502 08:16:34.479028 24174 ts_tablet_manager.cc:1398] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:34.478931 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ff7dea03cba44212bde370fe376ca4b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.479389 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff7dea03cba44212bde370fe376ca4b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.479665 24174 tablet_bootstrap.cc:492] T bce9392a8ad64004ba6ce9a236976cd6 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:34.481468 24174 tablet_bootstrap.cc:654] T bce9392a8ad64004ba6ce9a236976cd6 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.483248 24173 tablet_bootstrap.cc:492] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:34.483453 24173 ts_tablet_manager.cc:1367] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.001s
I20240502 08:16:34.484206 24173 raft_consensus.cc:348] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.484490 24173 raft_consensus.cc:374] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.484566 24173 raft_consensus.cc:729] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.484835 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet e1cf5f66859d416baad2f767060197c8 with delete_type TABLET_DATA_DELETED (replaced by tablet a15ff3120b1943a69ddb609d0575ae1a) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.485136 24173 consensus_queue.cc:260] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.487334 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9628134656eb4be1b3e1419e5c1accbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.487653 24173 ts_tablet_manager.cc:1398] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:16:34.487928 24173 tablet_bootstrap.cc:492] T 74aaea4b05e84f2ab31a0439f82016c1 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:34.487958 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e4e01edd1fa74d61be1f1d3381342c19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.488579 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4e01edd1fa74d61be1f1d3381342c19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.489454 24173 tablet_bootstrap.cc:654] T 74aaea4b05e84f2ab31a0439f82016c1 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.489666 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 774487be9bc644cab4e420c3d33b3cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.494292 24174 tablet_bootstrap.cc:492] T bce9392a8ad64004ba6ce9a236976cd6 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:34.497422 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fb2b0c0e7cc44f61b97a2863650307b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:34.494238 23897 tablet_service.cc:1467] Processing CreateTablet for tablet 768167b977c14b53a7537879e349d354 (DEFAULT_TABLE table=default.Table_5d8bead3c17041909cc72b6a7537570c [id=ab1669d2a12f4187809987ed41fa0379]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:34.497987 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb2b0c0e7cc44f61b97a2863650307b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.498795 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2ff50fa357448494434d1487ce83bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.499274 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet bce9392a8ad64004ba6ce9a236976cd6
I20240502 08:16:34.499521 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet e1cf5f66859d416baad2f767060197c8 with delete_type TABLET_DATA_DELETED (replaced by tablet a15ff3120b1943a69ddb609d0575ae1a) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.499675 23605 catalog_manager.cc:6554] ReplaceTablet: tablet bce9392a8ad64004ba6ce9a236976cd6 deleted and replaced by tablet c316e1aeab224634b760641a4ce188c8
I20240502 08:16:34.495092 24174 ts_tablet_manager.cc:1367] T bce9392a8ad64004ba6ce9a236976cd6 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240502 08:16:34.500540 23610 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 68bdb761ff36426c96949e97279e2756 (table default.Table_50b02617337c4151bdc0ba38abd68144 [id=eca4080e8ebf46e9b58c85ee1d8675ca]). Expected version 1 got 0
I20240502 08:16:34.500757 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet bce9392a8ad64004ba6ce9a236976cd6 with delete_type TABLET_DATA_DELETED (replaced by tablet c316e1aeab224634b760641a4ce188c8) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.501307 24174 raft_consensus.cc:348] T bce9392a8ad64004ba6ce9a236976cd6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.501536 24174 raft_consensus.cc:374] T bce9392a8ad64004ba6ce9a236976cd6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.501632 24174 raft_consensus.cc:729] T bce9392a8ad64004ba6ce9a236976cd6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.502043 24174 consensus_queue.cc:260] T bce9392a8ad64004ba6ce9a236976cd6 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.502142 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_457be8c1c0be48dca271ddd6b1a95561"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.502538 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_457be8c1c0be48dca271ddd6b1a95561 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.504614 24174 ts_tablet_manager.cc:1398] T bce9392a8ad64004ba6ce9a236976cd6 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:16:34.504617 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet bce9392a8ad64004ba6ce9a236976cd6 with delete_type TABLET_DATA_DELETED (replaced by tablet c316e1aeab224634b760641a4ce188c8) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.506822 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet 840aba1388ae4ff78fbb869d87dc1bf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.507078 24184 raft_consensus.cc:482] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:34.507278 24184 raft_consensus.cc:504] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:34.507895 24184 leader_election.cc:288] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:34.508085 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet 359eee8b86af469b94ab5e5323b7972c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:33 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.508675 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7cdeb95f30074a40b9daa0ccb4e47a04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.509224 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7cdeb95f30074a40b9daa0ccb4e47a04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.509869 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c08a89a72894f0b96d7863fd71edfd0" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:34.509968 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet 05cbbf9614094afaa6f7cee0af597242 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.510120 24060 raft_consensus.cc:2454] T 9c08a89a72894f0b96d7863fd71edfd0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:34.510565 23723 leader_election.cc:302] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:34.510671 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 768167b977c14b53a7537879e349d354. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.511168 24184 raft_consensus.cc:2781] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:34.511283 24184 raft_consensus.cc:482] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:34.511364 24184 raft_consensus.cc:3037] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:34.512476 24181 tablet_replica.cc:330] T bce9392a8ad64004ba6ce9a236976cd6 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:34.512667 24181 raft_consensus.cc:2229] T bce9392a8ad64004ba6ce9a236976cd6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.512838 24181 raft_consensus.cc:2258] T bce9392a8ad64004ba6ce9a236976cd6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.513337 23429 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2vVFYb/build/asan/bin/kudu
/tmp/dist-test-task2vVFYb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.126:43265
--webserver_interface=127.22.225.126
--webserver_port=34035
--builtin_ntp_servers=127.22.225.84:40725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.126:43265,127.22.225.125:36189,127.22.225.124:41745
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240502 08:16:34.513620 24181 ts_tablet_manager.cc:1875] T bce9392a8ad64004ba6ce9a236976cd6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.513816 24184 raft_consensus.cc:504] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:34.515134 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c08a89a72894f0b96d7863fd71edfd0" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:34.515318 24060 raft_consensus.cc:3037] T 9c08a89a72894f0b96d7863fd71edfd0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:34.517900 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 5def9306dfba4d019d3acece81df2f86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5def9306dfba4d019d3acece81df2f86 already in progress: creating tablet
W20240502 08:16:34.518220 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 774487be9bc644cab4e420c3d33b3cad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 774487be9bc644cab4e420c3d33b3cad already in progress: creating tablet
W20240502 08:16:34.518437 23597 catalog_manager.cc:4797] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 05cbbf9614094afaa6f7cee0af597242 because the tablet was not found. No further retry: Not found: Tablet not found: 05cbbf9614094afaa6f7cee0af597242
W20240502 08:16:34.518703 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e1cf5f66859d416baad2f767060197c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1cf5f66859d416baad2f767060197c8 already in progress: creating tablet
I20240502 08:16:34.519239 24184 leader_election.cc:288] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:34.522012 23597 catalog_manager.cc:4797] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 05cbbf9614094afaa6f7cee0af597242 because the tablet was not found. No further retry: Not found: Tablet not found: 05cbbf9614094afaa6f7cee0af597242
I20240502 08:16:34.522902 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 6c0a4b216c7e474db1992a299ff8eeb4
I20240502 08:16:34.523330 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet e1cf5f66859d416baad2f767060197c8 with delete_type TABLET_DATA_DELETED (replaced by tablet a15ff3120b1943a69ddb609d0575ae1a) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.523414 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c08a89a72894f0b96d7863fd71edfd0" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:34.524032 24180 ts_tablet_manager.cc:1888] T 359eee8b86af469b94ab5e5323b7972c P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.524022 23924 raft_consensus.cc:2454] T 9c08a89a72894f0b96d7863fd71edfd0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:34.524188 24180 log.cc:1196] T 359eee8b86af469b94ab5e5323b7972c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/359eee8b86af469b94ab5e5323b7972c
I20240502 08:16:34.524593 24180 ts_tablet_manager.cc:1909] T 359eee8b86af469b94ab5e5323b7972c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:34.525359 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f8286fd5bd9f4a1ca39afc1e8b57eae2" } modify_external_catalogs: true
I20240502 08:16:34.525543 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f8286fd5bd9f4a1ca39afc1e8b57eae2" } modify_external_catalogs: true
I20240502 08:16:34.526345 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 774487be9bc644cab4e420c3d33b3cad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.526574 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 5def9306dfba4d019d3acece81df2f86 with delete_type TABLET_DATA_DELETED (replaced by tablet 223c92ae8f63402f8643e03d914e8e7e) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.526742 24180 tablet_replica.cc:330] T bce9392a8ad64004ba6ce9a236976cd6 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:34.526731 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 05cbbf9614094afaa6f7cee0af597242 (DEFAULT_TABLE table=default.Table_6c583799e352452a87de6faa19d88490 [id=80701c77b9b04253bba1d4438e6f2a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.526929 24180 raft_consensus.cc:2229] T bce9392a8ad64004ba6ce9a236976cd6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.527096 24180 raft_consensus.cc:2258] T bce9392a8ad64004ba6ce9a236976cd6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.527271 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05cbbf9614094afaa6f7cee0af597242. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:34.527352 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e1cf5f66859d416baad2f767060197c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1cf5f66859d416baad2f767060197c8 already in progress: creating tablet
I20240502 08:16:34.527920 24180 ts_tablet_manager.cc:1875] T bce9392a8ad64004ba6ce9a236976cd6 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.528695 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 359eee8b86af469b94ab5e5323b7972c (table default.Table_5485d4b9db4f416691aea5c4f94ba412 [id=1fedc678a02342729dda973cb255bf14]) successfully deleted
W20240502 08:16:34.528925 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 774487be9bc644cab4e420c3d33b3cad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 774487be9bc644cab4e420c3d33b3cad already in progress: creating tablet
W20240502 08:16:34.529103 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 5def9306dfba4d019d3acece81df2f86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5def9306dfba4d019d3acece81df2f86 already in progress: creating tablet
W20240502 08:16:34.529287 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 05cbbf9614094afaa6f7cee0af597242 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05cbbf9614094afaa6f7cee0af597242 already in progress: creating tablet
W20240502 08:16:34.529454 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e1cf5f66859d416baad2f767060197c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1cf5f66859d416baad2f767060197c8 already in progress: creating tablet
I20240502 08:16:34.529464 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c08a89a72894f0b96d7863fd71edfd0" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:34.530537 24174 tablet_bootstrap.cc:492] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:34.531948 24174 tablet_bootstrap.cc:654] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:34.532910 23597 catalog_manager.cc:4797] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 774487be9bc644cab4e420c3d33b3cad because the tablet was not found. No further retry: Not found: Tablet not found: 774487be9bc644cab4e420c3d33b3cad
W20240502 08:16:34.533237 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 5def9306dfba4d019d3acece81df2f86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5def9306dfba4d019d3acece81df2f86 already in progress: creating tablet
I20240502 08:16:34.535027 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c0a4b216c7e474db1992a299ff8eeb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.535141 23924 raft_consensus.cc:3037] T 9c08a89a72894f0b96d7863fd71edfd0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:34.536978 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet bce9392a8ad64004ba6ce9a236976cd6 with delete_type TABLET_DATA_DELETED (replaced by tablet c316e1aeab224634b760641a4ce188c8) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.538544 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c0a4b216c7e474db1992a299ff8eeb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.540871 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 315263b66700471b982072b335eb70c4
I20240502 08:16:34.541353 23609 catalog_manager.cc:6554] ReplaceTablet: tablet 315263b66700471b982072b335eb70c4 deleted and replaced by tablet 6f7518b1e36147a9b9f0c58bca17061a
I20240502 08:16:34.542090 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 74aaea4b05e84f2ab31a0439f82016c1
I20240502 08:16:34.537750 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 71c0c2f3def643dfa99ff4efe65f6acf (DEFAULT_TABLE table=default.Table_68bf91a14ffa42cdbb10a1d7ca56cb08 [id=59d945a73daa4918ba17063eae0b9b93]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:34.545329 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet bce9392a8ad64004ba6ce9a236976cd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bce9392a8ad64004ba6ce9a236976cd6 already in progress: creating tablet
I20240502 08:16:34.547443 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet 315263b66700471b982072b335eb70c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f7518b1e36147a9b9f0c58bca17061a) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.547706 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 74aaea4b05e84f2ab31a0439f82016c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.552575 24174 tablet_bootstrap.cc:492] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:34.552773 24174 ts_tablet_manager.cc:1367] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.004s
I20240502 08:16:34.553597 24174 raft_consensus.cc:348] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.553820 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c0c2f3def643dfa99ff4efe65f6acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.553922 24174 raft_consensus.cc:374] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.554086 24174 raft_consensus.cc:729] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.554414 24174 consensus_queue.cc:260] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.557467 24174 ts_tablet_manager.cc:1398] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20240502 08:16:34.559056 23595 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 050c16739efb4037a4adc78a2ffd3cba (table default.Table_e0bf76fa2b2d451eb0102e9897669b6f [id=ba62a673079d424992daa0f6b0c5d6b9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 050c16739efb4037a4adc78a2ffd3cba consensus state has no leader
I20240502 08:16:34.560823 24174 tablet_bootstrap.cc:492] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:34.562292 23924 raft_consensus.cc:2454] T 9c08a89a72894f0b96d7863fd71edfd0 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:34.563890 23723 leader_election.cc:302] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:34.564321 24184 raft_consensus.cc:2781] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:34.564940 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_45e94198b75a4c62afc0f6f10d8c79f3" } new_table_name: "default.Table_00eb74cd24954c52b5484751f035f4c1" modify_external_catalogs: true
W20240502 08:16:34.565009 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 6c0a4b216c7e474db1992a299ff8eeb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c0a4b216c7e474db1992a299ff8eeb4 already in progress: creating tablet
I20240502 08:16:34.568477 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_51bd1babb80d4d00afe457d7a5d68b4b" } new_table_name: "default.Table_b96eb662e06846039807538794442017" modify_external_catalogs: true
I20240502 08:16:34.569197 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 32150c7e130f4069a2a3a02e625bfb93 (DEFAULT_TABLE table=default.Table_e4e01edd1fa74d61be1f1d3381342c19 [id=a895b3f375374ad6bb44a6f87fc7b98d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.569589 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32150c7e130f4069a2a3a02e625bfb93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.571513 24174 tablet_bootstrap.cc:654] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.573858 23904 tablet_service.cc:1514] Processing DeleteTablet for tablet 74aaea4b05e84f2ab31a0439f82016c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.574807 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_232703fab6964c29987c2f5096c5e118" } new_table_name: "default.Table_6c584db923e147d380281ecb1acd11a1" modify_external_catalogs: true
I20240502 08:16:34.575997 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ff7dea03cba44212bde370fe376ca4b4" } new_table_name: "default.Table_0170ebbe1b274aed934c0dd0445f3271" modify_external_catalogs: true
W20240502 08:16:34.578583 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 74aaea4b05e84f2ab31a0439f82016c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74aaea4b05e84f2ab31a0439f82016c1 already in progress: creating tablet
I20240502 08:16:34.582350 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 26c9913f1ca146fe84bc07ca1b4315c5
I20240502 08:16:34.584188 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c9913f1ca146fe84bc07ca1b4315c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.584729 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c9913f1ca146fe84bc07ca1b4315c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:34.585202 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 223c92ae8f63402f8643e03d914e8e7e consensus state has no leader
W20240502 08:16:34.585657 23597 catalog_manager.cc:4797] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 26c9913f1ca146fe84bc07ca1b4315c5 because the tablet was not found. No further retry: Not found: Tablet not found: 26c9913f1ca146fe84bc07ca1b4315c5
I20240502 08:16:34.588948 24174 tablet_bootstrap.cc:492] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:34.589109 24174 ts_tablet_manager.cc:1367] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
W20240502 08:16:34.589301 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1c2a988a3524c8fb7410e8508bd1abc consensus state has no leader
I20240502 08:16:34.589810 24174 raft_consensus.cc:348] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.590039 24174 raft_consensus.cc:374] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.590103 24174 raft_consensus.cc:729] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
W20240502 08:16:34.590225 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 485c772245fe48a39fc6ceb6377ac0b6 has no consensus state
I20240502 08:16:34.590325 24174 consensus_queue.cc:260] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
W20240502 08:16:34.591367 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8791834597254e7c9f730e03568c0748 consensus state has no leader
I20240502 08:16:34.592020 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2e0ee58bc7db40da88a1700e3830444d" } new_table_name: "default.Table_62b86da21eb84bc387b64725dabda733" modify_external_catalogs: true
I20240502 08:16:34.584470 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 32150c7e130f4069a2a3a02e625bfb93 (DEFAULT_TABLE table=default.Table_e4e01edd1fa74d61be1f1d3381342c19 [id=a895b3f375374ad6bb44a6f87fc7b98d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.593227 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32150c7e130f4069a2a3a02e625bfb93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.594298 24180 ts_tablet_manager.cc:1888] T bce9392a8ad64004ba6ce9a236976cd6 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.595104 24180 log.cc:1196] T bce9392a8ad64004ba6ce9a236976cd6 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/bce9392a8ad64004ba6ce9a236976cd6
I20240502 08:16:34.596475 24180 ts_tablet_manager.cc:1909] T bce9392a8ad64004ba6ce9a236976cd6 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:34.597496 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet bce9392a8ad64004ba6ce9a236976cd6 (table default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]) successfully deleted
W20240502 08:16:34.598793 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec4028a8e66442bfabac9e587aed00bb consensus state has no leader
I20240502 08:16:34.598848 23763 tablet_service.cc:1467] Processing CreateTablet for tablet ab70513985b54416856a29b32cecae4d (DEFAULT_TABLE table=default.Table_fb2b0c0e7cc44f61b97a2863650307b3 [id=259cb14e2ef6427f8dfba46f188050dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.599704 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab70513985b54416856a29b32cecae4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.602043 24174 ts_tablet_manager.cc:1398] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.013s user 0.000s sys 0.007s
I20240502 08:16:34.602852 23901 tablet_service.cc:1467] Processing CreateTablet for tablet ab70513985b54416856a29b32cecae4d (DEFAULT_TABLE table=default.Table_fb2b0c0e7cc44f61b97a2863650307b3 [id=259cb14e2ef6427f8dfba46f188050dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.603291 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab70513985b54416856a29b32cecae4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.603449 24174 tablet_bootstrap.cc:492] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:34.604919 24246 raft_consensus.cc:482] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:34.605414 24246 raft_consensus.cc:504] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.605099 24174 tablet_bootstrap.cc:654] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.607282 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 405ea360814e4593a289d753821277ff (DEFAULT_TABLE table=default.Table_d8c99815c26041c882419d42c6c32d90 [id=6d4996e259e34d51b7ee13c8efa7e2bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.607714 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 405ea360814e4593a289d753821277ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.607761 24246 leader_election.cc:288] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:34.617309 24175 ts_tablet_manager.cc:1398] T bf7678dbc7e641dfa259d36101d0dd5a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.217s user 0.001s sys 0.001s
I20240502 08:16:34.619123 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c9913f1ca146fe84bc07ca1b4315c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.619369 24060 raft_consensus.cc:2454] T 9c08a89a72894f0b96d7863fd71edfd0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:34.620250 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74aaea4b05e84f2ab31a0439f82016c1" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:34.621083 24181 ts_tablet_manager.cc:1888] T bce9392a8ad64004ba6ce9a236976cd6 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.621230 24181 log.cc:1196] T bce9392a8ad64004ba6ce9a236976cd6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/bce9392a8ad64004ba6ce9a236976cd6
I20240502 08:16:34.621238 24246 raft_consensus.cc:482] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:34.621443 24246 raft_consensus.cc:504] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.622228 24246 leader_election.cc:288] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:34.622844 23723 leader_election.cc:341] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:34.623426 23723 leader_election.cc:341] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Not found: Tablet not found: 840aba1388ae4ff78fbb869d87dc1bf1
I20240502 08:16:34.623756 24031 tablet_service.cc:1467] Processing CreateTablet for tablet ab70513985b54416856a29b32cecae4d (DEFAULT_TABLE table=default.Table_fb2b0c0e7cc44f61b97a2863650307b3 [id=259cb14e2ef6427f8dfba46f188050dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.625696 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "840aba1388ae4ff78fbb869d87dc1bf1" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:34.626438 24180 tablet_replica.cc:330] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:34.626695 24180 raft_consensus.cc:2229] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.626926 24180 raft_consensus.cc:2258] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.627800 24180 ts_tablet_manager.cc:1875] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.628021 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet 315263b66700471b982072b335eb70c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f7518b1e36147a9b9f0c58bca17061a) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.628224 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 32150c7e130f4069a2a3a02e625bfb93 (DEFAULT_TABLE table=default.Table_e4e01edd1fa74d61be1f1d3381342c19 [id=a895b3f375374ad6bb44a6f87fc7b98d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.628566 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74aaea4b05e84f2ab31a0439f82016c1" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:34.629133 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 1ce5ccfd29724dc49afd0892b99f3169 (DEFAULT_TABLE table=default.Table_53754b76a22f4f7ca0aebc5b98d31887 [id=3a9d7e8fe2c84cb4942fee9053a012b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.629169 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c0a4b216c7e474db1992a299ff8eeb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:34.629199 23723 leader_election.cc:341] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:34.629419 24211 raft_consensus.cc:482] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:34.629469 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab70513985b54416856a29b32cecae4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.629469 23723 leader_election.cc:302] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:34.629701 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32150c7e130f4069a2a3a02e625bfb93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.629626 24211 raft_consensus.cc:504] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.629974 24246 raft_consensus.cc:2726] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:34.630255 24211 leader_election.cc:288] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:34.630638 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ce5ccfd29724dc49afd0892b99f3169. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.629420 24181 ts_tablet_manager.cc:1909] T bce9392a8ad64004ba6ce9a236976cd6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:34.631940 24181 tablet_replica.cc:330] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:34.632158 24181 raft_consensus.cc:2229] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.632333 24181 raft_consensus.cc:2258] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.633085 24181 ts_tablet_manager.cc:1875] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.633646 24060 raft_consensus.cc:2454] T 840aba1388ae4ff78fbb869d87dc1bf1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:34.634384 23723 leader_election.cc:302] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
W20240502 08:16:34.634840 23723 raft_consensus.cc:2670] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:16:34.635473 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 5160fda2643e4b80bc401bc48f97eff9 (DEFAULT_TABLE table=default.Table_7e2b03f5ed46441d9c62c30319eab83b [id=8db1f2345d324647881132f139888dee]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:34.636232 23996 leader_election.cc:341] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Not found: Tablet not found: 359eee8b86af469b94ab5e5323b7972c
I20240502 08:16:34.636312 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 1ce5ccfd29724dc49afd0892b99f3169 (DEFAULT_TABLE table=default.Table_53754b76a22f4f7ca0aebc5b98d31887 [id=3a9d7e8fe2c84cb4942fee9053a012b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.636768 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ce5ccfd29724dc49afd0892b99f3169. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.637476 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet bce9392a8ad64004ba6ce9a236976cd6 (table default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]) successfully deleted
I20240502 08:16:34.637728 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 1ce5ccfd29724dc49afd0892b99f3169 (DEFAULT_TABLE table=default.Table_53754b76a22f4f7ca0aebc5b98d31887 [id=3a9d7e8fe2c84cb4942fee9053a012b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.638181 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ce5ccfd29724dc49afd0892b99f3169. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.638367 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5160fda2643e4b80bc401bc48f97eff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.638679 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 840aba1388ae4ff78fbb869d87dc1bf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.640858 24180 ts_tablet_manager.cc:1888] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.640977 24180 log.cc:1196] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/840aba1388ae4ff78fbb869d87dc1bf1
I20240502 08:16:34.641337 24180 ts_tablet_manager.cc:1909] T 840aba1388ae4ff78fbb869d87dc1bf1 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:34.642410 24180 tablet_replica.cc:330] T 315263b66700471b982072b335eb70c4 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:34.642627 24180 raft_consensus.cc:2229] T 315263b66700471b982072b335eb70c4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.642781 24180 raft_consensus.cc:2258] T 315263b66700471b982072b335eb70c4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.645730 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 840aba1388ae4ff78fbb869d87dc1bf1 (table default.Table_1d5afdec101c4b57864bc6d8eeefd20f [id=20e5e7a21d9f4439a3c3723a0fdfd264]) successfully deleted
W20240502 08:16:34.646157 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 6c0a4b216c7e474db1992a299ff8eeb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c0a4b216c7e474db1992a299ff8eeb4 already in progress: creating tablet
I20240502 08:16:34.650210 24180 ts_tablet_manager.cc:1875] T 315263b66700471b982072b335eb70c4 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.651692 24181 ts_tablet_manager.cc:1888] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.651831 24181 log.cc:1196] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/359eee8b86af469b94ab5e5323b7972c
I20240502 08:16:34.652159 24181 ts_tablet_manager.cc:1909] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:34.652756 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 359eee8b86af469b94ab5e5323b7972c (table default.Table_5485d4b9db4f416691aea5c4f94ba412 [id=1fedc678a02342729dda973cb255bf14]) successfully deleted
I20240502 08:16:34.655786 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 405ea360814e4593a289d753821277ff (DEFAULT_TABLE table=default.Table_d8c99815c26041c882419d42c6c32d90 [id=6d4996e259e34d51b7ee13c8efa7e2bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.656347 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 405ea360814e4593a289d753821277ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.659021 23902 tablet_service.cc:1467] Processing CreateTablet for tablet c7aa03e73acd4a9da5574b9ba93415ae (DEFAULT_TABLE table=default.Table_7cdeb95f30074a40b9daa0ccb4e47a04 [id=d37b40e0c2d942369def844a0a6db568]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.659082 23767 tablet_service.cc:1467] Processing CreateTablet for tablet c7aa03e73acd4a9da5574b9ba93415ae (DEFAULT_TABLE table=default.Table_7cdeb95f30074a40b9daa0ccb4e47a04 [id=d37b40e0c2d942369def844a0a6db568]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.659396 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7aa03e73acd4a9da5574b9ba93415ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.659408 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7aa03e73acd4a9da5574b9ba93415ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.659597 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 405ea360814e4593a289d753821277ff (DEFAULT_TABLE table=default.Table_d8c99815c26041c882419d42c6c32d90 [id=6d4996e259e34d51b7ee13c8efa7e2bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.663638 24180 ts_tablet_manager.cc:1888] T 315263b66700471b982072b335eb70c4 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.663784 24180 log.cc:1196] T 315263b66700471b982072b335eb70c4 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/315263b66700471b982072b335eb70c4
I20240502 08:16:34.664173 24180 ts_tablet_manager.cc:1909] T 315263b66700471b982072b335eb70c4 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:34.665144 24180 tablet_replica.cc:330] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:34.665326 24180 raft_consensus.cc:2229] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.665482 24180 raft_consensus.cc:2258] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.665529 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 315263b66700471b982072b335eb70c4 (table default.Table_4713741287e34e5d986ffb272ed6291d [id=38e955d6bd644d24a7c45172ab60282b]) successfully deleted
I20240502 08:16:34.667426 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 405ea360814e4593a289d753821277ff. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:34.667694 23993 leader_election.cc:341] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 359eee8b86af469b94ab5e5323b7972c
I20240502 08:16:34.667887 23993 leader_election.cc:302] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:34.668323 23993 raft_consensus.cc:2670] T 359eee8b86af469b94ab5e5323b7972c P 149ce744feab49b5aa5bcaa14bd6aba5: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:16:34.669631 24034 tablet_service.cc:1514] Processing DeleteTablet for tablet 74aaea4b05e84f2ab31a0439f82016c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.662819 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet bf7678dbc7e641dfa259d36101d0dd5a with delete_type TABLET_DATA_DELETED (replaced by tablet 476569457c3949c8a74007fcff8756f4) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.669848 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 5160fda2643e4b80bc401bc48f97eff9 (DEFAULT_TABLE table=default.Table_7e2b03f5ed46441d9c62c30319eab83b [id=8db1f2345d324647881132f139888dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.670089 24179 tablet_replica.cc:330] T bf7678dbc7e641dfa259d36101d0dd5a P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:34.670316 24179 raft_consensus.cc:2229] T bf7678dbc7e641dfa259d36101d0dd5a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.670495 24179 raft_consensus.cc:2258] T bf7678dbc7e641dfa259d36101d0dd5a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.670401 24031 tablet_service.cc:1467] Processing CreateTablet for tablet c7aa03e73acd4a9da5574b9ba93415ae (DEFAULT_TABLE table=default.Table_7cdeb95f30074a40b9daa0ccb4e47a04 [id=d37b40e0c2d942369def844a0a6db568]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.671205 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 (DEFAULT_TABLE table=default.Table_3ad20764d6174afbbf3c36e64babdb15 [id=134ba3221728463c9509f78ea3ea5418]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.671455 24179 ts_tablet_manager.cc:1875] T bf7678dbc7e641dfa259d36101d0dd5a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.665926 24175 tablet_bootstrap.cc:492] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:34.666131 23903 tablet_service.cc:1514] Processing DeleteTablet for tablet 315263b66700471b982072b335eb70c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f7518b1e36147a9b9f0c58bca17061a) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.660113 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 5160fda2643e4b80bc401bc48f97eff9 (DEFAULT_TABLE table=default.Table_7e2b03f5ed46441d9c62c30319eab83b [id=8db1f2345d324647881132f139888dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.672696 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 (DEFAULT_TABLE table=default.Table_3ad20764d6174afbbf3c36e64babdb15 [id=134ba3221728463c9509f78ea3ea5418]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.675024 24244 raft_consensus.cc:482] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:34.675215 24244 raft_consensus.cc:504] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.676577 24180 ts_tablet_manager.cc:1875] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.677772 24244 leader_election.cc:288] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:34.677834 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7aa03e73acd4a9da5574b9ba93415ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.678094 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea2dee2d5914cf0bf4b37d16865e9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.678651 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5160fda2643e4b80bc401bc48f97eff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.678716 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 (DEFAULT_TABLE table=default.Table_3ad20764d6174afbbf3c36e64babdb15 [id=134ba3221728463c9509f78ea3ea5418]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.678913 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5def9306dfba4d019d3acece81df2f86" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:34.679497 23768 tablet_service.cc:1467] Processing CreateTablet for tablet c316e1aeab224634b760641a4ce188c8 (DEFAULT_TABLE table=default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.679869 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c316e1aeab224634b760641a4ce188c8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:34.683699 23723 leader_election.cc:341] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:34.685796 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea2dee2d5914cf0bf4b37d16865e9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.688956 24180 ts_tablet_manager.cc:1888] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.689179 24180 log.cc:1196] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/74aaea4b05e84f2ab31a0439f82016c1
I20240502 08:16:34.689831 24180 ts_tablet_manager.cc:1909] T 74aaea4b05e84f2ab31a0439f82016c1 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:34.691381 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 74aaea4b05e84f2ab31a0439f82016c1 (table default.Table_39d3e9a8fea548abadf2eef7723684f2 [id=495bc1caa3e44d56abf4269f906ef74d]) successfully deleted
W20240502 08:16:34.691780 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 26c9913f1ca146fe84bc07ca1b4315c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26c9913f1ca146fe84bc07ca1b4315c5 already in progress: creating tablet
I20240502 08:16:34.691887 24037 tablet_service.cc:1467] Processing CreateTablet for tablet c316e1aeab224634b760641a4ce188c8 (DEFAULT_TABLE table=default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.692322 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c316e1aeab224634b760641a4ce188c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.693393 23767 tablet_service.cc:1467] Processing CreateTablet for tablet d8324e9b796047428e2fe918f07f9b32 (DEFAULT_TABLE table=default.Table_1956c23bde184dd4ba01ad0231165eb6 [id=e3f148f833bc4d99a3ab2ec8d5258507]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.694274 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8324e9b796047428e2fe918f07f9b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.695950 24033 tablet_service.cc:1467] Processing CreateTablet for tablet d8324e9b796047428e2fe918f07f9b32 (DEFAULT_TABLE table=default.Table_1956c23bde184dd4ba01ad0231165eb6 [id=e3f148f833bc4d99a3ab2ec8d5258507]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.696166 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 6f7518b1e36147a9b9f0c58bca17061a (DEFAULT_TABLE table=default.Table_4713741287e34e5d986ffb272ed6291d [id=38e955d6bd644d24a7c45172ab60282b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.696370 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8324e9b796047428e2fe918f07f9b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.696519 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f7518b1e36147a9b9f0c58bca17061a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:34.704967 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 26c9913f1ca146fe84bc07ca1b4315c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26c9913f1ca146fe84bc07ca1b4315c5 already in progress: creating tablet
I20240502 08:16:34.706897 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 6f7518b1e36147a9b9f0c58bca17061a (DEFAULT_TABLE table=default.Table_4713741287e34e5d986ffb272ed6291d [id=38e955d6bd644d24a7c45172ab60282b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.707903 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f7518b1e36147a9b9f0c58bca17061a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.708770 24040 tablet_service.cc:1467] Processing CreateTablet for tablet a15ff3120b1943a69ddb609d0575ae1a (DEFAULT_TABLE table=default.Table_bc6b3019e44547cda285ac0021565619 [id=c08bf22592064123bb9cab28935d81a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.709249 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a15ff3120b1943a69ddb609d0575ae1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.710582 24181 tablet_replica.cc:330] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:34.710850 24181 raft_consensus.cc:2229] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:34.711226 24181 pending_rounds.cc:62] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5: Trying to abort 1 pending ops.
I20240502 08:16:34.711308 24181 pending_rounds.cc:68] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156405683220480 op_type: NO_OP noop_request { }
I20240502 08:16:34.711484 24181 raft_consensus.cc:2866] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:34.711617 24181 raft_consensus.cc:2258] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.719121 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 223c92ae8f63402f8643e03d914e8e7e (DEFAULT_TABLE table=default.Table_00eb74cd24954c52b5484751f035f4c1 [id=b42e15a2350f4c9a8097ccb0cf1aaddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.719627 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 223c92ae8f63402f8643e03d914e8e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.722470 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea2dee2d5914cf0bf4b37d16865e9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.723644 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5160fda2643e4b80bc401bc48f97eff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.729197 23766 tablet_service.cc:1467] Processing CreateTablet for tablet a15ff3120b1943a69ddb609d0575ae1a (DEFAULT_TABLE table=default.Table_bc6b3019e44547cda285ac0021565619 [id=c08bf22592064123bb9cab28935d81a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.729843 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a15ff3120b1943a69ddb609d0575ae1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.729786 23903 tablet_service.cc:1467] Processing CreateTablet for tablet c316e1aeab224634b760641a4ce188c8 (DEFAULT_TABLE table=default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.732439 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 6f7518b1e36147a9b9f0c58bca17061a (DEFAULT_TABLE table=default.Table_4713741287e34e5d986ffb272ed6291d [id=38e955d6bd644d24a7c45172ab60282b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.734551 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5def9306dfba4d019d3acece81df2f86" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:34.735772 23723 leader_election.cc:341] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:34.737636 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 223c92ae8f63402f8643e03d914e8e7e (DEFAULT_TABLE table=default.Table_00eb74cd24954c52b5484751f035f4c1 [id=b42e15a2350f4c9a8097ccb0cf1aaddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.738174 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 223c92ae8f63402f8643e03d914e8e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.739470 23723 leader_election.cc:302] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:34.739840 24244 raft_consensus.cc:2726] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:34.746309 23896 tablet_service.cc:1467] Processing CreateTablet for tablet d8324e9b796047428e2fe918f07f9b32 (DEFAULT_TABLE table=default.Table_1956c23bde184dd4ba01ad0231165eb6 [id=e3f148f833bc4d99a3ab2ec8d5258507]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.746764 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 223c92ae8f63402f8643e03d914e8e7e (DEFAULT_TABLE table=default.Table_00eb74cd24954c52b5484751f035f4c1 [id=b42e15a2350f4c9a8097ccb0cf1aaddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.751904 23897 tablet_service.cc:1467] Processing CreateTablet for tablet a15ff3120b1943a69ddb609d0575ae1a (DEFAULT_TABLE table=default.Table_bc6b3019e44547cda285ac0021565619 [id=c08bf22592064123bb9cab28935d81a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.752611 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c316e1aeab224634b760641a4ce188c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.753082 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f7518b1e36147a9b9f0c58bca17061a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.753463 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8324e9b796047428e2fe918f07f9b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.764596 24179 ts_tablet_manager.cc:1888] T bf7678dbc7e641dfa259d36101d0dd5a P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.764817 24179 log.cc:1196] T bf7678dbc7e641dfa259d36101d0dd5a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/bf7678dbc7e641dfa259d36101d0dd5a
I20240502 08:16:34.765267 24179 ts_tablet_manager.cc:1909] T bf7678dbc7e641dfa259d36101d0dd5a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:34.765950 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet bf7678dbc7e641dfa259d36101d0dd5a (table default.Table_c60e3fa5881347d09ba464e544d7cbf8 [id=fac14487dd8840d0be3dcce182527117]) successfully deleted
I20240502 08:16:34.766355 24179 tablet_replica.cc:330] T 315263b66700471b982072b335eb70c4 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:34.766682 24179 raft_consensus.cc:2229] T 315263b66700471b982072b335eb70c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.766925 24179 raft_consensus.cc:2258] T 315263b66700471b982072b335eb70c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.767621 24179 ts_tablet_manager.cc:1875] T 315263b66700471b982072b335eb70c4 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.770226 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 223c92ae8f63402f8643e03d914e8e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.774278 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 485c772245fe48a39fc6ceb6377ac0b6 (DEFAULT_TABLE table=default.Table_0170ebbe1b274aed934c0dd0445f3271 [id=7dd2b25a6676488d8f3a8e2e6b65046e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.774698 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485c772245fe48a39fc6ceb6377ac0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.775009 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 9b1eb762780c450bbaf380d17db564b6 (DEFAULT_TABLE table=default.Table_457be8c1c0be48dca271ddd6b1a95561 [id=ba56850714c2422eb5dbbc0f49df7ce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.775158 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a15ff3120b1943a69ddb609d0575ae1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.775341 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b1eb762780c450bbaf380d17db564b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.780275 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 9b1eb762780c450bbaf380d17db564b6 (DEFAULT_TABLE table=default.Table_457be8c1c0be48dca271ddd6b1a95561 [id=ba56850714c2422eb5dbbc0f49df7ce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.780709 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b1eb762780c450bbaf380d17db564b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.781920 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 485c772245fe48a39fc6ceb6377ac0b6 (DEFAULT_TABLE table=default.Table_0170ebbe1b274aed934c0dd0445f3271 [id=7dd2b25a6676488d8f3a8e2e6b65046e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.782366 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485c772245fe48a39fc6ceb6377ac0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.782661 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 9b1eb762780c450bbaf380d17db564b6 (DEFAULT_TABLE table=default.Table_457be8c1c0be48dca271ddd6b1a95561 [id=ba56850714c2422eb5dbbc0f49df7ce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.783054 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b1eb762780c450bbaf380d17db564b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.789901 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 485c772245fe48a39fc6ceb6377ac0b6 (DEFAULT_TABLE table=default.Table_0170ebbe1b274aed934c0dd0445f3271 [id=7dd2b25a6676488d8f3a8e2e6b65046e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.790349 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485c772245fe48a39fc6ceb6377ac0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.798924 24175 tablet_bootstrap.cc:654] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.814636 24181 ts_tablet_manager.cc:1875] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.823853 24184 raft_consensus.cc:686] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:34.824441 24184 consensus_queue.cc:237] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:34.830567 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7e2b03f5ed46441d9c62c30319eab83b" } modify_external_catalogs: true
I20240502 08:16:34.830772 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7e2b03f5ed46441d9c62c30319eab83b" } modify_external_catalogs: true
I20240502 08:16:34.830523 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bc6b3019e44547cda285ac0021565619" } modify_external_catalogs: true
I20240502 08:16:34.833405 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bc6b3019e44547cda285ac0021565619" } modify_external_catalogs: true
I20240502 08:16:34.836043 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_34b1b262bb424bde8bd994ada1f03434"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.836549 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34b1b262bb424bde8bd994ada1f03434 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.838788 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 5160fda2643e4b80bc401bc48f97eff9
I20240502 08:16:34.839844 23904 tablet_service.cc:1514] Processing DeleteTablet for tablet 5160fda2643e4b80bc401bc48f97eff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.840646 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 5160fda2643e4b80bc401bc48f97eff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.840775 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_121d603047c14dacbb0f9442a22c23f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.841213 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_121d603047c14dacbb0f9442a22c23f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:34.842777 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 5160fda2643e4b80bc401bc48f97eff9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5160fda2643e4b80bc401bc48f97eff9 already in progress: creating tablet
I20240502 08:16:34.843299 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet a15ff3120b1943a69ddb609d0575ae1a
I20240502 08:16:34.845278 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet a15ff3120b1943a69ddb609d0575ae1a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.845759 23897 tablet_service.cc:1514] Processing DeleteTablet for tablet a15ff3120b1943a69ddb609d0575ae1a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.845782 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_3f6d07700a2e4aa1b7fb356c54c14b73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.846268 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f6d07700a2e4aa1b7fb356c54c14b73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.846788 24175 tablet_bootstrap.cc:492] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:34.846932 24175 ts_tablet_manager.cc:1367] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.181s user 0.004s sys 0.000s
W20240502 08:16:34.847412 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet a15ff3120b1943a69ddb609d0575ae1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a15ff3120b1943a69ddb609d0575ae1a already in progress: creating tablet
I20240502 08:16:34.847877 24175 raft_consensus.cc:348] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.848140 24175 raft_consensus.cc:374] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.848234 24175 raft_consensus.cc:729] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.848517 24175 consensus_queue.cc:260] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.851223 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2107771cb030402280a54f03d8b2bce4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.851658 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2107771cb030402280a54f03d8b2bce4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.852157 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b01c340249ea483683e385a7fa964a51" } modify_external_catalogs: true
I20240502 08:16:34.852303 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b01c340249ea483683e385a7fa964a51" } modify_external_catalogs: true
I20240502 08:16:34.852823 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_07371eaf91cb4ab199af62382b640d64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.853332 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07371eaf91cb4ab199af62382b640d64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.853415 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_4b967ef2b2fc4e2caa4219afcae841a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.853888 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b967ef2b2fc4e2caa4219afcae841a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.852949 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_00eb74cd24954c52b5484751f035f4c1" } modify_external_catalogs: true
I20240502 08:16:34.855454 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_00eb74cd24954c52b5484751f035f4c1" } modify_external_catalogs: true
I20240502 08:16:34.858531 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6ecc8a33231743169b89cf2bd44a49e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:34.858949 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ecc8a33231743169b89cf2bd44a49e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.859294 24173 tablet_bootstrap.cc:492] T 74aaea4b05e84f2ab31a0439f82016c1 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:34.859433 24173 ts_tablet_manager.cc:1367] T 74aaea4b05e84f2ab31a0439f82016c1 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.372s user 0.077s sys 0.106s
I20240502 08:16:34.860278 24173 raft_consensus.cc:348] T 74aaea4b05e84f2ab31a0439f82016c1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.860486 24173 raft_consensus.cc:374] T 74aaea4b05e84f2ab31a0439f82016c1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.860556 24173 raft_consensus.cc:729] T 74aaea4b05e84f2ab31a0439f82016c1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.860756 24173 consensus_queue.cc:260] T 74aaea4b05e84f2ab31a0439f82016c1 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.860987 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c84ff62b97d44dd2ad4a1d9927383457"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:34.861250 24173 ts_tablet_manager.cc:1398] T 74aaea4b05e84f2ab31a0439f82016c1 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240502 08:16:34.861418 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c84ff62b97d44dd2ad4a1d9927383457 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.861769 24173 tablet_bootstrap.cc:492] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:34.863250 24173 tablet_bootstrap.cc:654] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.867120 24179 ts_tablet_manager.cc:1888] T 315263b66700471b982072b335eb70c4 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.867400 24179 log.cc:1196] T 315263b66700471b982072b335eb70c4 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/315263b66700471b982072b335eb70c4
I20240502 08:16:34.867879 24179 ts_tablet_manager.cc:1909] T 315263b66700471b982072b335eb70c4 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:34.868500 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet a15ff3120b1943a69ddb609d0575ae1a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.869721 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 5160fda2643e4b80bc401bc48f97eff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.870113 23767 tablet_service.cc:1467] Processing CreateTablet for tablet c709efdd2ce74b5a8ab7278b402ebfe9 (DEFAULT_TABLE table=default.Table_34b1b262bb424bde8bd994ada1f03434 [id=322a04a326c24fc4ad2c3f1384db0fd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.870549 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c709efdd2ce74b5a8ab7278b402ebfe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.872124 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_12fab9a5f68349de8d48fb2acf1da1ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:34.872387 24175 ts_tablet_manager.cc:1398] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
W20240502 08:16:34.872603 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12fab9a5f68349de8d48fb2acf1da1ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:34.873670 24173 tablet_bootstrap.cc:492] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:34.874480 24175 tablet_bootstrap.cc:492] T bce9392a8ad64004ba6ce9a236976cd6 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:34.874709 24174 tablet_bootstrap.cc:492] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:34.874923 24174 ts_tablet_manager.cc:1367] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.272s user 0.026s sys 0.078s
I20240502 08:16:34.874819 23605 catalog_manager.cc:5484] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:34.875401 23605 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 9c08a89a72894f0b96d7863fd71edfd0 (table default.Table_d351b6717f35441cb5d9cf27f5c1087c [id=7338537cdd4249359d2af60930089412]). Expected version 2 got 0
I20240502 08:16:34.876058 24175 tablet_bootstrap.cc:654] T bce9392a8ad64004ba6ce9a236976cd6 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.876324 24174 raft_consensus.cc:348] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.876618 24174 raft_consensus.cc:374] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.876780 24174 raft_consensus.cc:729] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.877144 24174 consensus_queue.cc:260] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.878348 23606 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 68bdb761ff36426c96949e97279e2756 (table default.Table_50b02617337c4151bdc0ba38abd68144 [id=eca4080e8ebf46e9b58c85ee1d8675ca]). Expected version 1 got 0
I20240502 08:16:34.878441 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_295f41167e6e481ea2443d5f8d02a998"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:34.879000 23606 master_service.cc:946] ReplaceTablet: received request to replace tablet bce9392a8ad64004ba6ce9a236976cd6 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:34.879604 23606 master_service.cc:946] ReplaceTablet: received request to replace tablet 8791834597254e7c9f730e03568c0748 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:34.882388 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 315263b66700471b982072b335eb70c4 (table default.Table_4713741287e34e5d986ffb272ed6291d [id=38e955d6bd644d24a7c45172ab60282b]) successfully deleted
I20240502 08:16:34.882457 24174 ts_tablet_manager.cc:1398] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 08:16:34.882767 24174 tablet_bootstrap.cc:492] T e1cf5f66859d416baad2f767060197c8 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
W20240502 08:16:34.878998 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_295f41167e6e481ea2443d5f8d02a998 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:34.884150 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 5160fda2643e4b80bc401bc48f97eff9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5160fda2643e4b80bc401bc48f97eff9 already in progress: creating tablet
I20240502 08:16:34.884439 24174 tablet_bootstrap.cc:654] T e1cf5f66859d416baad2f767060197c8 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:34.884887 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet a15ff3120b1943a69ddb609d0575ae1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a15ff3120b1943a69ddb609d0575ae1a already in progress: creating tablet
I20240502 08:16:34.878015 24173 ts_tablet_manager.cc:1367] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20240502 08:16:34.878772 24037 tablet_service.cc:1467] Processing CreateTablet for tablet c709efdd2ce74b5a8ab7278b402ebfe9 (DEFAULT_TABLE table=default.Table_34b1b262bb424bde8bd994ada1f03434 [id=322a04a326c24fc4ad2c3f1384db0fd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.886013 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c709efdd2ce74b5a8ab7278b402ebfe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.886243 24173 raft_consensus.cc:348] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.886446 24173 raft_consensus.cc:374] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.886538 24173 raft_consensus.cc:729] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.886801 24173 consensus_queue.cc:260] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.887351 24173 ts_tablet_manager.cc:1398] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:34.887946 24173 tablet_bootstrap.cc:492] T 8dc8158255194b868af917c33c44f0f5 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:34.889612 24173 tablet_bootstrap.cc:654] T 8dc8158255194b868af917c33c44f0f5 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.890038 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet 5def9306dfba4d019d3acece81df2f86 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:34.890710 24181 ts_tablet_manager.cc:1888] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:34.890807 23608 master_service.cc:946] ReplaceTablet: received request to replace tablet 71c0c2f3def643dfa99ff4efe65f6acf from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:34.890861 24181 log.cc:1196] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/315263b66700471b982072b335eb70c4
I20240502 08:16:34.891284 24181 ts_tablet_manager.cc:1909] T 315263b66700471b982072b335eb70c4 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:34.893191 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 315263b66700471b982072b335eb70c4 (table default.Table_4713741287e34e5d986ffb272ed6291d [id=38e955d6bd644d24a7c45172ab60282b]) successfully deleted
I20240502 08:16:34.894591 24174 tablet_bootstrap.cc:492] T e1cf5f66859d416baad2f767060197c8 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:34.894762 24174 ts_tablet_manager.cc:1367] T e1cf5f66859d416baad2f767060197c8 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.004s
I20240502 08:16:34.895632 23602 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 68bdb761ff36426c96949e97279e2756 (table default.Table_50b02617337c4151bdc0ba38abd68144 [id=eca4080e8ebf46e9b58c85ee1d8675ca]). Expected version 1 got 0
I20240502 08:16:34.895926 24174 raft_consensus.cc:348] T e1cf5f66859d416baad2f767060197c8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.896308 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet 74aaea4b05e84f2ab31a0439f82016c1 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:34.896124 24174 raft_consensus.cc:374] T e1cf5f66859d416baad2f767060197c8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.896724 24174 raft_consensus.cc:729] T e1cf5f66859d416baad2f767060197c8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.896962 24174 consensus_queue.cc:260] T e1cf5f66859d416baad2f767060197c8 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.897454 24174 ts_tablet_manager.cc:1398] T e1cf5f66859d416baad2f767060197c8 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:34.897812 24174 tablet_bootstrap.cc:492] T 6c0a4b216c7e474db1992a299ff8eeb4 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:34.898036 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d351b6717f35441cb5d9cf27f5c1087c" } modify_external_catalogs: true
I20240502 08:16:34.898238 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d351b6717f35441cb5d9cf27f5c1087c" } modify_external_catalogs: true
I20240502 08:16:34.899674 24174 tablet_bootstrap.cc:654] T 6c0a4b216c7e474db1992a299ff8eeb4 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.900560 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d8c99815c26041c882419d42c6c32d90" } new_table_name: "default.Table_3602f20179c64041a63ab185d9359de6" modify_external_catalogs: true
I20240502 08:16:34.901213 24173 tablet_bootstrap.cc:492] T 8dc8158255194b868af917c33c44f0f5 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:34.901345 24173 ts_tablet_manager.cc:1367] T 8dc8158255194b868af917c33c44f0f5 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
W20240502 08:16:34.902051 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 6c0a4b216c7e474db1992a299ff8eeb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c0a4b216c7e474db1992a299ff8eeb4 already in progress: creating tablet
I20240502 08:16:34.902070 24173 raft_consensus.cc:348] T 8dc8158255194b868af917c33c44f0f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.902252 24173 raft_consensus.cc:374] T 8dc8158255194b868af917c33c44f0f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.902357 24173 raft_consensus.cc:729] T 8dc8158255194b868af917c33c44f0f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.902588 24173 consensus_queue.cc:260] T 8dc8158255194b868af917c33c44f0f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.903123 24173 ts_tablet_manager.cc:1398] T 8dc8158255194b868af917c33c44f0f5 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:34.903393 24173 tablet_bootstrap.cc:492] T 6c0a4b216c7e474db1992a299ff8eeb4 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:34.904237 24175 tablet_bootstrap.cc:492] T bce9392a8ad64004ba6ce9a236976cd6 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:34.905000 24173 tablet_bootstrap.cc:654] T 6c0a4b216c7e474db1992a299ff8eeb4 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.905148 24175 ts_tablet_manager.cc:1367] T bce9392a8ad64004ba6ce9a236976cd6 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.002s
I20240502 08:16:34.905416 23902 tablet_service.cc:1467] Processing CreateTablet for tablet c709efdd2ce74b5a8ab7278b402ebfe9 (DEFAULT_TABLE table=default.Table_34b1b262bb424bde8bd994ada1f03434 [id=322a04a326c24fc4ad2c3f1384db0fd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:34.906394 24181 tablet_replica.cc:330] T 840aba1388ae4ff78fbb869d87dc1bf1 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:34.906605 24181 raft_consensus.cc:2229] T 840aba1388ae4ff78fbb869d87dc1bf1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.906821 24181 raft_consensus.cc:2258] T 840aba1388ae4ff78fbb869d87dc1bf1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:34.907464 24181 ts_tablet_manager.cc:1875] T 840aba1388ae4ff78fbb869d87dc1bf1 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.906021 24175 raft_consensus.cc:348] T bce9392a8ad64004ba6ce9a236976cd6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.907519 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c709efdd2ce74b5a8ab7278b402ebfe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:34.908479 24174 tablet_bootstrap.cc:492] T 6c0a4b216c7e474db1992a299ff8eeb4 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:34.908640 24174 ts_tablet_manager.cc:1367] T 6c0a4b216c7e474db1992a299ff8eeb4 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240502 08:16:34.907720 24175 raft_consensus.cc:374] T bce9392a8ad64004ba6ce9a236976cd6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.909313 24174 raft_consensus.cc:348] T 6c0a4b216c7e474db1992a299ff8eeb4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.909492 24174 raft_consensus.cc:374] T 6c0a4b216c7e474db1992a299ff8eeb4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.909575 24174 raft_consensus.cc:729] T 6c0a4b216c7e474db1992a299ff8eeb4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.909803 24174 consensus_queue.cc:260] T 6c0a4b216c7e474db1992a299ff8eeb4 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.910285 24174 ts_tablet_manager.cc:1398] T 6c0a4b216c7e474db1992a299ff8eeb4 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:34.910564 24174 tablet_bootstrap.cc:492] T 8dc8158255194b868af917c33c44f0f5 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:34.910840 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 8791834597254e7c9f730e03568c0748
I20240502 08:16:34.910784 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1956c23bde184dd4ba01ad0231165eb6" } new_table_name: "default.Table_adde4be2323646faa59e19d7e9d189e5" modify_external_catalogs: true
I20240502 08:16:34.910975 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e17eec8f2c1a4190945b434794635b52
I20240502 08:16:34.911383 23606 catalog_manager.cc:6554] ReplaceTablet: tablet 8791834597254e7c9f730e03568c0748 deleted and replaced by tablet e9592a041a9543e8b41f043927807131
I20240502 08:16:34.911634 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_53754b76a22f4f7ca0aebc5b98d31887" } new_table_name: "default.Table_bdbe807258a5454795eaff8c27bb17dd" modify_external_catalogs: true
I20240502 08:16:34.911792 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 223c92ae8f63402f8643e03d914e8e7e
I20240502 08:16:34.911763 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3ad20764d6174afbbf3c36e64babdb15" } new_table_name: "default.Table_e70e56a53c984b2e8066dd16caafde85" modify_external_catalogs: true
I20240502 08:16:34.912391 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_457be8c1c0be48dca271ddd6b1a95561" } new_table_name: "default.Table_c3fb1a324a4b4920a7799bbcf3229806" modify_external_catalogs: true
I20240502 08:16:34.913944 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet e17eec8f2c1a4190945b434794635b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.914105 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet e17eec8f2c1a4190945b434794635b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.914196 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 223c92ae8f63402f8643e03d914e8e7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.914331 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet 223c92ae8f63402f8643e03d914e8e7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.914856 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 8791834597254e7c9f730e03568c0748 with delete_type TABLET_DATA_DELETED (replaced by tablet e9592a041a9543e8b41f043927807131) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.915441 24174 tablet_bootstrap.cc:654] T 8dc8158255194b868af917c33c44f0f5 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:34.916735 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e17eec8f2c1a4190945b434794635b52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e17eec8f2c1a4190945b434794635b52 already in progress: creating tablet
W20240502 08:16:34.916988 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 223c92ae8f63402f8643e03d914e8e7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 223c92ae8f63402f8643e03d914e8e7e already in progress: creating tablet
W20240502 08:16:34.920092 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 405ea360814e4593a289d753821277ff consensus state has no leader
I20240502 08:16:34.920132 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 8791834597254e7c9f730e03568c0748 with delete_type TABLET_DATA_DELETED (replaced by tablet e9592a041a9543e8b41f043927807131) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:34.920857 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 8791834597254e7c9f730e03568c0748 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8791834597254e7c9f730e03568c0748 already in progress: creating tablet
I20240502 08:16:34.914580 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet 8791834597254e7c9f730e03568c0748 with delete_type TABLET_DATA_DELETED (replaced by tablet e9592a041a9543e8b41f043927807131) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.914666 23895 tablet_service.cc:1514] Processing DeleteTablet for tablet 223c92ae8f63402f8643e03d914e8e7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.914758 23903 tablet_service.cc:1514] Processing DeleteTablet for tablet e17eec8f2c1a4190945b434794635b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.923697 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 71c0c2f3def643dfa99ff4efe65f6acf
W20240502 08:16:34.924105 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 8791834597254e7c9f730e03568c0748 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8791834597254e7c9f730e03568c0748 already in progress: creating tablet
I20240502 08:16:34.924331 23608 catalog_manager.cc:6554] ReplaceTablet: tablet 71c0c2f3def643dfa99ff4efe65f6acf deleted and replaced by tablet 203ed05d0bdc4b1f99207423b7fc848b
W20240502 08:16:34.924593 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 223c92ae8f63402f8643e03d914e8e7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 223c92ae8f63402f8643e03d914e8e7e already in progress: creating tablet
W20240502 08:16:34.924850 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e17eec8f2c1a4190945b434794635b52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e17eec8f2c1a4190945b434794635b52 already in progress: creating tablet
I20240502 08:16:34.924997 23903 tablet_service.cc:1514] Processing DeleteTablet for tablet 71c0c2f3def643dfa99ff4efe65f6acf with delete_type TABLET_DATA_DELETED (replaced by tablet 203ed05d0bdc4b1f99207423b7fc848b) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:34.925810 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 71c0c2f3def643dfa99ff4efe65f6acf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71c0c2f3def643dfa99ff4efe65f6acf already in progress: creating tablet
I20240502 08:16:34.926241 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 71c0c2f3def643dfa99ff4efe65f6acf with delete_type TABLET_DATA_DELETED (replaced by tablet 203ed05d0bdc4b1f99207423b7fc848b) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.927284 24173 tablet_bootstrap.cc:492] T 6c0a4b216c7e474db1992a299ff8eeb4 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:34.927486 24173 ts_tablet_manager.cc:1367] T 6c0a4b216c7e474db1992a299ff8eeb4 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.001s
W20240502 08:16:34.927448 23595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.126:43265, user_credentials={real_user=slave}} blocked reactor thread for 52461.2us
I20240502 08:16:34.928387 24173 raft_consensus.cc:348] T 6c0a4b216c7e474db1992a299ff8eeb4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.928637 24173 raft_consensus.cc:374] T 6c0a4b216c7e474db1992a299ff8eeb4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.928704 24173 raft_consensus.cc:729] T 6c0a4b216c7e474db1992a299ff8eeb4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.928975 24173 consensus_queue.cc:260] T 6c0a4b216c7e474db1992a299ff8eeb4 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.929507 24173 ts_tablet_manager.cc:1398] T 6c0a4b216c7e474db1992a299ff8eeb4 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:34.929857 24173 tablet_bootstrap.cc:492] T 7a10f86a0f3f4385bef39eda14d4fa9f P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:34.931476 24173 tablet_bootstrap.cc:654] T 7a10f86a0f3f4385bef39eda14d4fa9f P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.933418 24173 tablet_bootstrap.cc:492] T 7a10f86a0f3f4385bef39eda14d4fa9f P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:34.933600 24173 ts_tablet_manager.cc:1367] T 7a10f86a0f3f4385bef39eda14d4fa9f P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 08:16:34.934383 24173 raft_consensus.cc:348] T 7a10f86a0f3f4385bef39eda14d4fa9f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.934597 24173 raft_consensus.cc:374] T 7a10f86a0f3f4385bef39eda14d4fa9f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.934679 24173 raft_consensus.cc:729] T 7a10f86a0f3f4385bef39eda14d4fa9f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.935994 24173 consensus_queue.cc:260] T 7a10f86a0f3f4385bef39eda14d4fa9f P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.936358 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 71c0c2f3def643dfa99ff4efe65f6acf with delete_type TABLET_DATA_DELETED (replaced by tablet 203ed05d0bdc4b1f99207423b7fc848b) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.936483 24173 ts_tablet_manager.cc:1398] T 7a10f86a0f3f4385bef39eda14d4fa9f P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:16:34.936733 24173 tablet_bootstrap.cc:492] T e1cf5f66859d416baad2f767060197c8 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:34.937059 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 71c0c2f3def643dfa99ff4efe65f6acf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71c0c2f3def643dfa99ff4efe65f6acf already in progress: creating tablet
I20240502 08:16:34.909356 24175 raft_consensus.cc:729] T bce9392a8ad64004ba6ce9a236976cd6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.937611 24175 consensus_queue.cc:260] T bce9392a8ad64004ba6ce9a236976cd6 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.937742 24181 ts_tablet_manager.cc:1888] T 840aba1388ae4ff78fbb869d87dc1bf1 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.937884 24181 log.cc:1196] T 840aba1388ae4ff78fbb869d87dc1bf1 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/840aba1388ae4ff78fbb869d87dc1bf1
I20240502 08:16:34.938287 24181 ts_tablet_manager.cc:1909] T 840aba1388ae4ff78fbb869d87dc1bf1 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:34.940426 23903 tablet_service.cc:1514] Processing DeleteTablet for tablet bce9392a8ad64004ba6ce9a236976cd6 with delete_type TABLET_DATA_DELETED (replaced by tablet c316e1aeab224634b760641a4ce188c8) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.941236 24174 tablet_bootstrap.cc:492] T 8dc8158255194b868af917c33c44f0f5 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:34.941404 24174 ts_tablet_manager.cc:1367] T 8dc8158255194b868af917c33c44f0f5 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.009s
I20240502 08:16:34.941784 24175 ts_tablet_manager.cc:1398] T bce9392a8ad64004ba6ce9a236976cd6 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.037s user 0.001s sys 0.001s
I20240502 08:16:34.942138 24174 raft_consensus.cc:348] T 8dc8158255194b868af917c33c44f0f5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.942376 24174 raft_consensus.cc:374] T 8dc8158255194b868af917c33c44f0f5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.942438 24174 raft_consensus.cc:729] T 8dc8158255194b868af917c33c44f0f5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.942744 24175 tablet_bootstrap.cc:492] T 74aaea4b05e84f2ab31a0439f82016c1 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:34.942682 24174 consensus_queue.cc:260] T 8dc8158255194b868af917c33c44f0f5 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.943089 24174 ts_tablet_manager.cc:1398] T 8dc8158255194b868af917c33c44f0f5 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:34.943410 24174 tablet_bootstrap.cc:492] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:34.944990 24174 tablet_bootstrap.cc:654] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:34.945765 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet bce9392a8ad64004ba6ce9a236976cd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bce9392a8ad64004ba6ce9a236976cd6 already in progress: creating tablet
I20240502 08:16:34.946310 24173 tablet_bootstrap.cc:654] T e1cf5f66859d416baad2f767060197c8 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.948529 23903 tablet_service.cc:1514] Processing DeleteTablet for tablet bce9392a8ad64004ba6ce9a236976cd6 with delete_type TABLET_DATA_DELETED (replaced by tablet c316e1aeab224634b760641a4ce188c8) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.948906 24179 tablet_replica.cc:330] T bce9392a8ad64004ba6ce9a236976cd6 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:34.949141 24179 raft_consensus.cc:2229] T bce9392a8ad64004ba6ce9a236976cd6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.949343 24179 raft_consensus.cc:2258] T bce9392a8ad64004ba6ce9a236976cd6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:34.949405 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ce5ccfd29724dc49afd0892b99f3169 consensus state has no leader
I20240502 08:16:34.949923 24184 raft_consensus.cc:482] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:34.950114 24184 raft_consensus.cc:504] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.950901 24184 leader_election.cc:288] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:34.952262 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bdb761ff36426c96949e97279e2756" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:34.952498 24060 raft_consensus.cc:2454] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:34.953488 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bdb761ff36426c96949e97279e2756" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:34.953739 23924 raft_consensus.cc:2454] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:34.954380 23723 leader_election.cc:302] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:34.955749 24179 ts_tablet_manager.cc:1875] T bce9392a8ad64004ba6ce9a236976cd6 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:34.957682 24174 tablet_bootstrap.cc:492] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:34.957741 24184 raft_consensus.cc:2781] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:34.957816 24184 raft_consensus.cc:482] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:34.957834 24174 ts_tablet_manager.cc:1367] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.005s
I20240502 08:16:34.957922 24184 raft_consensus.cc:3037] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:34.958070 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 consensus state has no leader
I20240502 08:16:34.958655 24174 raft_consensus.cc:348] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.958844 24174 raft_consensus.cc:374] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.958923 24174 raft_consensus.cc:729] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.959115 24174 consensus_queue.cc:260] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.959604 24174 ts_tablet_manager.cc:1398] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:34.959879 24174 tablet_bootstrap.cc:492] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:34.960844 24184 raft_consensus.cc:504] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
W20240502 08:16:34.963822 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b1eb762780c450bbaf380d17db564b6 consensus state has no leader
I20240502 08:16:34.963996 24175 tablet_bootstrap.cc:654] T 74aaea4b05e84f2ab31a0439f82016c1 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.961354 24174 tablet_bootstrap.cc:654] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.965286 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bdb761ff36426c96949e97279e2756" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:34.965503 23924 raft_consensus.cc:3037] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:34.968739 23924 raft_consensus.cc:2454] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:34.969331 23723 leader_election.cc:302] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:34.969659 24243 raft_consensus.cc:2781] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:34.969796 24243 raft_consensus.cc:686] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
W20240502 08:16:34.970150 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8324e9b796047428e2fe918f07f9b32 consensus state has no leader
I20240502 08:16:34.970125 24243 consensus_queue.cc:237] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.974543 24175 tablet_bootstrap.cc:492] T 74aaea4b05e84f2ab31a0439f82016c1 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:34.974745 24175 ts_tablet_manager.cc:1367] T 74aaea4b05e84f2ab31a0439f82016c1 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.001s
I20240502 08:16:34.975598 24175 raft_consensus.cc:348] T 74aaea4b05e84f2ab31a0439f82016c1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.975867 24175 raft_consensus.cc:374] T 74aaea4b05e84f2ab31a0439f82016c1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.976027 24175 raft_consensus.cc:729] T 74aaea4b05e84f2ab31a0439f82016c1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.976300 24175 consensus_queue.cc:260] T 74aaea4b05e84f2ab31a0439f82016c1 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.976591 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 9c08a89a72894f0b96d7863fd71edfd0
I20240502 08:16:34.976794 24175 ts_tablet_manager.cc:1398] T 74aaea4b05e84f2ab31a0439f82016c1 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:34.977114 24175 tablet_bootstrap.cc:492] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:34.977278 24179 ts_tablet_manager.cc:1888] T bce9392a8ad64004ba6ce9a236976cd6 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:34.977370 24179 log.cc:1196] T bce9392a8ad64004ba6ce9a236976cd6 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/bce9392a8ad64004ba6ce9a236976cd6
I20240502 08:16:34.978300 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bdb761ff36426c96949e97279e2756" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:34.978528 24060 raft_consensus.cc:3037] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:34.978749 24175 tablet_bootstrap.cc:654] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.982327 24174 tablet_bootstrap.cc:492] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:34.982496 24174 ts_tablet_manager.cc:1367] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.003s
I20240502 08:16:34.982766 24184 leader_election.cc:288] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:34.982895 24060 raft_consensus.cc:2454] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:34.983234 24174 raft_consensus.cc:348] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.983449 24174 raft_consensus.cc:374] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.983536 24174 raft_consensus.cc:729] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.983747 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c08a89a72894f0b96d7863fd71edfd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:34.983762 24174 consensus_queue.cc:260] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.984002 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c08a89a72894f0b96d7863fd71edfd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.984259 24174 ts_tablet_manager.cc:1398] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:34.984575 24174 tablet_bootstrap.cc:492] T 8791834597254e7c9f730e03568c0748 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:34.987120 24175 tablet_bootstrap.cc:492] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:34.987378 24175 ts_tablet_manager.cc:1367] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 08:16:34.988134 24175 raft_consensus.cc:348] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.988886 24175 raft_consensus.cc:374] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:34.989089 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet 74aaea4b05e84f2ab31a0439f82016c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:34.989104 24175 raft_consensus.cc:729] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:34.989464 24175 consensus_queue.cc:260] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:34.989953 24175 ts_tablet_manager.cc:1398] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:34.990247 24175 tablet_bootstrap.cc:492] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:34.992785 24179 ts_tablet_manager.cc:1909] T bce9392a8ad64004ba6ce9a236976cd6 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:34.995987 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 5def9306dfba4d019d3acece81df2f86 with delete_type TABLET_DATA_DELETED (replaced by tablet 223c92ae8f63402f8643e03d914e8e7e) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:34.996248 24180 tablet_replica.cc:330] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:34.996742 24174 tablet_bootstrap.cc:654] T 8791834597254e7c9f730e03568c0748 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:34.997426 24180 raft_consensus.cc:2229] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:34.999886 24180 raft_consensus.cc:2258] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.000023 23608 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 050c16739efb4037a4adc78a2ffd3cba (table default.Table_e0bf76fa2b2d451eb0102e9897669b6f [id=ba62a673079d424992daa0f6b0c5d6b9]). Expected version 1 got 0
I20240502 08:16:35.000653 24180 ts_tablet_manager.cc:1875] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.001076 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c08a89a72894f0b96d7863fd71edfd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.002171 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet bce9392a8ad64004ba6ce9a236976cd6 (table default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]) successfully deleted
I20240502 08:16:35.002609 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c0a4b216c7e474db1992a299ff8eeb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.002664 23902 tablet_service.cc:1467] Processing CreateTablet for tablet da4be1d381a64b7baeb1c60cb17d924a (DEFAULT_TABLE table=default.Table_121d603047c14dacbb0f9442a22c23f4 [id=d5c976b2debc472fa1d7c28ae73ca32a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.002888 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 1df565c1c6564efd88fa557c50522d7e (DEFAULT_TABLE table=default.Table_2107771cb030402280a54f03d8b2bce4 [id=0a35939b821f468c9c7c86425fd688a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.003042 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da4be1d381a64b7baeb1c60cb17d924a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.003232 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df565c1c6564efd88fa557c50522d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.003708 24179 tablet_replica.cc:330] T 9c08a89a72894f0b96d7863fd71edfd0 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:35.003934 24179 raft_consensus.cc:2229] T 9c08a89a72894f0b96d7863fd71edfd0 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.004122 24179 raft_consensus.cc:2258] T 9c08a89a72894f0b96d7863fd71edfd0 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.004680 24179 ts_tablet_manager.cc:1875] T 9c08a89a72894f0b96d7863fd71edfd0 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.005364 24181 tablet_replica.cc:330] T 74aaea4b05e84f2ab31a0439f82016c1 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:35.005575 24181 raft_consensus.cc:2229] T 74aaea4b05e84f2ab31a0439f82016c1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.005713 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dc8158255194b868af917c33c44f0f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.006129 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 840aba1388ae4ff78fbb869d87dc1bf1 (table default.Table_1d5afdec101c4b57864bc6d8eeefd20f [id=20e5e7a21d9f4439a3c3723a0fdfd264]) successfully deleted
I20240502 08:16:35.006603 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 1df565c1c6564efd88fa557c50522d7e (DEFAULT_TABLE table=default.Table_2107771cb030402280a54f03d8b2bce4 [id=0a35939b821f468c9c7c86425fd688a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.007015 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df565c1c6564efd88fa557c50522d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.007009 24180 ts_tablet_manager.cc:1888] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.007124 24180 log.cc:1196] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/5def9306dfba4d019d3acece81df2f86
I20240502 08:16:35.007498 24180 ts_tablet_manager.cc:1909] T 5def9306dfba4d019d3acece81df2f86 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:35.008450 24180 tablet_replica.cc:330] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:35.008620 24180 raft_consensus.cc:2229] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:35.008816 24180 pending_rounds.cc:62] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:16:35.008857 24180 pending_rounds.cc:68] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156407605530624 op_type: NO_OP noop_request { }
I20240502 08:16:35.008975 24180 raft_consensus.cc:2866] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:35.009056 24180 raft_consensus.cc:2258] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.009730 24180 ts_tablet_manager.cc:1875] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.010325 24037 tablet_service.cc:1467] Processing CreateTablet for tablet da4be1d381a64b7baeb1c60cb17d924a (DEFAULT_TABLE table=default.Table_121d603047c14dacbb0f9442a22c23f4 [id=d5c976b2debc472fa1d7c28ae73ca32a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.010658 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da4be1d381a64b7baeb1c60cb17d924a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.015120 24180 ts_tablet_manager.cc:1888] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:34.995252 24175 tablet_bootstrap.cc:654] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.017446 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 5def9306dfba4d019d3acece81df2f86 (table default.Table_00eb74cd24954c52b5484751f035f4c1 [id=b42e15a2350f4c9a8097ccb0cf1aaddc]) successfully deleted
I20240502 08:16:35.018807 23602 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 050c16739efb4037a4adc78a2ffd3cba (table default.Table_e0bf76fa2b2d451eb0102e9897669b6f [id=ba62a673079d424992daa0f6b0c5d6b9]). Expected version 1 got 0
I20240502 08:16:35.017827 23767 tablet_service.cc:1467] Processing CreateTablet for tablet da4be1d381a64b7baeb1c60cb17d924a (DEFAULT_TABLE table=default.Table_121d603047c14dacbb0f9442a22c23f4 [id=d5c976b2debc472fa1d7c28ae73ca32a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.020967 24173 tablet_bootstrap.cc:492] T e1cf5f66859d416baad2f767060197c8 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:35.020938 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da4be1d381a64b7baeb1c60cb17d924a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.021126 24173 ts_tablet_manager.cc:1367] T e1cf5f66859d416baad2f767060197c8 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.084s user 0.010s sys 0.012s
I20240502 08:16:35.021991 24173 raft_consensus.cc:348] T e1cf5f66859d416baad2f767060197c8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.022210 24173 raft_consensus.cc:374] T e1cf5f66859d416baad2f767060197c8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.022280 24173 raft_consensus.cc:729] T e1cf5f66859d416baad2f767060197c8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.022548 24173 consensus_queue.cc:260] T e1cf5f66859d416baad2f767060197c8 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.023022 24173 ts_tablet_manager.cc:1398] T e1cf5f66859d416baad2f767060197c8 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:16:35.023396 24173 tablet_bootstrap.cc:492] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:35.025053 24173 tablet_bootstrap.cc:654] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.018157 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 1df565c1c6564efd88fa557c50522d7e (DEFAULT_TABLE table=default.Table_2107771cb030402280a54f03d8b2bce4 [id=0a35939b821f468c9c7c86425fd688a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.025913 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df565c1c6564efd88fa557c50522d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.005744 24181 raft_consensus.cc:2258] T 74aaea4b05e84f2ab31a0439f82016c1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.028070 24181 ts_tablet_manager.cc:1875] T 74aaea4b05e84f2ab31a0439f82016c1 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.030630 23605 catalog_manager.cc:5484] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:35.031118 23605 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 68bdb761ff36426c96949e97279e2756 (table default.Table_50b02617337c4151bdc0ba38abd68144 [id=eca4080e8ebf46e9b58c85ee1d8675ca]). Expected version 1 got 0
I20240502 08:16:35.031327 23605 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 175adf0ecb504e28a3d39f001f977839 (table default.Table_9530003aa5b841b683920da217d58576 [id=a6a5f4c65807479392c01ac5c955c959]). Expected version 1 got 0
I20240502 08:16:35.031558 23605 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 050c16739efb4037a4adc78a2ffd3cba (table default.Table_e0bf76fa2b2d451eb0102e9897669b6f [id=ba62a673079d424992daa0f6b0c5d6b9]). Expected version 1 got 0
I20240502 08:16:35.034040 24180 log.cc:1196] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/9c08a89a72894f0b96d7863fd71edfd0
I20240502 08:16:35.034557 24180 ts_tablet_manager.cc:1909] T 9c08a89a72894f0b96d7863fd71edfd0 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:35.034575 24175 tablet_bootstrap.cc:492] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:35.034726 24175 ts_tablet_manager.cc:1367] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.001s
I20240502 08:16:35.035696 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 9c08a89a72894f0b96d7863fd71edfd0 (table default.Table_d351b6717f35441cb5d9cf27f5c1087c [id=7338537cdd4249359d2af60930089412]) successfully deleted
I20240502 08:16:35.036700 24173 tablet_bootstrap.cc:492] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:35.036928 24184 raft_consensus.cc:482] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:35.037084 24184 raft_consensus.cc:504] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.037737 24184 leader_election.cc:288] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:35.038363 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a10f86a0f3f4385bef39eda14d4fa9f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:35.038945 23723 leader_election.cc:341] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:35.039587 24175 raft_consensus.cc:348] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.039876 24175 raft_consensus.cc:374] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.040033 24175 raft_consensus.cc:729] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.040470 24173 ts_tablet_manager.cc:1367] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.003s
I20240502 08:16:35.040388 24175 consensus_queue.cc:260] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.040983 24175 ts_tablet_manager.cc:1398] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:16:35.041237 24173 raft_consensus.cc:348] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.041414 24173 raft_consensus.cc:374] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.041505 24173 raft_consensus.cc:729] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.041692 24175 tablet_bootstrap.cc:492] T e1cf5f66859d416baad2f767060197c8 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:35.041759 24173 consensus_queue.cc:260] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.044611 24173 ts_tablet_manager.cc:1398] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:16:35.044862 24175 tablet_bootstrap.cc:654] T e1cf5f66859d416baad2f767060197c8 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.044939 24173 tablet_bootstrap.cc:492] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:35.046736 24173 tablet_bootstrap.cc:654] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.039793 24181 ts_tablet_manager.cc:1888] T 74aaea4b05e84f2ab31a0439f82016c1 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.038429 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a10f86a0f3f4385bef39eda14d4fa9f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:35.047855 24181 log.cc:1196] T 74aaea4b05e84f2ab31a0439f82016c1 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/74aaea4b05e84f2ab31a0439f82016c1
I20240502 08:16:35.047957 24060 raft_consensus.cc:2454] T 7a10f86a0f3f4385bef39eda14d4fa9f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:35.048364 24181 ts_tablet_manager.cc:1909] T 74aaea4b05e84f2ab31a0439f82016c1 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:35.048835 23723 leader_election.cc:302] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:35.049278 24184 raft_consensus.cc:2781] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:35.049357 24184 raft_consensus.cc:482] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:35.049445 24184 raft_consensus.cc:3037] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:35.051144 24181 tablet_replica.cc:330] T 9c08a89a72894f0b96d7863fd71edfd0 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:35.051400 24181 raft_consensus.cc:2229] T 9c08a89a72894f0b96d7863fd71edfd0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.051581 24181 raft_consensus.cc:2258] T 9c08a89a72894f0b96d7863fd71edfd0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.051716 24184 raft_consensus.cc:504] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.052181 24181 ts_tablet_manager.cc:1875] T 9c08a89a72894f0b96d7863fd71edfd0 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.052330 24184 leader_election.cc:288] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:35.053643 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a10f86a0f3f4385bef39eda14d4fa9f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:35.054229 23723 leader_election.cc:341] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:35.054306 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a10f86a0f3f4385bef39eda14d4fa9f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:35.054502 24060 raft_consensus.cc:3037] T 7a10f86a0f3f4385bef39eda14d4fa9f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:35.056850 24060 raft_consensus.cc:2454] T 7a10f86a0f3f4385bef39eda14d4fa9f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:35.054098 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 74aaea4b05e84f2ab31a0439f82016c1 (table default.Table_39d3e9a8fea548abadf2eef7723684f2 [id=495bc1caa3e44d56abf4269f906ef74d]) successfully deleted
W20240502 08:16:35.057384 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet a15ff3120b1943a69ddb609d0575ae1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a15ff3120b1943a69ddb609d0575ae1a already in progress: creating tablet
W20240502 08:16:35.057621 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 5160fda2643e4b80bc401bc48f97eff9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5160fda2643e4b80bc401bc48f97eff9 already in progress: creating tablet
W20240502 08:16:35.057839 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e17eec8f2c1a4190945b434794635b52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e17eec8f2c1a4190945b434794635b52 already in progress: creating tablet
W20240502 08:16:35.057992 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 223c92ae8f63402f8643e03d914e8e7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 223c92ae8f63402f8643e03d914e8e7e already in progress: creating tablet
W20240502 08:16:35.058154 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 8791834597254e7c9f730e03568c0748 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8791834597254e7c9f730e03568c0748 already in progress: creating tablet
W20240502 08:16:35.058328 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 71c0c2f3def643dfa99ff4efe65f6acf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71c0c2f3def643dfa99ff4efe65f6acf already in progress: creating tablet
I20240502 08:16:35.059988 24183 raft_consensus.cc:482] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:35.060145 24183 raft_consensus.cc:504] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.061800 24183 leader_election.cc:288] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:35.066294 24181 ts_tablet_manager.cc:1888] T 9c08a89a72894f0b96d7863fd71edfd0 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.066500 24181 log.cc:1196] T 9c08a89a72894f0b96d7863fd71edfd0 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/9c08a89a72894f0b96d7863fd71edfd0
I20240502 08:16:35.066922 24181 ts_tablet_manager.cc:1909] T 9c08a89a72894f0b96d7863fd71edfd0 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:35.068251 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 9c08a89a72894f0b96d7863fd71edfd0 (table default.Table_d351b6717f35441cb5d9cf27f5c1087c [id=7338537cdd4249359d2af60930089412]) successfully deleted
I20240502 08:16:35.068677 24181 tablet_replica.cc:330] T 6c0a4b216c7e474db1992a299ff8eeb4 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:35.068970 24181 raft_consensus.cc:2229] T 6c0a4b216c7e474db1992a299ff8eeb4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.069180 24181 raft_consensus.cc:2258] T 6c0a4b216c7e474db1992a299ff8eeb4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.069918 24181 ts_tablet_manager.cc:1875] T 6c0a4b216c7e474db1992a299ff8eeb4 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.072655 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet e1cf5f66859d416baad2f767060197c8 with delete_type TABLET_DATA_DELETED (replaced by tablet a15ff3120b1943a69ddb609d0575ae1a) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.072932 24180 tablet_replica.cc:330] T e1cf5f66859d416baad2f767060197c8 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:35.073099 24180 raft_consensus.cc:2229] T e1cf5f66859d416baad2f767060197c8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.073271 24180 raft_consensus.cc:2258] T e1cf5f66859d416baad2f767060197c8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.074103 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dc8158255194b868af917c33c44f0f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.075613 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c0a4b216c7e474db1992a299ff8eeb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.075961 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 5def9306dfba4d019d3acece81df2f86 with delete_type TABLET_DATA_DELETED (replaced by tablet 223c92ae8f63402f8643e03d914e8e7e) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.076666 24180 ts_tablet_manager.cc:1875] T e1cf5f66859d416baad2f767060197c8 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.079784 24181 ts_tablet_manager.cc:1888] T 6c0a4b216c7e474db1992a299ff8eeb4 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.079946 24181 log.cc:1196] T 6c0a4b216c7e474db1992a299ff8eeb4 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/6c0a4b216c7e474db1992a299ff8eeb4
I20240502 08:16:35.080328 24181 ts_tablet_manager.cc:1909] T 6c0a4b216c7e474db1992a299ff8eeb4 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:35.081440 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 6c0a4b216c7e474db1992a299ff8eeb4 (table default.Table_1f432841712848378bc24e8050ac57c7 [id=a9cdc35f95ec47d9a41d375133749c88]) successfully deleted
I20240502 08:16:35.082939 23723 leader_election.cc:302] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:35.083365 24184 raft_consensus.cc:2781] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:35.083591 24184 raft_consensus.cc:686] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:35.083786 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet e1cf5f66859d416baad2f767060197c8 with delete_type TABLET_DATA_DELETED (replaced by tablet a15ff3120b1943a69ddb609d0575ae1a) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.084035 24184 consensus_queue.cc:237] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.086742 24181 tablet_replica.cc:330] T 8dc8158255194b868af917c33c44f0f5 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:35.087338 24181 raft_consensus.cc:2229] T 8dc8158255194b868af917c33c44f0f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.093084 24180 ts_tablet_manager.cc:1888] T e1cf5f66859d416baad2f767060197c8 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.093283 24180 log.cc:1196] T e1cf5f66859d416baad2f767060197c8 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/e1cf5f66859d416baad2f767060197c8
I20240502 08:16:35.093745 24180 ts_tablet_manager.cc:1909] T e1cf5f66859d416baad2f767060197c8 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:35.094832 24180 tablet_replica.cc:330] T 8dc8158255194b868af917c33c44f0f5 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:35.094887 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet e1cf5f66859d416baad2f767060197c8 (table default.Table_bc6b3019e44547cda285ac0021565619 [id=c08bf22592064123bb9cab28935d81a5]) successfully deleted
I20240502 08:16:35.095199 24180 raft_consensus.cc:2229] T 8dc8158255194b868af917c33c44f0f5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.095432 24180 raft_consensus.cc:2258] T 8dc8158255194b868af917c33c44f0f5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.096256 24180 ts_tablet_manager.cc:1875] T 8dc8158255194b868af917c33c44f0f5 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.101300 23608 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 175adf0ecb504e28a3d39f001f977839 (table default.Table_9530003aa5b841b683920da217d58576 [id=a6a5f4c65807479392c01ac5c955c959]). Expected version 1 got 0
I20240502 08:16:35.101773 24181 raft_consensus.cc:2258] T 8dc8158255194b868af917c33c44f0f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.103521 24181 ts_tablet_manager.cc:1875] T 8dc8158255194b868af917c33c44f0f5 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.103691 24037 tablet_service.cc:1467] Processing CreateTablet for tablet ba3d4b4584dd45b58ff0daa4f1c09a0a (DEFAULT_TABLE table=default.Table_12fab9a5f68349de8d48fb2acf1da1ba [id=fd781805eb4c45f09fe6460144472816]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.106572 23605 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 175adf0ecb504e28a3d39f001f977839 (table default.Table_9530003aa5b841b683920da217d58576 [id=a6a5f4c65807479392c01ac5c955c959]). Expected version 1 got 0
I20240502 08:16:35.107342 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3d4b4584dd45b58ff0daa4f1c09a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.109964 23767 tablet_service.cc:1467] Processing CreateTablet for tablet ba3d4b4584dd45b58ff0daa4f1c09a0a (DEFAULT_TABLE table=default.Table_12fab9a5f68349de8d48fb2acf1da1ba [id=fd781805eb4c45f09fe6460144472816]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.110879 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3d4b4584dd45b58ff0daa4f1c09a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.112048 23605 catalog_manager.cc:5484] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:35.113269 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 5738cd77bbe147b49215ba9639d7f547 (DEFAULT_TABLE table=default.Table_c84ff62b97d44dd2ad4a1d9927383457 [id=1bfba58d949847308b00af3275167658]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.113677 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5738cd77bbe147b49215ba9639d7f547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.120596 24180 ts_tablet_manager.cc:1888] T 8dc8158255194b868af917c33c44f0f5 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.121260 24180 log.cc:1196] T 8dc8158255194b868af917c33c44f0f5 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/8dc8158255194b868af917c33c44f0f5
I20240502 08:16:35.122123 24180 ts_tablet_manager.cc:1909] T 8dc8158255194b868af917c33c44f0f5 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:35.124307 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 8dc8158255194b868af917c33c44f0f5 (table default.Table_017d876815e94b87b8c60e535aa06ad5 [id=307769d00cda4d679da90a24bd461dfe]) successfully deleted
I20240502 08:16:35.124728 24180 tablet_replica.cc:330] T 6c0a4b216c7e474db1992a299ff8eeb4 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:35.125268 24180 raft_consensus.cc:2229] T 6c0a4b216c7e474db1992a299ff8eeb4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.125766 24180 raft_consensus.cc:2258] T 6c0a4b216c7e474db1992a299ff8eeb4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.126739 24180 ts_tablet_manager.cc:1875] T 6c0a4b216c7e474db1992a299ff8eeb4 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.129870 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_34b1b262bb424bde8bd994ada1f03434" } modify_external_catalogs: true
I20240502 08:16:35.130075 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_34b1b262bb424bde8bd994ada1f03434" } modify_external_catalogs: true
I20240502 08:16:35.130798 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e54897c4ce62463f9cf7baba62b2706a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.131289 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e54897c4ce62463f9cf7baba62b2706a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.132396 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_59a2066b429b47548476adbb6cbbedc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.132912 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59a2066b429b47548476adbb6cbbedc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.134119 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d8aa2cd6c42444c48c383bf13e3728fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.134701 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8aa2cd6c42444c48c383bf13e3728fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.139364 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 08183523bf4a4d1c9020eb693aa07878 (DEFAULT_TABLE table=default.Table_3f6d07700a2e4aa1b7fb356c54c14b73 [id=0a2f8dbb62a64b0a96931ab5f8c17540]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.140115 23767 tablet_service.cc:1467] Processing CreateTablet for tablet a7c3800fb2784a1dba12d01c56c226e5 (DEFAULT_TABLE table=default.Table_295f41167e6e481ea2443d5f8d02a998 [id=73559755c4594bcfa591d054964154ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.140671 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08183523bf4a4d1c9020eb693aa07878. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.141274 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c3800fb2784a1dba12d01c56c226e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.141860 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e4e01edd1fa74d61be1f1d3381342c19" } new_table_name: "default.Table_90cefac5d31b4c449140db4b7772a0d2" modify_external_catalogs: true
I20240502 08:16:35.143522 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b9e3e2982acf4ad7994dd28c78b1bb47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.144209 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9e3e2982acf4ad7994dd28c78b1bb47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.145475 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6c584db923e147d380281ecb1acd11a1" } modify_external_catalogs: true
I20240502 08:16:35.143078 23768 tablet_service.cc:1467] Processing CreateTablet for tablet e9592a041a9543e8b41f043927807131 (DEFAULT_TABLE table=default.Table_6c584db923e147d380281ecb1acd11a1 [id=467a7f6db73641ca85a1898b251938fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.145684 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6c584db923e147d380281ecb1acd11a1" } modify_external_catalogs: true
I20240502 08:16:35.146067 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9592a041a9543e8b41f043927807131. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.147436 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c84ff62b97d44dd2ad4a1d9927383457" } modify_external_catalogs: true
I20240502 08:16:35.147624 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c84ff62b97d44dd2ad4a1d9927383457" } modify_external_catalogs: true
I20240502 08:16:35.148828 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b96eb662e06846039807538794442017" } modify_external_catalogs: true
I20240502 08:16:35.148912 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1ce01bea1dbd40d18ca5073c848b506d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.149351 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ce01bea1dbd40d18ca5073c848b506d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.148983 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b96eb662e06846039807538794442017" } modify_external_catalogs: true
I20240502 08:16:35.165582 24180 ts_tablet_manager.cc:1888] T 6c0a4b216c7e474db1992a299ff8eeb4 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.165728 24180 log.cc:1196] T 6c0a4b216c7e474db1992a299ff8eeb4 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/6c0a4b216c7e474db1992a299ff8eeb4
I20240502 08:16:35.165871 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet c709efdd2ce74b5a8ab7278b402ebfe9
I20240502 08:16:35.166137 24180 ts_tablet_manager.cc:1909] T 6c0a4b216c7e474db1992a299ff8eeb4 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:35.167583 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9837c3c4dad548c18b554594be454c46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.168180 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9837c3c4dad548c18b554594be454c46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.168458 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet c709efdd2ce74b5a8ab7278b402ebfe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.169437 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 6c0a4b216c7e474db1992a299ff8eeb4 (table default.Table_1f432841712848378bc24e8050ac57c7 [id=a9cdc35f95ec47d9a41d375133749c88]) successfully deleted
W20240502 08:16:35.169790 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 5def9306dfba4d019d3acece81df2f86 because the tablet was not found. No further retry: Not found: Tablet not found: 5def9306dfba4d019d3acece81df2f86
W20240502 08:16:35.169997 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet c709efdd2ce74b5a8ab7278b402ebfe9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c709efdd2ce74b5a8ab7278b402ebfe9 already in progress: creating tablet
I20240502 08:16:35.170670 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_556db0363a3d4e609dedadc67c985fda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.171093 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_556db0363a3d4e609dedadc67c985fda in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.173895 24179 ts_tablet_manager.cc:1888] T 9c08a89a72894f0b96d7863fd71edfd0 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.174135 24179 log.cc:1196] T 9c08a89a72894f0b96d7863fd71edfd0 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/9c08a89a72894f0b96d7863fd71edfd0
I20240502 08:16:35.174576 24179 ts_tablet_manager.cc:1909] T 9c08a89a72894f0b96d7863fd71edfd0 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:35.181355 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_482a1b0b7a064e02805ac3b86fc62f99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.181869 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_482a1b0b7a064e02805ac3b86fc62f99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.187330 24033 tablet_service.cc:1467] Processing CreateTablet for tablet a7c3800fb2784a1dba12d01c56c226e5 (DEFAULT_TABLE table=default.Table_295f41167e6e481ea2443d5f8d02a998 [id=73559755c4594bcfa591d054964154ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.187846 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c3800fb2784a1dba12d01c56c226e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.190516 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 08183523bf4a4d1c9020eb693aa07878 (DEFAULT_TABLE table=default.Table_3f6d07700a2e4aa1b7fb356c54c14b73 [id=0a2f8dbb62a64b0a96931ab5f8c17540]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.190907 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08183523bf4a4d1c9020eb693aa07878. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.192554 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet c709efdd2ce74b5a8ab7278b402ebfe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.193871 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 5738cd77bbe147b49215ba9639d7f547
I20240502 08:16:35.194803 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9507475756c3401fa75fc34d4a4b7814"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.195381 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9507475756c3401fa75fc34d4a4b7814 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.195489 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 5738cd77bbe147b49215ba9639d7f547 (DEFAULT_TABLE table=default.Table_c84ff62b97d44dd2ad4a1d9927383457 [id=1bfba58d949847308b00af3275167658]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.196102 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5738cd77bbe147b49215ba9639d7f547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.194779 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0f3416651b83425bbfdd6a9adfc2709e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.196717 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f3416651b83425bbfdd6a9adfc2709e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:35.197460 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32150c7e130f4069a2a3a02e625bfb93 consensus state has no leader
I20240502 08:16:35.197881 23606 master_service.cc:946] ReplaceTablet: received request to replace tablet 774487be9bc644cab4e420c3d33b3cad from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:35.198338 23606 master_service.cc:946] ReplaceTablet: received request to replace tablet a15ff3120b1943a69ddb609d0575ae1a from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:35.198889 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4713741287e34e5d986ffb272ed6291d" } new_table_name: "default.Table_549acf5604c74ae2bac902a1118e5baa" modify_external_catalogs: true
I20240502 08:16:35.200973 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet c316e1aeab224634b760641a4ce188c8 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:35.202239 23610 master_service.cc:946] ReplaceTablet: received request to replace tablet 3052510d77f344bf87decb9c2e5ec0bd from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:35.208679 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet 5738cd77bbe147b49215ba9639d7f547 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.230979 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 5738cd77bbe147b49215ba9639d7f547 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:35.242442 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 5738cd77bbe147b49215ba9639d7f547 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5738cd77bbe147b49215ba9639d7f547 already in progress: creating tablet
I20240502 08:16:35.243388 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet c1c2a988a3524c8fb7410e8508bd1abc
I20240502 08:16:35.244521 23608 master_service.cc:946] ReplaceTablet: received request to replace tablet 6f7518b1e36147a9b9f0c58bca17061a from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:35.246100 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_121d603047c14dacbb0f9442a22c23f4" } new_table_name: "default.Table_1e7f2516f34b483d913cf5080a8febec" modify_external_catalogs: true
I20240502 08:16:35.246536 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5d8bead3c17041909cc72b6a7537570c" } new_table_name: "default.Table_0a242b17e3d2450eb3add7dbff397394" modify_external_catalogs: true
I20240502 08:16:35.247550 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 3052510d77f344bf87decb9c2e5ec0bd
I20240502 08:16:35.247887 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_50b02617337c4151bdc0ba38abd68144" } new_table_name: "default.Table_a757855f2b244c548d8c2add77baaff7" modify_external_catalogs: true
I20240502 08:16:35.248060 23610 catalog_manager.cc:6554] ReplaceTablet: tablet 3052510d77f344bf87decb9c2e5ec0bd deleted and replaced by tablet 23ce83c8001d4b2988b88c0ed9bf948c
I20240502 08:16:35.248396 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2107771cb030402280a54f03d8b2bce4" } modify_external_catalogs: true
I20240502 08:16:35.248558 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2107771cb030402280a54f03d8b2bce4" } modify_external_catalogs: true
W20240502 08:16:35.249975 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f7518b1e36147a9b9f0c58bca17061a consensus state has no leader
I20240502 08:16:35.253865 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet c316e1aeab224634b760641a4ce188c8
I20240502 08:16:35.254259 23609 catalog_manager.cc:6554] ReplaceTablet: tablet c316e1aeab224634b760641a4ce188c8 deleted and replaced by tablet d5bab1b7c48248a7973c27d0a865b34b
I20240502 08:16:35.261179 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e9592a041a9543e8b41f043927807131
I20240502 08:16:35.267446 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c2a988a3524c8fb7410e8508bd1abc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.268211 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 3052510d77f344bf87decb9c2e5ec0bd with delete_type TABLET_DATA_DELETED (replaced by tablet 23ce83c8001d4b2988b88c0ed9bf948c) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.268842 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet c316e1aeab224634b760641a4ce188c8 with delete_type TABLET_DATA_DELETED (replaced by tablet d5bab1b7c48248a7973c27d0a865b34b) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.269356 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet e9592a041a9543e8b41f043927807131 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.270099 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 642744194e22461f8316a78689fd99ff (DEFAULT_TABLE table=default.Table_6ecc8a33231743169b89cf2bd44a49e6 [id=18ab2c0716464d82a0e33bd99bd0735d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.270529 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642744194e22461f8316a78689fd99ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.285910 24031 tablet_service.cc:1467] Processing CreateTablet for tablet b63fe5b6d4db4750800ecd429fa97ed1 (DEFAULT_TABLE table=default.Table_07371eaf91cb4ab199af62382b640d64 [id=0026ee826a3e489087f873579507c110]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.287847 23768 tablet_service.cc:1467] Processing CreateTablet for tablet f36ceb0222ef473a828ca795f393174d (DEFAULT_TABLE table=default.Table_4b967ef2b2fc4e2caa4219afcae841a6 [id=c32fc96889554c32bdf1d4550633e91f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.288353 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f36ceb0222ef473a828ca795f393174d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.292960 23767 tablet_service.cc:1467] Processing CreateTablet for tablet b63fe5b6d4db4750800ecd429fa97ed1 (DEFAULT_TABLE table=default.Table_07371eaf91cb4ab199af62382b640d64 [id=0026ee826a3e489087f873579507c110]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.293292 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b63fe5b6d4db4750800ecd429fa97ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.294099 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 203ed05d0bdc4b1f99207423b7fc848b (DEFAULT_TABLE table=default.Table_68bf91a14ffa42cdbb10a1d7ca56cb08 [id=59d945a73daa4918ba17063eae0b9b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.294456 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 203ed05d0bdc4b1f99207423b7fc848b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.299253 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b63fe5b6d4db4750800ecd429fa97ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.301334 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 203ed05d0bdc4b1f99207423b7fc848b (DEFAULT_TABLE table=default.Table_68bf91a14ffa42cdbb10a1d7ca56cb08 [id=59d945a73daa4918ba17063eae0b9b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.301719 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 203ed05d0bdc4b1f99207423b7fc848b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.302496 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 3052510d77f344bf87decb9c2e5ec0bd with delete_type TABLET_DATA_DELETED (replaced by tablet 23ce83c8001d4b2988b88c0ed9bf948c) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.303092 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet e9592a041a9543e8b41f043927807131 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.303342 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 642744194e22461f8316a78689fd99ff (DEFAULT_TABLE table=default.Table_6ecc8a33231743169b89cf2bd44a49e6 [id=18ab2c0716464d82a0e33bd99bd0735d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.303215 24040 tablet_service.cc:1467] Processing CreateTablet for tablet f36ceb0222ef473a828ca795f393174d (DEFAULT_TABLE table=default.Table_4b967ef2b2fc4e2caa4219afcae841a6 [id=c32fc96889554c32bdf1d4550633e91f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.325510 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet c316e1aeab224634b760641a4ce188c8 with delete_type TABLET_DATA_DELETED (replaced by tablet d5bab1b7c48248a7973c27d0a865b34b) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.327721 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f36ceb0222ef473a828ca795f393174d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.329902 24032 tablet_service.cc:1467] Processing CreateTablet for tablet e9592a041a9543e8b41f043927807131 (DEFAULT_TABLE table=default.Table_6c584db923e147d380281ecb1acd11a1 [id=467a7f6db73641ca85a1898b251938fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.334659 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642744194e22461f8316a78689fd99ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.341807 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9592a041a9543e8b41f043927807131. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.349298 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c2a988a3524c8fb7410e8508bd1abc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.358709 24174 tablet_bootstrap.cc:492] T 8791834597254e7c9f730e03568c0748 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:35.358867 24174 ts_tablet_manager.cc:1367] T 8791834597254e7c9f730e03568c0748 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.374s user 0.035s sys 0.093s
I20240502 08:16:35.282173 23903 tablet_service.cc:1514] Processing DeleteTablet for tablet 5def9306dfba4d019d3acece81df2f86 with delete_type TABLET_DATA_DELETED (replaced by tablet 223c92ae8f63402f8643e03d914e8e7e) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.282933 23902 tablet_service.cc:1467] Processing CreateTablet for tablet ba3d4b4584dd45b58ff0daa4f1c09a0a (DEFAULT_TABLE table=default.Table_12fab9a5f68349de8d48fb2acf1da1ba [id=fd781805eb4c45f09fe6460144472816]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.359371 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3d4b4584dd45b58ff0daa4f1c09a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.359536 24174 raft_consensus.cc:348] T 8791834597254e7c9f730e03568c0748 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.359750 24174 raft_consensus.cc:374] T 8791834597254e7c9f730e03568c0748 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.359829 24174 raft_consensus.cc:729] T 8791834597254e7c9f730e03568c0748 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.360008 24174 consensus_queue.cc:260] T 8791834597254e7c9f730e03568c0748 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.360162 24179 tablet_replica.cc:330] T 74aaea4b05e84f2ab31a0439f82016c1 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:35.360345 24179 raft_consensus.cc:2229] T 74aaea4b05e84f2ab31a0439f82016c1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.360425 24174 ts_tablet_manager.cc:1398] T 8791834597254e7c9f730e03568c0748 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:16:35.360513 24179 raft_consensus.cc:2258] T 74aaea4b05e84f2ab31a0439f82016c1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:35.361109 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet c1c2a988a3524c8fb7410e8508bd1abc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1c2a988a3524c8fb7410e8508bd1abc already in progress: creating tablet
I20240502 08:16:35.361294 24179 ts_tablet_manager.cc:1875] T 74aaea4b05e84f2ab31a0439f82016c1 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:35.361327 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 3052510d77f344bf87decb9c2e5ec0bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3052510d77f344bf87decb9c2e5ec0bd already in progress: creating tablet
W20240502 08:16:35.361521 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet c316e1aeab224634b760641a4ce188c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c316e1aeab224634b760641a4ce188c8 already in progress: creating tablet
W20240502 08:16:35.361739 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e9592a041a9543e8b41f043927807131 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9592a041a9543e8b41f043927807131 already in progress: creating tablet
I20240502 08:16:35.283126 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 5738cd77bbe147b49215ba9639d7f547 (DEFAULT_TABLE table=default.Table_c84ff62b97d44dd2ad4a1d9927383457 [id=1bfba58d949847308b00af3275167658]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.365691 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5738cd77bbe147b49215ba9639d7f547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.366700 24174 tablet_bootstrap.cc:492] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:35.368711 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_dd3c838f4a014e178351d135f5d4567d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.369251 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd3c838f4a014e178351d135f5d4567d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.283540 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 08183523bf4a4d1c9020eb693aa07878 (DEFAULT_TABLE table=default.Table_3f6d07700a2e4aa1b7fb356c54c14b73 [id=0a2f8dbb62a64b0a96931ab5f8c17540]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.370002 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08183523bf4a4d1c9020eb693aa07878. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.370945 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_90cefac5d31b4c449140db4b7772a0d2" } modify_external_catalogs: true
I20240502 08:16:35.371112 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_90cefac5d31b4c449140db4b7772a0d2" } modify_external_catalogs: true
I20240502 08:16:35.283877 23901 tablet_service.cc:1467] Processing CreateTablet for tablet a7c3800fb2784a1dba12d01c56c226e5 (DEFAULT_TABLE table=default.Table_295f41167e6e481ea2443d5f8d02a998 [id=73559755c4594bcfa591d054964154ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.374105 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c3800fb2784a1dba12d01c56c226e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.378356 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 9c08a89a72894f0b96d7863fd71edfd0 (table default.Table_d351b6717f35441cb5d9cf27f5c1087c [id=7338537cdd4249359d2af60930089412]) successfully deleted
I20240502 08:16:35.284302 23899 tablet_service.cc:1467] Processing CreateTablet for tablet e9592a041a9543e8b41f043927807131 (DEFAULT_TABLE table=default.Table_6c584db923e147d380281ecb1acd11a1 [id=467a7f6db73641ca85a1898b251938fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.284564 23897 tablet_service.cc:1514] Processing DeleteTablet for tablet c709efdd2ce74b5a8ab7278b402ebfe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.379220 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9592a041a9543e8b41f043927807131. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.379695 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_75cadbecaad04383945a6e0b1e226d86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.380178 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75cadbecaad04383945a6e0b1e226d86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.284595 23891 tablet_service.cc:1467] Processing CreateTablet for tablet f36ceb0222ef473a828ca795f393174d (DEFAULT_TABLE table=default.Table_4b967ef2b2fc4e2caa4219afcae841a6 [id=c32fc96889554c32bdf1d4550633e91f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.284749 23904 tablet_service.cc:1514] Processing DeleteTablet for tablet 5738cd77bbe147b49215ba9639d7f547 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.284881 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c2a988a3524c8fb7410e8508bd1abc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.285027 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 3052510d77f344bf87decb9c2e5ec0bd with delete_type TABLET_DATA_DELETED (replaced by tablet 23ce83c8001d4b2988b88c0ed9bf948c) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.285194 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet c316e1aeab224634b760641a4ce188c8 with delete_type TABLET_DATA_DELETED (replaced by tablet d5bab1b7c48248a7973c27d0a865b34b) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.285331 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet e9592a041a9543e8b41f043927807131 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.285650 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 642744194e22461f8316a78689fd99ff (DEFAULT_TABLE table=default.Table_6ecc8a33231743169b89cf2bd44a49e6 [id=18ab2c0716464d82a0e33bd99bd0735d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.382615 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f36ceb0222ef473a828ca795f393174d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.382649 24211 raft_consensus.cc:482] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:35.382822 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642744194e22461f8316a78689fd99ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.382797 24211 raft_consensus.cc:504] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.383505 24211 leader_election.cc:288] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:35.384254 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175adf0ecb504e28a3d39f001f977839" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:35.384851 23996 leader_election.cc:341] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:35.286037 23898 tablet_service.cc:1467] Processing CreateTablet for tablet b63fe5b6d4db4750800ecd429fa97ed1 (DEFAULT_TABLE table=default.Table_07371eaf91cb4ab199af62382b640d64 [id=0026ee826a3e489087f873579507c110]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.385948 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b63fe5b6d4db4750800ecd429fa97ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.286496 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 203ed05d0bdc4b1f99207423b7fc848b (DEFAULT_TABLE table=default.Table_68bf91a14ffa42cdbb10a1d7ca56cb08 [id=59d945a73daa4918ba17063eae0b9b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.388371 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 203ed05d0bdc4b1f99207423b7fc848b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.393803 24179 ts_tablet_manager.cc:1888] T 74aaea4b05e84f2ab31a0439f82016c1 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240502 08:16:35.394141 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 768167b977c14b53a7537879e349d354 consensus state has no leader
W20240502 08:16:35.395931 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da4be1d381a64b7baeb1c60cb17d924a consensus state has no leader
I20240502 08:16:35.396365 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 1df565c1c6564efd88fa557c50522d7e
I20240502 08:16:35.397621 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 6f7518b1e36147a9b9f0c58bca17061a
I20240502 08:16:35.397706 24179 log.cc:1196] T 74aaea4b05e84f2ab31a0439f82016c1 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/74aaea4b05e84f2ab31a0439f82016c1
I20240502 08:16:35.398111 23608 catalog_manager.cc:6554] ReplaceTablet: tablet 6f7518b1e36147a9b9f0c58bca17061a deleted and replaced by tablet d8f7b5f018d84680ac5533bf99102d31
I20240502 08:16:35.399947 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f7518b1e36147a9b9f0c58bca17061a with delete_type TABLET_DATA_DELETED (replaced by tablet d8f7b5f018d84680ac5533bf99102d31) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.400270 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df565c1c6564efd88fa557c50522d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.402734 24184 raft_consensus.cc:482] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:35.403024 24179 ts_tablet_manager.cc:1909] T 74aaea4b05e84f2ab31a0439f82016c1 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:35.402912 24184 raft_consensus.cc:504] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.403571 24184 leader_election.cc:288] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:35.403921 24174 tablet_bootstrap.cc:654] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.404980 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175adf0ecb504e28a3d39f001f977839" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:35.405228 24060 raft_consensus.cc:2454] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:35.406797 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_59a2066b429b47548476adbb6cbbedc7" } modify_external_catalogs: true
I20240502 08:16:35.407122 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_59a2066b429b47548476adbb6cbbedc7" } modify_external_catalogs: true
I20240502 08:16:35.409600 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c422ad5c5f164908bfae0d2977635bb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.410140 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c422ad5c5f164908bfae0d2977635bb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.412101 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_18007012fdb34e80ad040694f0f110bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.412561 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18007012fdb34e80ad040694f0f110bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.414907 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_556db0363a3d4e609dedadc67c985fda" } new_table_name: "default.Table_da5e259b0d0c47cf94ab30e3e98d1540" modify_external_catalogs: true
I20240502 08:16:35.414906 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_178527d31d7044fc98fa41886d651049"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.415441 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_178527d31d7044fc98fa41886d651049 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.417660 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175adf0ecb504e28a3d39f001f977839" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:35.420936 23723 leader_election.cc:302] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:35.421247 23787 raft_consensus.cc:2454] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:35.421768 23993 leader_election.cc:302] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:35.421828 24184 raft_consensus.cc:2781] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:35.422176 24184 raft_consensus.cc:482] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:35.422292 24184 raft_consensus.cc:3037] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:35.422343 24211 raft_consensus.cc:2781] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:35.422474 24211 raft_consensus.cc:482] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:35.422562 24211 raft_consensus.cc:3037] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:35.423137 24243 raft_consensus.cc:482] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:35.423619 24243 raft_consensus.cc:504] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.426779 24211 raft_consensus.cc:504] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.428718 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175adf0ecb504e28a3d39f001f977839" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:35.428994 24211 leader_election.cc:288] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:35.434873 24174 tablet_bootstrap.cc:492] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:35.403888 24179 tablet_replica.cc:330] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:35.420657 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df565c1c6564efd88fa557c50522d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.440032 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5def9306dfba4d019d3acece81df2f86" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:35.440752 23861 leader_election.cc:341] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:35.443812 24184 raft_consensus.cc:504] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.444274 23787 raft_consensus.cc:2379] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 149ce744feab49b5aa5bcaa14bd6aba5 in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:16:35.446292 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175adf0ecb504e28a3d39f001f977839" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:35.446600 24060 raft_consensus.cc:2379] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a297a6c7c2504ca68e87b24f6691ffb2 in current term 1: Already voted for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in this term.
I20240502 08:16:35.446705 23608 master_service.cc:946] ReplaceTablet: received request to replace tablet db80a159f14e485a971fd8bf58a21c4a from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:35.447165 24184 leader_election.cc:288] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:35.447513 24243 leader_election.cc:288] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:35.420734 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f7518b1e36147a9b9f0c58bca17061a with delete_type TABLET_DATA_DELETED (replaced by tablet d8f7b5f018d84680ac5533bf99102d31) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.428018 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175adf0ecb504e28a3d39f001f977839" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:35.449045 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f7518b1e36147a9b9f0c58bca17061a with delete_type TABLET_DATA_DELETED (replaced by tablet d8f7b5f018d84680ac5533bf99102d31) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.449303 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 8791834597254e7c9f730e03568c0748 with delete_type TABLET_DATA_DELETED (replaced by tablet e9592a041a9543e8b41f043927807131) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:35.449469 23996 leader_election.cc:341] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:35.449676 23996 leader_election.cc:302] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:35.449743 24180 tablet_replica.cc:330] T 8791834597254e7c9f730e03568c0748 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:35.449800 24179 raft_consensus.cc:2229] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.449968 24180 raft_consensus.cc:2229] T 8791834597254e7c9f730e03568c0748 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.449996 24179 raft_consensus.cc:2258] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.450114 24180 raft_consensus.cc:2258] T 8791834597254e7c9f730e03568c0748 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.449770 24174 ts_tablet_manager.cc:1367] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.089s user 0.001s sys 0.006s
I20240502 08:16:35.450215 24211 raft_consensus.cc:2726] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:16:35.450727 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet 32150c7e130f4069a2a3a02e625bfb93 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:35.450743 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df565c1c6564efd88fa557c50522d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.450883 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "050c16739efb4037a4adc78a2ffd3cba" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:35.451103 24060 raft_consensus.cc:2454] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:35.451443 24174 raft_consensus.cc:348] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.451680 24174 raft_consensus.cc:374] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.451751 24174 raft_consensus.cc:729] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.451998 24174 consensus_queue.cc:260] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.452167 23723 leader_election.cc:302] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:35.452520 24243 raft_consensus.cc:2781] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:35.452618 24243 raft_consensus.cc:482] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:35.452721 24243 raft_consensus.cc:3037] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:35.455420 24243 raft_consensus.cc:504] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.457484 24181 ts_tablet_manager.cc:1888] T 8dc8158255194b868af917c33c44f0f5 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.457679 24181 log.cc:1196] T 8dc8158255194b868af917c33c44f0f5 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/8dc8158255194b868af917c33c44f0f5
I20240502 08:16:35.458096 24181 ts_tablet_manager.cc:1909] T 8dc8158255194b868af917c33c44f0f5 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:35.452019 24180 ts_tablet_manager.cc:1875] T 8791834597254e7c9f730e03568c0748 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.461143 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 8dc8158255194b868af917c33c44f0f5 (table default.Table_017d876815e94b87b8c60e535aa06ad5 [id=307769d00cda4d679da90a24bd461dfe]) successfully deleted
I20240502 08:16:35.461539 24181 tablet_replica.cc:330] T e1cf5f66859d416baad2f767060197c8 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:35.462143 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 74aaea4b05e84f2ab31a0439f82016c1 (table default.Table_39d3e9a8fea548abadf2eef7723684f2 [id=495bc1caa3e44d56abf4269f906ef74d]) successfully deleted
I20240502 08:16:35.462337 24243 leader_election.cc:288] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:35.463027 24179 ts_tablet_manager.cc:1875] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.464856 24180 ts_tablet_manager.cc:1888] T 8791834597254e7c9f730e03568c0748 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.465019 24180 log.cc:1196] T 8791834597254e7c9f730e03568c0748 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/8791834597254e7c9f730e03568c0748
W20240502 08:16:35.466221 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 6f7518b1e36147a9b9f0c58bca17061a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f7518b1e36147a9b9f0c58bca17061a already in progress: creating tablet
I20240502 08:16:35.452548 24174 ts_tablet_manager.cc:1398] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:35.466921 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175adf0ecb504e28a3d39f001f977839" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:35.467039 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "050c16739efb4037a4adc78a2ffd3cba" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:35.467347 23925 raft_consensus.cc:2454] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:35.468060 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "175adf0ecb504e28a3d39f001f977839" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:35.468607 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "050c16739efb4037a4adc78a2ffd3cba" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:35.468768 23925 raft_consensus.cc:3037] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:35.471678 23925 raft_consensus.cc:2454] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:35.479629 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_065c3fa8292242d59b5fbebbfa233fdc" } modify_external_catalogs: true
I20240502 08:16:35.479854 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_065c3fa8292242d59b5fbebbfa233fdc" } modify_external_catalogs: true
I20240502 08:16:35.480875 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c45e7b653ae641b293b87c150748c864"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.481545 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c45e7b653ae641b293b87c150748c864 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.482901 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 32150c7e130f4069a2a3a02e625bfb93
I20240502 08:16:35.480875 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_47643e78190448bf93ad852cabe9b8ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.484880 23723 leader_election.cc:341] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:35.485042 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c79a0ae768dd4d23af276094b9723a7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.485505 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c79a0ae768dd4d23af276094b9723a7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:35.484864 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47643e78190448bf93ad852cabe9b8ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:35.488984 23723 leader_election.cc:341] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:35.489130 23723 leader_election.cc:302] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:35.489717 24184 raft_consensus.cc:2726] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:16:35.490420 24179 ts_tablet_manager.cc:1888] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.490578 24179 log.cc:1196] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/5def9306dfba4d019d3acece81df2f86
I20240502 08:16:35.490978 24179 ts_tablet_manager.cc:1909] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:35.491246 24174 tablet_bootstrap.cc:492] T 4ce3e95d5ea7435593b7cfe5a670a76a P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:35.493074 24174 tablet_bootstrap.cc:654] T 4ce3e95d5ea7435593b7cfe5a670a76a P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.501582 24180 ts_tablet_manager.cc:1909] T 8791834597254e7c9f730e03568c0748 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:35.513648 23723 leader_election.cc:302] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:35.515655 24184 raft_consensus.cc:2781] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:35.516417 24184 raft_consensus.cc:686] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:35.517230 24184 consensus_queue.cc:237] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.518844 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 5def9306dfba4d019d3acece81df2f86 (table default.Table_00eb74cd24954c52b5484751f035f4c1 [id=b42e15a2350f4c9a8097ccb0cf1aaddc]) successfully deleted
I20240502 08:16:35.519131 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet 32150c7e130f4069a2a3a02e625bfb93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:35.519207 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet c709efdd2ce74b5a8ab7278b402ebfe9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c709efdd2ce74b5a8ab7278b402ebfe9 already in progress: creating tablet
W20240502 08:16:35.519464 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 5738cd77bbe147b49215ba9639d7f547 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5738cd77bbe147b49215ba9639d7f547 already in progress: creating tablet
W20240502 08:16:35.519676 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet c1c2a988a3524c8fb7410e8508bd1abc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1c2a988a3524c8fb7410e8508bd1abc already in progress: creating tablet
W20240502 08:16:35.519896 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 3052510d77f344bf87decb9c2e5ec0bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3052510d77f344bf87decb9c2e5ec0bd already in progress: creating tablet
I20240502 08:16:35.522095 24174 tablet_bootstrap.cc:492] T 4ce3e95d5ea7435593b7cfe5a670a76a P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:35.522256 24174 ts_tablet_manager.cc:1367] T 4ce3e95d5ea7435593b7cfe5a670a76a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.056s user 0.006s sys 0.007s
I20240502 08:16:35.462167 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "050c16739efb4037a4adc78a2ffd3cba" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:35.474511 24173 tablet_bootstrap.cc:492] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:35.523581 24060 raft_consensus.cc:3037] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:35.523638 24173 ts_tablet_manager.cc:1367] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.479s user 0.069s sys 0.138s
I20240502 08:16:35.523597 24174 raft_consensus.cc:348] T 4ce3e95d5ea7435593b7cfe5a670a76a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.523806 24174 raft_consensus.cc:374] T 4ce3e95d5ea7435593b7cfe5a670a76a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.523895 24174 raft_consensus.cc:729] T 4ce3e95d5ea7435593b7cfe5a670a76a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.524149 24174 consensus_queue.cc:260] T 4ce3e95d5ea7435593b7cfe5a670a76a P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.524459 24173 raft_consensus.cc:348] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.524716 24173 raft_consensus.cc:374] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.524806 24173 raft_consensus.cc:729] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.524919 24174 ts_tablet_manager.cc:1398] T 4ce3e95d5ea7435593b7cfe5a670a76a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:16:35.525228 24174 tablet_bootstrap.cc:492] T db80a159f14e485a971fd8bf58a21c4a P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:35.525069 24173 consensus_queue.cc:260] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.488181 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 32150c7e130f4069a2a3a02e625bfb93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.525614 24173 ts_tablet_manager.cc:1398] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:35.522567 24059 raft_consensus.cc:1261] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:35.526525 24173 tablet_bootstrap.cc:492] T 8791834597254e7c9f730e03568c0748 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:35.526984 24174 tablet_bootstrap.cc:654] T db80a159f14e485a971fd8bf58a21c4a P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.527704 24181 raft_consensus.cc:2229] T e1cf5f66859d416baad2f767060197c8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.528223 24181 raft_consensus.cc:2258] T e1cf5f66859d416baad2f767060197c8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.528630 24243 consensus_queue.cc:1035] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:35.529723 24060 raft_consensus.cc:2454] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:35.530515 24173 tablet_bootstrap.cc:654] T 8791834597254e7c9f730e03568c0748 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.534405 24175 tablet_bootstrap.cc:492] T e1cf5f66859d416baad2f767060197c8 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:35.534624 24175 ts_tablet_manager.cc:1367] T e1cf5f66859d416baad2f767060197c8 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.493s user 0.048s sys 0.120s
I20240502 08:16:35.536113 24175 raft_consensus.cc:348] T e1cf5f66859d416baad2f767060197c8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.537240 24175 raft_consensus.cc:374] T e1cf5f66859d416baad2f767060197c8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.537325 24175 raft_consensus.cc:729] T e1cf5f66859d416baad2f767060197c8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.537535 24175 consensus_queue.cc:260] T e1cf5f66859d416baad2f767060197c8 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.538025 24175 ts_tablet_manager.cc:1398] T e1cf5f66859d416baad2f767060197c8 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:35.538316 24175 tablet_bootstrap.cc:492] T 6c0a4b216c7e474db1992a299ff8eeb4 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:35.538025 24181 ts_tablet_manager.cc:1875] T e1cf5f66859d416baad2f767060197c8 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.543067 24175 tablet_bootstrap.cc:654] T 6c0a4b216c7e474db1992a299ff8eeb4 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.545221 24174 tablet_bootstrap.cc:492] T db80a159f14e485a971fd8bf58a21c4a P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:35.545425 24174 ts_tablet_manager.cc:1367] T db80a159f14e485a971fd8bf58a21c4a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.001s
I20240502 08:16:35.546275 24174 raft_consensus.cc:348] T db80a159f14e485a971fd8bf58a21c4a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.548091 24173 tablet_bootstrap.cc:492] T 8791834597254e7c9f730e03568c0748 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:35.548245 24173 ts_tablet_manager.cc:1367] T 8791834597254e7c9f730e03568c0748 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.001s
I20240502 08:16:35.548697 24174 raft_consensus.cc:374] T db80a159f14e485a971fd8bf58a21c4a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.548802 24174 raft_consensus.cc:729] T db80a159f14e485a971fd8bf58a21c4a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.549935 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 8791834597254e7c9f730e03568c0748 (table default.Table_6c584db923e147d380281ecb1acd11a1 [id=467a7f6db73641ca85a1898b251938fb]) successfully deleted
W20240502 08:16:35.550387 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 1df565c1c6564efd88fa557c50522d7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1df565c1c6564efd88fa557c50522d7e already in progress: creating tablet
W20240502 08:16:35.550622 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 32150c7e130f4069a2a3a02e625bfb93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32150c7e130f4069a2a3a02e625bfb93 already in progress: creating tablet
W20240502 08:16:35.551102 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet c316e1aeab224634b760641a4ce188c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c316e1aeab224634b760641a4ce188c8 already in progress: creating tablet
W20240502 08:16:35.551352 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e9592a041a9543e8b41f043927807131 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9592a041a9543e8b41f043927807131 already in progress: creating tablet
I20240502 08:16:35.551328 24174 consensus_queue.cc:260] T db80a159f14e485a971fd8bf58a21c4a P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:35.551586 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 1df565c1c6564efd88fa557c50522d7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1df565c1c6564efd88fa557c50522d7e already in progress: creating tablet
I20240502 08:16:35.552050 24174 ts_tablet_manager.cc:1398] T db80a159f14e485a971fd8bf58a21c4a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 08:16:35.552408 24174 tablet_bootstrap.cc:492] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
W20240502 08:16:35.553922 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 6f7518b1e36147a9b9f0c58bca17061a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f7518b1e36147a9b9f0c58bca17061a already in progress: creating tablet
W20240502 08:16:35.554062 23858 leader_election.cc:341] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 5def9306dfba4d019d3acece81df2f86
I20240502 08:16:35.554237 23858 leader_election.cc:302] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:35.554719 23858 raft_consensus.cc:2670] T 5def9306dfba4d019d3acece81df2f86 P 5b89a1f0485a44b7ac1710e2369a809b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:16:35.556597 24174 tablet_bootstrap.cc:654] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.556237 24173 raft_consensus.cc:348] T 8791834597254e7c9f730e03568c0748 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.559682 24173 raft_consensus.cc:374] T 8791834597254e7c9f730e03568c0748 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 08:16:35.560081 23595 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 9b1eb762780c450bbaf380d17db564b6 (table default.Table_c3fb1a324a4b4920a7799bbcf3229806 [id=ba56850714c2422eb5dbbc0f49df7ce7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9b1eb762780c450bbaf380d17db564b6 consensus state has no leader
I20240502 08:16:35.560773 24173 raft_consensus.cc:729] T 8791834597254e7c9f730e03568c0748 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.561097 24173 consensus_queue.cc:260] T 8791834597254e7c9f730e03568c0748 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.561249 24174 tablet_bootstrap.cc:492] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:35.561389 24174 ts_tablet_manager.cc:1367] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240502 08:16:35.561903 24173 ts_tablet_manager.cc:1398] T 8791834597254e7c9f730e03568c0748 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.013s user 0.002s sys 0.001s
I20240502 08:16:35.563227 24173 tablet_bootstrap.cc:492] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:35.567442 23610 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 812aab009a584e768aba536d6133ed9f
I20240502 08:16:35.568226 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 32150c7e130f4069a2a3a02e625bfb93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.568544 23610 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet ec4028a8e66442bfabac9e587aed00bb (table default.Table_62b86da21eb84bc387b64725dabda733 [id=1e590d80e3e34e1aac8a0c7908a9d813]). Expected version 3 got 0
I20240502 08:16:35.568881 24175 tablet_bootstrap.cc:492] T 6c0a4b216c7e474db1992a299ff8eeb4 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:35.569134 24175 ts_tablet_manager.cc:1367] T 6c0a4b216c7e474db1992a299ff8eeb4 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.003s
W20240502 08:16:35.569728 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 32150c7e130f4069a2a3a02e625bfb93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32150c7e130f4069a2a3a02e625bfb93 already in progress: creating tablet
I20240502 08:16:35.570015 24175 raft_consensus.cc:348] T 6c0a4b216c7e474db1992a299ff8eeb4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.570226 24175 raft_consensus.cc:374] T 6c0a4b216c7e474db1992a299ff8eeb4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.570312 24175 raft_consensus.cc:729] T 6c0a4b216c7e474db1992a299ff8eeb4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.570542 24175 consensus_queue.cc:260] T 6c0a4b216c7e474db1992a299ff8eeb4 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.571048 24175 ts_tablet_manager.cc:1398] T 6c0a4b216c7e474db1992a299ff8eeb4 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:35.571367 24175 tablet_bootstrap.cc:492] T 8dc8158255194b868af917c33c44f0f5 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:35.572541 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_59462d2ccf514f0da864562120217931"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.573199 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59462d2ccf514f0da864562120217931 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.573470 24175 tablet_bootstrap.cc:654] T 8dc8158255194b868af917c33c44f0f5 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.573905 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_cb0ea06f5e0b47e5978fc9a5ed8880ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.574438 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb0ea06f5e0b47e5978fc9a5ed8880ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:35.575069 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f51b26043ef44b30846498cd1497607c has no consensus state
I20240502 08:16:35.575460 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_afaca5b9d2774c0baff53ca022b9c4dd" } new_table_name: "default.Table_cc80d1cedafa4e71906e5e9530fdd828" modify_external_catalogs: true
I20240502 08:16:35.578840 23925 raft_consensus.cc:1261] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:35.579025 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet d8324e9b796047428e2fe918f07f9b32 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:35.579572 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b9e3e2982acf4ad7994dd28c78b1bb47" } modify_external_catalogs: true
I20240502 08:16:35.579690 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet db80a159f14e485a971fd8bf58a21c4a
I20240502 08:16:35.579772 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b9e3e2982acf4ad7994dd28c78b1bb47" } modify_external_catalogs: true
I20240502 08:16:35.580204 23606 master_service.cc:946] ReplaceTablet: received request to replace tablet 6c0a4b216c7e474db1992a299ff8eeb4 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:35.580260 23608 catalog_manager.cc:6554] ReplaceTablet: tablet db80a159f14e485a971fd8bf58a21c4a deleted and replaced by tablet e134e90efb4a48bb82b53a80bda708b3
I20240502 08:16:35.581425 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet db80a159f14e485a971fd8bf58a21c4a with delete_type TABLET_DATA_DELETED (replaced by tablet e134e90efb4a48bb82b53a80bda708b3) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.581939 24180 tablet_replica.cc:330] T db80a159f14e485a971fd8bf58a21c4a P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:35.582302 24180 raft_consensus.cc:2229] T db80a159f14e485a971fd8bf58a21c4a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.582588 24180 raft_consensus.cc:2258] T db80a159f14e485a971fd8bf58a21c4a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.583367 24173 tablet_bootstrap.cc:654] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.584256 23602 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet ec4028a8e66442bfabac9e587aed00bb (table default.Table_62b86da21eb84bc387b64725dabda733 [id=1e590d80e3e34e1aac8a0c7908a9d813]). Expected version 3 got 0
I20240502 08:16:35.584877 24180 ts_tablet_manager.cc:1875] T db80a159f14e485a971fd8bf58a21c4a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.584949 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet db80a159f14e485a971fd8bf58a21c4a with delete_type TABLET_DATA_DELETED (replaced by tablet e134e90efb4a48bb82b53a80bda708b3) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.588409 24211 raft_consensus.cc:482] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:35.588410 24181 ts_tablet_manager.cc:1888] T e1cf5f66859d416baad2f767060197c8 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.588774 24211 raft_consensus.cc:504] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.589440 24211 leader_election.cc:288] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:35.590073 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec4028a8e66442bfabac9e587aed00bb" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:35.590335 23787 raft_consensus.cc:2454] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:35.591342 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec4028a8e66442bfabac9e587aed00bb" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:35.592159 23993 leader_election.cc:302] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
W20240502 08:16:35.592418 23996 leader_election.cc:341] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:35.592947 24211 raft_consensus.cc:2781] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:35.593061 24211 raft_consensus.cc:482] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:35.593175 24211 raft_consensus.cc:3037] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:35.595777 24211 raft_consensus.cc:504] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.596002 24184 consensus_queue.cc:1035] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:35.596849 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec4028a8e66442bfabac9e587aed00bb" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:35.597002 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet db80a159f14e485a971fd8bf58a21c4a with delete_type TABLET_DATA_DELETED (replaced by tablet e134e90efb4a48bb82b53a80bda708b3) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.597044 23787 raft_consensus.cc:3037] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:35.597416 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet e1cf5f66859d416baad2f767060197c8 with delete_type TABLET_DATA_DELETED (replaced by tablet a15ff3120b1943a69ddb609d0575ae1a) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.597690 24179 tablet_replica.cc:330] T e1cf5f66859d416baad2f767060197c8 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:35.597862 24179 raft_consensus.cc:2229] T e1cf5f66859d416baad2f767060197c8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.598063 24179 raft_consensus.cc:2258] T e1cf5f66859d416baad2f767060197c8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.598765 24179 ts_tablet_manager.cc:1875] T e1cf5f66859d416baad2f767060197c8 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.598779 24174 raft_consensus.cc:348] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.598963 24174 raft_consensus.cc:374] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.599046 24174 raft_consensus.cc:729] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.599275 24174 consensus_queue.cc:260] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.599766 23787 raft_consensus.cc:2454] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:35.600144 23993 leader_election.cc:302] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:35.601773 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec4028a8e66442bfabac9e587aed00bb" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:35.602310 23996 leader_election.cc:341] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:35.602761 24211 leader_election.cc:288] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:35.602964 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c79a0ae768dd4d23af276094b9723a7a" } new_table_name: "default.Table_dae0bce7072a4d50aa63b702ee775fcf" modify_external_catalogs: true
I20240502 08:16:35.603273 24211 raft_consensus.cc:2781] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:35.603541 24211 raft_consensus.cc:686] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:35.604017 24211 consensus_queue.cc:237] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.605692 24179 ts_tablet_manager.cc:1888] T e1cf5f66859d416baad2f767060197c8 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.605834 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c3fb1a324a4b4920a7799bbcf3229806" } new_table_name: "default.Table_4dd22967bcf147388ea4ceb9cd24b065" modify_external_catalogs: true
I20240502 08:16:35.605945 24179 log.cc:1196] T e1cf5f66859d416baad2f767060197c8 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/e1cf5f66859d416baad2f767060197c8
I20240502 08:16:35.606637 24179 ts_tablet_manager.cc:1909] T e1cf5f66859d416baad2f767060197c8 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:35.608028 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0f3416651b83425bbfdd6a9adfc2709e" } modify_external_catalogs: true
I20240502 08:16:35.608253 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0f3416651b83425bbfdd6a9adfc2709e" } modify_external_catalogs: true
I20240502 08:16:35.610628 24174 ts_tablet_manager.cc:1398] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.049s user 0.002s sys 0.000s
I20240502 08:16:35.614622 24174 tablet_bootstrap.cc:492] T c1c2a988a3524c8fb7410e8508bd1abc P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:35.615703 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet d8324e9b796047428e2fe918f07f9b32
I20240502 08:16:35.616158 23604 catalog_manager.cc:6554] ReplaceTablet: tablet d8324e9b796047428e2fe918f07f9b32 deleted and replaced by tablet 4302fec5c51e4c6c841710bab50b2997
I20240502 08:16:35.616420 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet 7a10f86a0f3f4385bef39eda14d4fa9f from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:35.616283 24174 tablet_bootstrap.cc:654] T c1c2a988a3524c8fb7410e8508bd1abc P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.617935 23609 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet dad20d7e9e3f41dcac11d333dc0dff85
I20240502 08:16:35.619966 24181 log.cc:1196] T e1cf5f66859d416baad2f767060197c8 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/e1cf5f66859d416baad2f767060197c8
I20240502 08:16:35.620342 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet d8324e9b796047428e2fe918f07f9b32 with delete_type TABLET_DATA_DELETED (replaced by tablet 4302fec5c51e4c6c841710bab50b2997) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.621991 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0170ebbe1b274aed934c0dd0445f3271" } new_table_name: "default.Table_0528f5d019fe489db8bed1f71adac3cb" modify_external_catalogs: true
I20240502 08:16:35.624146 24173 tablet_bootstrap.cc:492] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:35.625861 23603 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 68bdb761ff36426c96949e97279e2756 (table default.Table_a757855f2b244c548d8c2add77baaff7 [id=eca4080e8ebf46e9b58c85ee1d8675ca]). Expected version 2 got 0
I20240502 08:16:35.626448 24173 ts_tablet_manager.cc:1367] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.062s user 0.001s sys 0.014s
I20240502 08:16:35.626559 24181 ts_tablet_manager.cc:1909] T e1cf5f66859d416baad2f767060197c8 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:35.628091 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet d8324e9b796047428e2fe918f07f9b32 with delete_type TABLET_DATA_DELETED (replaced by tablet 4302fec5c51e4c6c841710bab50b2997) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:35.629545 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dead13bba95f43cb809e17f6d7fba028 has no consensus state
I20240502 08:16:35.630272 23605 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet fe389d10ccb548879d9267f49dded3e2
I20240502 08:16:35.630357 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet e1cf5f66859d416baad2f767060197c8 (table default.Table_bc6b3019e44547cda285ac0021565619 [id=c08bf22592064123bb9cab28935d81a5]) successfully deleted
W20240502 08:16:35.630726 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b1eb762780c450bbaf380d17db564b6 consensus state has no leader
W20240502 08:16:35.632261 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet c709efdd2ce74b5a8ab7278b402ebfe9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c709efdd2ce74b5a8ab7278b402ebfe9 already in progress: creating tablet
I20240502 08:16:35.632675 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 8791834597254e7c9f730e03568c0748 with delete_type TABLET_DATA_DELETED (replaced by tablet e9592a041a9543e8b41f043927807131) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.633045 24173 raft_consensus.cc:348] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
W20240502 08:16:35.634104 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 5738cd77bbe147b49215ba9639d7f547 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5738cd77bbe147b49215ba9639d7f547 already in progress: creating tablet
I20240502 08:16:35.634523 24173 raft_consensus.cc:374] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 08:16:35.634912 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 3052510d77f344bf87decb9c2e5ec0bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3052510d77f344bf87decb9c2e5ec0bd already in progress: creating tablet
W20240502 08:16:35.635733 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e9592a041a9543e8b41f043927807131 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9592a041a9543e8b41f043927807131 already in progress: creating tablet
I20240502 08:16:35.636368 24173 raft_consensus.cc:729] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.639597 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 7a10f86a0f3f4385bef39eda14d4fa9f
I20240502 08:16:35.640340 23604 catalog_manager.cc:6554] ReplaceTablet: tablet 7a10f86a0f3f4385bef39eda14d4fa9f deleted and replaced by tablet c470d420c45843a69a63d9800b7cfc97
I20240502 08:16:35.640658 24173 consensus_queue.cc:260] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.640885 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a10f86a0f3f4385bef39eda14d4fa9f with delete_type TABLET_DATA_DELETED (replaced by tablet c470d420c45843a69a63d9800b7cfc97) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:35.640985 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 485c772245fe48a39fc6ceb6377ac0b6 consensus state has no leader
I20240502 08:16:35.641270 24173 ts_tablet_manager.cc:1398] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
W20240502 08:16:35.641600 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet c316e1aeab224634b760641a4ce188c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c316e1aeab224634b760641a4ce188c8 already in progress: creating tablet
I20240502 08:16:35.639828 24293 tablet.cc:1722] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:35.642982 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a10f86a0f3f4385bef39eda14d4fa9f with delete_type TABLET_DATA_DELETED (replaced by tablet c470d420c45843a69a63d9800b7cfc97) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.643862 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c0a4b216c7e474db1992a299ff8eeb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.644174 24179 tablet_replica.cc:330] T 6c0a4b216c7e474db1992a299ff8eeb4 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:35.644398 24179 raft_consensus.cc:2229] T 6c0a4b216c7e474db1992a299ff8eeb4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.644595 24179 raft_consensus.cc:2258] T 6c0a4b216c7e474db1992a299ff8eeb4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.644877 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet d8324e9b796047428e2fe918f07f9b32 with delete_type TABLET_DATA_DELETED (replaced by tablet 4302fec5c51e4c6c841710bab50b2997) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:35.644901 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet db80a159f14e485a971fd8bf58a21c4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db80a159f14e485a971fd8bf58a21c4a already in progress: creating tablet
I20240502 08:16:35.645128 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet e1cf5f66859d416baad2f767060197c8 (table default.Table_bc6b3019e44547cda285ac0021565619 [id=c08bf22592064123bb9cab28935d81a5]) successfully deleted
I20240502 08:16:35.645274 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a10f86a0f3f4385bef39eda14d4fa9f with delete_type TABLET_DATA_DELETED (replaced by tablet c470d420c45843a69a63d9800b7cfc97) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.646174 24179 ts_tablet_manager.cc:1875] T 6c0a4b216c7e474db1992a299ff8eeb4 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.652122 23604 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 68bdb761ff36426c96949e97279e2756 (table default.Table_a757855f2b244c548d8c2add77baaff7 [id=eca4080e8ebf46e9b58c85ee1d8675ca]). Expected version 2 got 0
I20240502 08:16:35.653023 24173 tablet_bootstrap.cc:492] T 5def9306dfba4d019d3acece81df2f86 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:35.656622 24179 ts_tablet_manager.cc:1888] T 6c0a4b216c7e474db1992a299ff8eeb4 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.657004 24179 log.cc:1196] T 6c0a4b216c7e474db1992a299ff8eeb4 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/6c0a4b216c7e474db1992a299ff8eeb4
I20240502 08:16:35.657917 24179 ts_tablet_manager.cc:1909] T 6c0a4b216c7e474db1992a299ff8eeb4 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:35.662328 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 6c0a4b216c7e474db1992a299ff8eeb4 (table default.Table_1f432841712848378bc24e8050ac57c7 [id=a9cdc35f95ec47d9a41d375133749c88]) successfully deleted
W20240502 08:16:35.662793 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet c1c2a988a3524c8fb7410e8508bd1abc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1c2a988a3524c8fb7410e8508bd1abc already in progress: creating tablet
W20240502 08:16:35.663800 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet d8324e9b796047428e2fe918f07f9b32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8324e9b796047428e2fe918f07f9b32 already in progress: creating tablet
W20240502 08:16:35.665069 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 7a10f86a0f3f4385bef39eda14d4fa9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a10f86a0f3f4385bef39eda14d4fa9f already in progress: creating tablet
W20240502 08:16:35.667091 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 6f7518b1e36147a9b9f0c58bca17061a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f7518b1e36147a9b9f0c58bca17061a already in progress: creating tablet
I20240502 08:16:35.668503 24294 mvcc.cc:205] Tried to move back new op lower bound from 7023156410333323264 to 7023156408201125888. Current Snapshot: MvccSnapshot[applied={T|T < 7023156410333323264}]
I20240502 08:16:35.669356 23610 catalog_manager.cc:5484] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:35.669817 23610 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 050c16739efb4037a4adc78a2ffd3cba (table default.Table_e0bf76fa2b2d451eb0102e9897669b6f [id=ba62a673079d424992daa0f6b0c5d6b9]). Expected version 1 got 0
W20240502 08:16:35.670207 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 1df565c1c6564efd88fa557c50522d7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1df565c1c6564efd88fa557c50522d7e already in progress: creating tablet
W20240502 08:16:35.670784 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e134e90efb4a48bb82b53a80bda708b3 has no consensus state
I20240502 08:16:35.670964 24173 tablet_bootstrap.cc:654] T 5def9306dfba4d019d3acece81df2f86 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:35.671154 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 32150c7e130f4069a2a3a02e625bfb93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32150c7e130f4069a2a3a02e625bfb93 already in progress: creating tablet
I20240502 08:16:35.672009 24174 tablet_bootstrap.cc:492] T c1c2a988a3524c8fb7410e8508bd1abc P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:35.672201 24174 ts_tablet_manager.cc:1367] T c1c2a988a3524c8fb7410e8508bd1abc P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.058s user 0.008s sys 0.003s
I20240502 08:16:35.674140 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 0f2ff50fa357448494434d1487ce83bc
I20240502 08:16:35.674355 24174 raft_consensus.cc:348] T c1c2a988a3524c8fb7410e8508bd1abc P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.675452 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f2ff50fa357448494434d1487ce83bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.674669 24174 raft_consensus.cc:374] T c1c2a988a3524c8fb7410e8508bd1abc P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.675577 24174 raft_consensus.cc:729] T c1c2a988a3524c8fb7410e8508bd1abc P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.675573 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f2ff50fa357448494434d1487ce83bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.675839 24174 consensus_queue.cc:260] T c1c2a988a3524c8fb7410e8508bd1abc P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:35.676215 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 0f2ff50fa357448494434d1487ce83bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f2ff50fa357448494434d1487ce83bc already in progress: creating tablet
I20240502 08:16:35.676301 24174 ts_tablet_manager.cc:1398] T c1c2a988a3524c8fb7410e8508bd1abc P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20240502 08:16:35.676434 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet db80a159f14e485a971fd8bf58a21c4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db80a159f14e485a971fd8bf58a21c4a already in progress: creating tablet
I20240502 08:16:35.676736 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f2ff50fa357448494434d1487ce83bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:35.677230 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet d8324e9b796047428e2fe918f07f9b32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8324e9b796047428e2fe918f07f9b32 already in progress: creating tablet
I20240502 08:16:35.677148 24297 tablet.cc:1722] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:35.677518 24181 tablet_replica.cc:330] T 8791834597254e7c9f730e03568c0748 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:35.677716 24181 raft_consensus.cc:2229] T 8791834597254e7c9f730e03568c0748 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.677913 24181 raft_consensus.cc:2258] T 8791834597254e7c9f730e03568c0748 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.679255 23602 catalog_manager.cc:5484] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67). New cstate: current_term: 1 leader_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:35.679857 23602 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet ec4028a8e66442bfabac9e587aed00bb (table default.Table_62b86da21eb84bc387b64725dabda733 [id=1e590d80e3e34e1aac8a0c7908a9d813]). Expected version 3 got 0
I20240502 08:16:35.680907 23607 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 68bdb761ff36426c96949e97279e2756 (table default.Table_a757855f2b244c548d8c2add77baaff7 [id=eca4080e8ebf46e9b58c85ee1d8675ca]). Expected version 2 got 0
I20240502 08:16:35.681344 24296 tablet.cc:1722] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:35.684478 24181 ts_tablet_manager.cc:1875] T 8791834597254e7c9f730e03568c0748 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.685916 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_40bac43800874d1789e8ee318c61f0ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.686489 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40bac43800874d1789e8ee318c61f0ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.687266 24174 tablet_bootstrap.cc:492] T 9628134656eb4be1b3e1419e5c1accbe P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:35.687748 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c201704a319a4e078054899061f76615"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.688289 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c201704a319a4e078054899061f76615 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.688915 24276 mvcc.cc:205] Tried to move back new op lower bound from 7023156410333323264 to 7023156408201125888. Current Snapshot: MvccSnapshot[applied={T|T < 7023156410333323264}]
I20240502 08:16:35.690400 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d8aa2cd6c42444c48c383bf13e3728fd" } modify_external_catalogs: true
I20240502 08:16:35.690611 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d8aa2cd6c42444c48c383bf13e3728fd" } modify_external_catalogs: true
I20240502 08:16:35.693795 24293 tablet.cc:1722] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:35.694994 24173 tablet_bootstrap.cc:492] T 5def9306dfba4d019d3acece81df2f86 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:35.695163 24173 ts_tablet_manager.cc:1367] T 5def9306dfba4d019d3acece81df2f86 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.007s
I20240502 08:16:35.697172 24173 raft_consensus.cc:348] T 5def9306dfba4d019d3acece81df2f86 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.697417 24173 raft_consensus.cc:374] T 5def9306dfba4d019d3acece81df2f86 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.697499 23606 master_service.cc:946] ReplaceTablet: received request to replace tablet 3052510d77f344bf87decb9c2e5ec0bd from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:35.697530 24173 raft_consensus.cc:729] T 5def9306dfba4d019d3acece81df2f86 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.697767 24173 consensus_queue.cc:260] T 5def9306dfba4d019d3acece81df2f86 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.698328 24173 ts_tablet_manager.cc:1398] T 5def9306dfba4d019d3acece81df2f86 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:35.699280 24174 tablet_bootstrap.cc:654] T 9628134656eb4be1b3e1419e5c1accbe P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.700843 23894 tablet_service.cc:1467] Processing CreateTablet for tablet b742c51f6204449a804889815604d5cb (DEFAULT_TABLE table=default.Table_9507475756c3401fa75fc34d4a4b7814 [id=f42bb435dc484af7990e8cf4671e1c85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.701226 24032 tablet_service.cc:1467] Processing CreateTablet for tablet b742c51f6204449a804889815604d5cb (DEFAULT_TABLE table=default.Table_9507475756c3401fa75fc34d4a4b7814 [id=f42bb435dc484af7990e8cf4671e1c85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.701329 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b742c51f6204449a804889815604d5cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.701678 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b742c51f6204449a804889815604d5cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.706746 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb2d0a4898544f182ab8070d29a7940 (DEFAULT_TABLE table=default.Table_482a1b0b7a064e02805ac3b86fc62f99 [id=3c1d614616374739aeb928e797479df4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.706909 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 3cc30014a1ac45de937751eed36877a9 (DEFAULT_TABLE table=default.Table_d8aa2cd6c42444c48c383bf13e3728fd [id=9cbc066db19142d3bf0fd50cbe8fc91a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.707080 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 8633864e248f416cb651da3b0c2aa936 (DEFAULT_TABLE table=default.Table_1ce01bea1dbd40d18ca5073c848b506d [id=1b5536950db244d1978b154da0be33e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.707156 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb2d0a4898544f182ab8070d29a7940. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.707427 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8633864e248f416cb651da3b0c2aa936. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.707700 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cc30014a1ac45de937751eed36877a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.707664 23895 tablet_service.cc:1467] Processing CreateTablet for tablet f51b26043ef44b30846498cd1497607c (DEFAULT_TABLE table=default.Table_da5e259b0d0c47cf94ab30e3e98d1540 [id=956913447d9b4f55920aa4382692d608]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.708030 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 43e870c3a0e340d1bad1c1fa0380a762 (DEFAULT_TABLE table=default.Table_e54897c4ce62463f9cf7baba62b2706a [id=7e0c798bc77a4770a4bbe8658f181ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.708252 23898 tablet_service.cc:1467] Processing CreateTablet for tablet d5bab1b7c48248a7973c27d0a865b34b (DEFAULT_TABLE table=default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.708782 23896 tablet_service.cc:1467] Processing CreateTablet for tablet b6c52f1cae3a4ad38a9f95bfaf055e76 (DEFAULT_TABLE table=default.Table_9837c3c4dad548c18b554594be454c46 [id=922fb979b5894023a68820aece0b8c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.709251 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 23ce83c8001d4b2988b88c0ed9bf948c (DEFAULT_TABLE table=default.Table_623eac7101f94d959fb83b6f51d8fe0a [id=54454efbe9394170822f237dc8e6115e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.709583 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51b26043ef44b30846498cd1497607c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.709822 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e870c3a0e340d1bad1c1fa0380a762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.710038 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c52f1cae3a4ad38a9f95bfaf055e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.711702 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ce83c8001d4b2988b88c0ed9bf948c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.713039 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce3e95d5ea7435593b7cfe5a670a76a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.714550 23767 tablet_service.cc:1467] Processing CreateTablet for tablet b742c51f6204449a804889815604d5cb (DEFAULT_TABLE table=default.Table_9507475756c3401fa75fc34d4a4b7814 [id=f42bb435dc484af7990e8cf4671e1c85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.715021 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b742c51f6204449a804889815604d5cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.715749 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb2d0a4898544f182ab8070d29a7940 (DEFAULT_TABLE table=default.Table_482a1b0b7a064e02805ac3b86fc62f99 [id=3c1d614616374739aeb928e797479df4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.716102 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb2d0a4898544f182ab8070d29a7940. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.716431 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb2d0a4898544f182ab8070d29a7940 (DEFAULT_TABLE table=default.Table_482a1b0b7a064e02805ac3b86fc62f99 [id=3c1d614616374739aeb928e797479df4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.716660 24033 tablet_service.cc:1467] Processing CreateTablet for tablet d5bab1b7c48248a7973c27d0a865b34b (DEFAULT_TABLE table=default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.716823 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb2d0a4898544f182ab8070d29a7940. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.717150 24038 tablet_service.cc:1467] Processing CreateTablet for tablet b6c52f1cae3a4ad38a9f95bfaf055e76 (DEFAULT_TABLE table=default.Table_9837c3c4dad548c18b554594be454c46 [id=922fb979b5894023a68820aece0b8c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.717465 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c52f1cae3a4ad38a9f95bfaf055e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.716425 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 8633864e248f416cb651da3b0c2aa936 (DEFAULT_TABLE table=default.Table_1ce01bea1dbd40d18ca5073c848b506d [id=1b5536950db244d1978b154da0be33e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.721308 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5bab1b7c48248a7973c27d0a865b34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.721416 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5bab1b7c48248a7973c27d0a865b34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.722921 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 8633864e248f416cb651da3b0c2aa936 (DEFAULT_TABLE table=default.Table_1ce01bea1dbd40d18ca5073c848b506d [id=1b5536950db244d1978b154da0be33e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.723304 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8633864e248f416cb651da3b0c2aa936. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.723500 23762 tablet_service.cc:1467] Processing CreateTablet for tablet d5bab1b7c48248a7973c27d0a865b34b (DEFAULT_TABLE table=default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.723846 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5bab1b7c48248a7973c27d0a865b34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.725836 23763 tablet_service.cc:1467] Processing CreateTablet for tablet f51b26043ef44b30846498cd1497607c (DEFAULT_TABLE table=default.Table_da5e259b0d0c47cf94ab30e3e98d1540 [id=956913447d9b4f55920aa4382692d608]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.726181 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51b26043ef44b30846498cd1497607c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.730190 23762 tablet_service.cc:1467] Processing CreateTablet for tablet b6c52f1cae3a4ad38a9f95bfaf055e76 (DEFAULT_TABLE table=default.Table_9837c3c4dad548c18b554594be454c46 [id=922fb979b5894023a68820aece0b8c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.730485 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c52f1cae3a4ad38a9f95bfaf055e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.736207 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 3cc30014a1ac45de937751eed36877a9 (DEFAULT_TABLE table=default.Table_d8aa2cd6c42444c48c383bf13e3728fd [id=9cbc066db19142d3bf0fd50cbe8fc91a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.736901 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 43e870c3a0e340d1bad1c1fa0380a762 (DEFAULT_TABLE table=default.Table_e54897c4ce62463f9cf7baba62b2706a [id=7e0c798bc77a4770a4bbe8658f181ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.737012 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cc30014a1ac45de937751eed36877a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.737231 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e870c3a0e340d1bad1c1fa0380a762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.738569 24040 tablet_service.cc:1467] Processing CreateTablet for tablet f51b26043ef44b30846498cd1497607c (DEFAULT_TABLE table=default.Table_da5e259b0d0c47cf94ab30e3e98d1540 [id=956913447d9b4f55920aa4382692d608]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.738679 24180 ts_tablet_manager.cc:1888] T db80a159f14e485a971fd8bf58a21c4a P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.738793 24180 log.cc:1196] T db80a159f14e485a971fd8bf58a21c4a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/db80a159f14e485a971fd8bf58a21c4a
I20240502 08:16:35.738857 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 23ce83c8001d4b2988b88c0ed9bf948c (DEFAULT_TABLE table=default.Table_623eac7101f94d959fb83b6f51d8fe0a [id=54454efbe9394170822f237dc8e6115e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.739156 24180 ts_tablet_manager.cc:1909] T db80a159f14e485a971fd8bf58a21c4a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:35.739295 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 43e870c3a0e340d1bad1c1fa0380a762 (DEFAULT_TABLE table=default.Table_e54897c4ce62463f9cf7baba62b2706a [id=7e0c798bc77a4770a4bbe8658f181ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.739579 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8633864e248f416cb651da3b0c2aa936. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.740168 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e870c3a0e340d1bad1c1fa0380a762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.740717 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51b26043ef44b30846498cd1497607c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.741029 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ce83c8001d4b2988b88c0ed9bf948c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.741164 24180 tablet_replica.cc:330] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:35.742780 24180 raft_consensus.cc:2229] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:35.743736 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 3cc30014a1ac45de937751eed36877a9 (DEFAULT_TABLE table=default.Table_d8aa2cd6c42444c48c383bf13e3728fd [id=9cbc066db19142d3bf0fd50cbe8fc91a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.744087 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cc30014a1ac45de937751eed36877a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.746115 24180 pending_rounds.cc:62] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:16:35.746685 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet db80a159f14e485a971fd8bf58a21c4a (table default.Table_cc80d1cedafa4e71906e5e9530fdd828 [id=47682c26a85842c7b8991bf3828a9336]) successfully deleted
W20240502 08:16:35.746941 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet d8324e9b796047428e2fe918f07f9b32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8324e9b796047428e2fe918f07f9b32 already in progress: creating tablet
I20240502 08:16:35.749089 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 68bdb761ff36426c96949e97279e2756 (table default.Table_a757855f2b244c548d8c2add77baaff7 [id=eca4080e8ebf46e9b58c85ee1d8675ca]). Expected version 2 got 1
I20240502 08:16:35.751379 23602 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 68bdb761ff36426c96949e97279e2756 (table default.Table_a757855f2b244c548d8c2add77baaff7 [id=eca4080e8ebf46e9b58c85ee1d8675ca]). Expected version 2 got 1
I20240502 08:16:35.754604 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 3cc30014a1ac45de937751eed36877a9
I20240502 08:16:35.759012 24175 tablet_bootstrap.cc:492] T 8dc8158255194b868af917c33c44f0f5 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:35.759335 24175 ts_tablet_manager.cc:1367] T 8dc8158255194b868af917c33c44f0f5 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.188s user 0.009s sys 0.000s
I20240502 08:16:35.759876 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cc30014a1ac45de937751eed36877a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:35.760807 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 3cc30014a1ac45de937751eed36877a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cc30014a1ac45de937751eed36877a9 already in progress: creating tablet
I20240502 08:16:35.763092 24175 raft_consensus.cc:348] T 8dc8158255194b868af917c33c44f0f5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.763517 24175 raft_consensus.cc:374] T 8dc8158255194b868af917c33c44f0f5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.763775 24175 raft_consensus.cc:729] T 8dc8158255194b868af917c33c44f0f5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.764128 23610 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_a757855f2b244c548d8c2add77baaff7 [id=eca4080e8ebf46e9b58c85ee1d8675ca] alter complete (version 2)
I20240502 08:16:35.764267 24175 consensus_queue.cc:260] T 8dc8158255194b868af917c33c44f0f5 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.765408 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cc30014a1ac45de937751eed36877a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.767812 24175 ts_tablet_manager.cc:1398] T 8dc8158255194b868af917c33c44f0f5 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:16:35.769294 24173 tablet_bootstrap.cc:492] T 9628134656eb4be1b3e1419e5c1accbe P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:35.771184 24175 tablet_bootstrap.cc:492] T 7a10f86a0f3f4385bef39eda14d4fa9f P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:35.772745 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dc8158255194b868af917c33c44f0f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.773000 24179 tablet_replica.cc:330] T 8dc8158255194b868af917c33c44f0f5 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:35.773151 24179 raft_consensus.cc:2229] T 8dc8158255194b868af917c33c44f0f5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.773326 24179 raft_consensus.cc:2258] T 8dc8158255194b868af917c33c44f0f5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.774267 24179 ts_tablet_manager.cc:1875] T 8dc8158255194b868af917c33c44f0f5 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.775935 24297 tablet.cc:1722] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:35.794448 24180 pending_rounds.cc:68] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156408667602944 op_type: NO_OP noop_request { }
I20240502 08:16:35.794692 24180 raft_consensus.cc:2866] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:35.795655 24180 raft_consensus.cc:2258] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.795949 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 23ce83c8001d4b2988b88c0ed9bf948c (DEFAULT_TABLE table=default.Table_623eac7101f94d959fb83b6f51d8fe0a [id=54454efbe9394170822f237dc8e6115e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.796455 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ce83c8001d4b2988b88c0ed9bf948c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.797868 24179 ts_tablet_manager.cc:1888] T 8dc8158255194b868af917c33c44f0f5 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.798916 24179 log.cc:1196] T 8dc8158255194b868af917c33c44f0f5 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/8dc8158255194b868af917c33c44f0f5
I20240502 08:16:35.800228 24180 ts_tablet_manager.cc:1875] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.800688 24179 ts_tablet_manager.cc:1909] T 8dc8158255194b868af917c33c44f0f5 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:35.801818 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c2a988a3524c8fb7410e8508bd1abc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.801908 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 774487be9bc644cab4e420c3d33b3cad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.802148 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce3e95d5ea7435593b7cfe5a670a76a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.802772 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cc30014a1ac45de937751eed36877a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.803457 24034 tablet_service.cc:1514] Processing DeleteTablet for tablet 5def9306dfba4d019d3acece81df2f86 with delete_type TABLET_DATA_DELETED (replaced by tablet 223c92ae8f63402f8643e03d914e8e7e) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.805270 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 8dc8158255194b868af917c33c44f0f5 (table default.Table_017d876815e94b87b8c60e535aa06ad5 [id=307769d00cda4d679da90a24bd461dfe]) successfully deleted
I20240502 08:16:35.804209 24175 tablet_bootstrap.cc:654] T 7a10f86a0f3f4385bef39eda14d4fa9f P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.808343 24173 tablet_bootstrap.cc:654] T 9628134656eb4be1b3e1419e5c1accbe P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.808359 24296 tablet.cc:1722] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:35.809252 24211 raft_consensus.cc:482] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:35.814492 24174 tablet_bootstrap.cc:492] T 9628134656eb4be1b3e1419e5c1accbe P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:35.814661 24174 ts_tablet_manager.cc:1367] T 9628134656eb4be1b3e1419e5c1accbe P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.128s user 0.021s sys 0.021s
I20240502 08:16:35.814786 24180 ts_tablet_manager.cc:1888] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:35.815029 24180 log.cc:1196] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/7a10f86a0f3f4385bef39eda14d4fa9f
I20240502 08:16:35.815541 24180 ts_tablet_manager.cc:1909] T 7a10f86a0f3f4385bef39eda14d4fa9f P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:35.815515 24174 raft_consensus.cc:348] T 9628134656eb4be1b3e1419e5c1accbe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.815768 24174 raft_consensus.cc:374] T 9628134656eb4be1b3e1419e5c1accbe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.815876 24174 raft_consensus.cc:729] T 9628134656eb4be1b3e1419e5c1accbe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.816118 24174 consensus_queue.cc:260] T 9628134656eb4be1b3e1419e5c1accbe P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.816710 24174 ts_tablet_manager.cc:1398] T 9628134656eb4be1b3e1419e5c1accbe P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:35.817471 24180 tablet_replica.cc:330] T 4ce3e95d5ea7435593b7cfe5a670a76a P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:35.817693 24180 raft_consensus.cc:2229] T 4ce3e95d5ea7435593b7cfe5a670a76a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.817902 24180 raft_consensus.cc:2258] T 4ce3e95d5ea7435593b7cfe5a670a76a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.820191 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 7a10f86a0f3f4385bef39eda14d4fa9f (table default.Table_95acc6e36a0b448eb3fd6d5df86e2de4 [id=1328b5d30ef24f98bf00c4429383ce92]) successfully deleted
W20240502 08:16:35.820511 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 0f2ff50fa357448494434d1487ce83bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f2ff50fa357448494434d1487ce83bc already in progress: creating tablet
I20240502 08:16:35.821058 24180 ts_tablet_manager.cc:1875] T 4ce3e95d5ea7435593b7cfe5a670a76a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.821604 24175 tablet_bootstrap.cc:492] T 7a10f86a0f3f4385bef39eda14d4fa9f P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:35.821753 24175 ts_tablet_manager.cc:1367] T 7a10f86a0f3f4385bef39eda14d4fa9f P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.002s
I20240502 08:16:35.822697 24175 raft_consensus.cc:348] T 7a10f86a0f3f4385bef39eda14d4fa9f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.823025 24175 raft_consensus.cc:374] T 7a10f86a0f3f4385bef39eda14d4fa9f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.823176 24175 raft_consensus.cc:729] T 7a10f86a0f3f4385bef39eda14d4fa9f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.825680 24174 tablet_bootstrap.cc:492] T 71c0c2f3def643dfa99ff4efe65f6acf P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:35.828567 24174 tablet_bootstrap.cc:654] T 71c0c2f3def643dfa99ff4efe65f6acf P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.820786 24211 raft_consensus.cc:504] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.836678 24175 consensus_queue.cc:260] T 7a10f86a0f3f4385bef39eda14d4fa9f P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:35.840559 24175 ts_tablet_manager.cc:1398] T 7a10f86a0f3f4385bef39eda14d4fa9f P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240502 08:16:35.841238 24180 ts_tablet_manager.cc:1888] T 4ce3e95d5ea7435593b7cfe5a670a76a P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.841620 24181 ts_tablet_manager.cc:1888] T 8791834597254e7c9f730e03568c0748 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.841774 24181 log.cc:1196] T 8791834597254e7c9f730e03568c0748 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/8791834597254e7c9f730e03568c0748
I20240502 08:16:35.841632 24180 log.cc:1196] T 4ce3e95d5ea7435593b7cfe5a670a76a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/4ce3e95d5ea7435593b7cfe5a670a76a
I20240502 08:16:35.842172 24181 ts_tablet_manager.cc:1909] T 8791834597254e7c9f730e03568c0748 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:35.842622 24175 tablet_bootstrap.cc:492] T 8791834597254e7c9f730e03568c0748 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:35.843040 24181 tablet_replica.cc:330] T 7a10f86a0f3f4385bef39eda14d4fa9f P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:35.843418 24180 ts_tablet_manager.cc:1909] T 4ce3e95d5ea7435593b7cfe5a670a76a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:35.843809 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce3e95d5ea7435593b7cfe5a670a76a" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:35.844019 24181 raft_consensus.cc:2229] T 7a10f86a0f3f4385bef39eda14d4fa9f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.844197 24181 raft_consensus.cc:2258] T 7a10f86a0f3f4385bef39eda14d4fa9f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:35.844417 23996 leader_election.cc:341] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:35.844360 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce3e95d5ea7435593b7cfe5a670a76a" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:35.844741 24243 raft_consensus.cc:482] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:35.844879 24243 raft_consensus.cc:504] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.844990 24181 ts_tablet_manager.cc:1875] T 7a10f86a0f3f4385bef39eda14d4fa9f P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.846125 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "774487be9bc644cab4e420c3d33b3cad" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:35.846323 23993 leader_election.cc:341] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: Tablet not RUNNING: STOPPED
I20240502 08:16:35.846465 23993 leader_election.cc:302] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:35.846762 24290 raft_consensus.cc:2726] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:35.845564 24243 leader_election.cc:288] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:35.847126 23723 leader_election.cc:341] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:35.848428 24180 tablet_replica.cc:330] T c1c2a988a3524c8fb7410e8508bd1abc P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:35.848798 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "774487be9bc644cab4e420c3d33b3cad" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:35.849121 24211 leader_election.cc:288] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
W20240502 08:16:35.849253 23723 leader_election.cc:341] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:35.849853 23723 leader_election.cc:302] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:35.850242 24243 raft_consensus.cc:2726] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:35.850569 24180 raft_consensus.cc:2229] T c1c2a988a3524c8fb7410e8508bd1abc P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.850971 24180 raft_consensus.cc:2258] T c1c2a988a3524c8fb7410e8508bd1abc P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.854823 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 8791834597254e7c9f730e03568c0748 (table default.Table_6c584db923e147d380281ecb1acd11a1 [id=467a7f6db73641ca85a1898b251938fb]) successfully deleted
I20240502 08:16:35.855453 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 4ce3e95d5ea7435593b7cfe5a670a76a (table default.Table_6d48e2ca21a2409fae7c972ec53adbf5 [id=d06bffc628cd4e07a8651559cbe38e09]) successfully deleted
I20240502 08:16:35.856056 24180 ts_tablet_manager.cc:1875] T c1c2a988a3524c8fb7410e8508bd1abc P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.856091 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a10f86a0f3f4385bef39eda14d4fa9f with delete_type TABLET_DATA_DELETED (replaced by tablet c470d420c45843a69a63d9800b7cfc97) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.856364 24179 tablet_replica.cc:330] T 7a10f86a0f3f4385bef39eda14d4fa9f P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:35.856564 24179 raft_consensus.cc:2229] T 7a10f86a0f3f4385bef39eda14d4fa9f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.856719 24179 raft_consensus.cc:2258] T 7a10f86a0f3f4385bef39eda14d4fa9f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.857393 24179 ts_tablet_manager.cc:1875] T 7a10f86a0f3f4385bef39eda14d4fa9f P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.861752 24180 ts_tablet_manager.cc:1888] T c1c2a988a3524c8fb7410e8508bd1abc P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.863719 24180 log.cc:1196] T c1c2a988a3524c8fb7410e8508bd1abc P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/c1c2a988a3524c8fb7410e8508bd1abc
I20240502 08:16:35.864033 24181 ts_tablet_manager.cc:1888] T 7a10f86a0f3f4385bef39eda14d4fa9f P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.864185 24181 log.cc:1196] T 7a10f86a0f3f4385bef39eda14d4fa9f P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/7a10f86a0f3f4385bef39eda14d4fa9f
I20240502 08:16:35.864537 24181 ts_tablet_manager.cc:1909] T 7a10f86a0f3f4385bef39eda14d4fa9f P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:35.864573 24180 ts_tablet_manager.cc:1909] T c1c2a988a3524c8fb7410e8508bd1abc P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:35.866835 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 7a10f86a0f3f4385bef39eda14d4fa9f (table default.Table_95acc6e36a0b448eb3fd6d5df86e2de4 [id=1328b5d30ef24f98bf00c4429383ce92]) successfully deleted
I20240502 08:16:35.867059 24181 tablet_replica.cc:330] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:35.867321 24175 tablet_bootstrap.cc:654] T 8791834597254e7c9f730e03568c0748 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.867323 24181 raft_consensus.cc:2229] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240502 08:16:35.867347 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 0f2ff50fa357448494434d1487ce83bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f2ff50fa357448494434d1487ce83bc already in progress: creating tablet
I20240502 08:16:35.867525 24181 raft_consensus.cc:2258] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:35.867631 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 3cc30014a1ac45de937751eed36877a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cc30014a1ac45de937751eed36877a9 already in progress: creating tablet
I20240502 08:16:35.868515 24174 tablet_bootstrap.cc:492] T 71c0c2f3def643dfa99ff4efe65f6acf P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:35.868465 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e077b6d407694d3eaa7f0c34e3f8b917" } modify_external_catalogs: true
I20240502 08:16:35.868652 24174 ts_tablet_manager.cc:1367] T 71c0c2f3def643dfa99ff4efe65f6acf P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20240502 08:16:35.868639 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e077b6d407694d3eaa7f0c34e3f8b917" } modify_external_catalogs: true
I20240502 08:16:35.869355 24174 raft_consensus.cc:348] T 71c0c2f3def643dfa99ff4efe65f6acf P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.869550 24174 raft_consensus.cc:374] T 71c0c2f3def643dfa99ff4efe65f6acf P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.869437 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_f52a4fa0e0c04f19a517f946b59a850f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:35.869669 24174 raft_consensus.cc:729] T 71c0c2f3def643dfa99ff4efe65f6acf P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.869891 24174 consensus_queue.cc:260] T 71c0c2f3def643dfa99ff4efe65f6acf P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.869979 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_12fab9a5f68349de8d48fb2acf1da1ba" } new_table_name: "default.Table_4e40fa1cf0ab4dedb2c6aadb2418901b" modify_external_catalogs: true
W20240502 08:16:35.870091 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f52a4fa0e0c04f19a517f946b59a850f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.870525 24181 ts_tablet_manager.cc:1875] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.871250 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_66741640bb3e41808da39a2de1103d0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.871726 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66741640bb3e41808da39a2de1103d0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.872721 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_cdfdcb2e5f8b4b7bbd2d75d2ea00936e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.873116 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdfdcb2e5f8b4b7bbd2d75d2ea00936e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.867795 24179 ts_tablet_manager.cc:1888] T 7a10f86a0f3f4385bef39eda14d4fa9f P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.874218 24179 log.cc:1196] T 7a10f86a0f3f4385bef39eda14d4fa9f P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/7a10f86a0f3f4385bef39eda14d4fa9f
I20240502 08:16:35.874217 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9ae8c17e33fc47d5b93166ab4d269138"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:35.874548 24179 ts_tablet_manager.cc:1909] T 7a10f86a0f3f4385bef39eda14d4fa9f P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
W20240502 08:16:35.874642 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ae8c17e33fc47d5b93166ab4d269138 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.875533 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 7a10f86a0f3f4385bef39eda14d4fa9f (table default.Table_95acc6e36a0b448eb3fd6d5df86e2de4 [id=1328b5d30ef24f98bf00c4429383ce92]) successfully deleted
I20240502 08:16:35.875669 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c29e7767625846bc83fdd0c850f54966"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.876081 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c29e7767625846bc83fdd0c850f54966 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.877059 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a6b10e6dd2db4b5eb3e3d639cac19f54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.877450 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6b10e6dd2db4b5eb3e3d639cac19f54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.878057 24174 ts_tablet_manager.cc:1398] T 71c0c2f3def643dfa99ff4efe65f6acf P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 08:16:35.878329 23603 master_service.cc:946] ReplaceTablet: received request to replace tablet 0f2ff50fa357448494434d1487ce83bc from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:35.879794 23603 master_service.cc:946] ReplaceTablet: received request to replace tablet 6f7518b1e36147a9b9f0c58bca17061a from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:35.880255 24174 tablet_bootstrap.cc:492] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:35.880303 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4dd22967bcf147388ea4ceb9cd24b065" } modify_external_catalogs: true
I20240502 08:16:35.880461 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4dd22967bcf147388ea4ceb9cd24b065" } modify_external_catalogs: true
I20240502 08:16:35.881630 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ce1b3e3032d84f72b573bf1049c30237"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:35.881834 24174 tablet_bootstrap.cc:654] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:35.882083 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce1b3e3032d84f72b573bf1049c30237 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.882684 24175 tablet_bootstrap.cc:492] T 8791834597254e7c9f730e03568c0748 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:35.882841 24175 ts_tablet_manager.cc:1367] T 8791834597254e7c9f730e03568c0748 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.005s
I20240502 08:16:35.883277 24180 tablet_replica.cc:330] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:35.883450 24180 raft_consensus.cc:2229] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.883595 24180 raft_consensus.cc:2258] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.883546 24175 raft_consensus.cc:348] T 8791834597254e7c9f730e03568c0748 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.883908 24175 raft_consensus.cc:374] T 8791834597254e7c9f730e03568c0748 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.884025 24175 raft_consensus.cc:729] T 8791834597254e7c9f730e03568c0748 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.884315 24175 consensus_queue.cc:260] T 8791834597254e7c9f730e03568c0748 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.886852 24175 ts_tablet_manager.cc:1398] T 8791834597254e7c9f730e03568c0748 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:16:35.888551 24175 tablet_bootstrap.cc:492] T 175adf0ecb504e28a3d39f001f977839 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:35.890314 24181 ts_tablet_manager.cc:1888] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.890376 24175 tablet_bootstrap.cc:654] T 175adf0ecb504e28a3d39f001f977839 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.890445 24181 log.cc:1196] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/4ce3e95d5ea7435593b7cfe5a670a76a
I20240502 08:16:35.890810 24181 ts_tablet_manager.cc:1909] T 4ce3e95d5ea7435593b7cfe5a670a76a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:35.891748 24181 tablet_replica.cc:330] T 5def9306dfba4d019d3acece81df2f86 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:35.891943 24181 raft_consensus.cc:2229] T 5def9306dfba4d019d3acece81df2f86 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.892100 24181 raft_consensus.cc:2258] T 5def9306dfba4d019d3acece81df2f86 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.893975 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 4ce3e95d5ea7435593b7cfe5a670a76a (table default.Table_6d48e2ca21a2409fae7c972ec53adbf5 [id=d06bffc628cd4e07a8651559cbe38e09]) successfully deleted
I20240502 08:16:35.895529 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet c1c2a988a3524c8fb7410e8508bd1abc (table default.Table_b96eb662e06846039807538794442017 [id=31ddf074089243a89bab5928860c625e]) successfully deleted
I20240502 08:16:35.895964 24180 ts_tablet_manager.cc:1875] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.898954 24181 ts_tablet_manager.cc:1875] T 5def9306dfba4d019d3acece81df2f86 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.907280 24180 ts_tablet_manager.cc:1888] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.907444 24180 log.cc:1196] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/774487be9bc644cab4e420c3d33b3cad
I20240502 08:16:35.907776 24180 ts_tablet_manager.cc:1909] T 774487be9bc644cab4e420c3d33b3cad P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:35.912887 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c5d06caccb424872b965a751c171a0a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:35.914072 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 8791834597254e7c9f730e03568c0748 with delete_type TABLET_DATA_DELETED (replaced by tablet e9592a041a9543e8b41f043927807131) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.914373 24179 tablet_replica.cc:330] T 8791834597254e7c9f730e03568c0748 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:35.914556 24179 raft_consensus.cc:2229] T 8791834597254e7c9f730e03568c0748 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.914793 24179 raft_consensus.cc:2258] T 8791834597254e7c9f730e03568c0748 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:35.915287 24181 ts_tablet_manager.cc:1888] T 5def9306dfba4d019d3acece81df2f86 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.915542 24179 ts_tablet_manager.cc:1875] T 8791834597254e7c9f730e03568c0748 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:35.916147 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5d06caccb424872b965a751c171a0a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.918242 24181 log.cc:1196] T 5def9306dfba4d019d3acece81df2f86 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/5def9306dfba4d019d3acece81df2f86
I20240502 08:16:35.919059 24181 ts_tablet_manager.cc:1909] T 5def9306dfba4d019d3acece81df2f86 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:35.920231 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_07371eaf91cb4ab199af62382b640d64" } modify_external_catalogs: true
I20240502 08:16:35.920403 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_07371eaf91cb4ab199af62382b640d64" } modify_external_catalogs: true
I20240502 08:16:35.921036 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 5def9306dfba4d019d3acece81df2f86 (table default.Table_00eb74cd24954c52b5484751f035f4c1 [id=b42e15a2350f4c9a8097ccb0cf1aaddc]) successfully deleted
I20240502 08:16:35.923553 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_549acf5604c74ae2bac902a1118e5baa" } new_table_name: "default.Table_498b372c15a04b70b87bd67b543501cf" modify_external_catalogs: true
I20240502 08:16:35.924086 24179 ts_tablet_manager.cc:1888] T 8791834597254e7c9f730e03568c0748 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:35.924211 24179 log.cc:1196] T 8791834597254e7c9f730e03568c0748 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/8791834597254e7c9f730e03568c0748
I20240502 08:16:35.924633 24179 ts_tablet_manager.cc:1909] T 8791834597254e7c9f730e03568c0748 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
W20240502 08:16:35.916616 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba3d4b4584dd45b58ff0daa4f1c09a0a consensus state has no leader
I20240502 08:16:35.925277 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet 1df565c1c6564efd88fa557c50522d7e from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:35.925839 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet 9628134656eb4be1b3e1419e5c1accbe from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:35.926546 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 8791834597254e7c9f730e03568c0748 (table default.Table_6c584db923e147d380281ecb1acd11a1 [id=467a7f6db73641ca85a1898b251938fb]) successfully deleted
I20240502 08:16:35.918397 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fb2b0c0e7cc44f61b97a2863650307b3" } new_table_name: "default.Table_d9b6d863d79341ebb994770c34ae2174" modify_external_catalogs: true
I20240502 08:16:35.919451 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e70e56a53c984b2e8066dd16caafde85" } new_table_name: "default.Table_01a5cb4b00a54d719c7f057e20bbb6e4" modify_external_catalogs: true
I20240502 08:16:35.929920 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 9628134656eb4be1b3e1419e5c1accbe
I20240502 08:16:35.930945 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 9628134656eb4be1b3e1419e5c1accbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.931821 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3f6d07700a2e4aa1b7fb356c54c14b73" } new_table_name: "default.Table_fcd8e870383b4c8f9b40db6fb2a7149b" modify_external_catalogs: true
I20240502 08:16:35.931907 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet 9628134656eb4be1b3e1419e5c1accbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:35.932559 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 9628134656eb4be1b3e1419e5c1accbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9628134656eb4be1b3e1419e5c1accbe already in progress: creating tablet
I20240502 08:16:35.933368 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_68bf91a14ffa42cdbb10a1d7ca56cb08" } modify_external_catalogs: true
I20240502 08:16:35.933702 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_68bf91a14ffa42cdbb10a1d7ca56cb08" } modify_external_catalogs: true
I20240502 08:16:35.935809 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 774487be9bc644cab4e420c3d33b3cad (table default.Table_a0ca472d0e4748e6995f3b55e306e483 [id=70dab926ba6c4a1e8b4cf29853ba5755]) successfully deleted
W20240502 08:16:35.936074 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 3cc30014a1ac45de937751eed36877a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cc30014a1ac45de937751eed36877a9 already in progress: creating tablet
I20240502 08:16:35.937006 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 71c0c2f3def643dfa99ff4efe65f6acf with delete_type TABLET_DATA_DELETED (replaced by tablet 203ed05d0bdc4b1f99207423b7fc848b) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.937274 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 9628134656eb4be1b3e1419e5c1accbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.937413 24180 tablet_replica.cc:330] T 71c0c2f3def643dfa99ff4efe65f6acf P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:35.937608 24180 raft_consensus.cc:2229] T 71c0c2f3def643dfa99ff4efe65f6acf P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.937788 24180 raft_consensus.cc:2258] T 71c0c2f3def643dfa99ff4efe65f6acf P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:35.940238 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 9628134656eb4be1b3e1419e5c1accbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9628134656eb4be1b3e1419e5c1accbe already in progress: creating tablet
I20240502 08:16:35.940410 24180 ts_tablet_manager.cc:1875] T 71c0c2f3def643dfa99ff4efe65f6acf P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.958709 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7b904b0af0a8449581e717c1a56c01ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.959378 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b904b0af0a8449581e717c1a56c01ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.961024 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet b63fe5b6d4db4750800ecd429fa97ed1 from {username='slave'} at 127.0.0.1:36668
W20240502 08:16:35.961539 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 consensus state has no leader
I20240502 08:16:35.961946 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2468b65097c244298ba963e7aa155353"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:35.962313 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 08183523bf4a4d1c9020eb693aa07878 consensus state has no leader
I20240502 08:16:35.964011 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c422ad5c5f164908bfae0d2977635bb3" } modify_external_catalogs: true
I20240502 08:16:35.963043 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 9b1eb762780c450bbaf380d17db564b6
I20240502 08:16:35.964380 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c422ad5c5f164908bfae0d2977635bb3" } modify_external_catalogs: true
I20240502 08:16:35.965698 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b1eb762780c450bbaf380d17db564b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.966288 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b1eb762780c450bbaf380d17db564b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:35.966513 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b1eb762780c450bbaf380d17db564b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.966758 24180 ts_tablet_manager.cc:1888] T 71c0c2f3def643dfa99ff4efe65f6acf P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240502 08:16:35.966805 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 9b1eb762780c450bbaf380d17db564b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b1eb762780c450bbaf380d17db564b6 already in progress: creating tablet
I20240502 08:16:35.967300 24180 log.cc:1196] T 71c0c2f3def643dfa99ff4efe65f6acf P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/71c0c2f3def643dfa99ff4efe65f6acf
W20240502 08:16:35.963310 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8f7b5f018d84680ac5533bf99102d31 consensus state has no leader
I20240502 08:16:35.967793 24180 ts_tablet_manager.cc:1909] T 71c0c2f3def643dfa99ff4efe65f6acf P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:35.963543 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet b63fe5b6d4db4750800ecd429fa97ed1
I20240502 08:16:35.968597 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet b63fe5b6d4db4750800ecd429fa97ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:35.969286 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet b63fe5b6d4db4750800ecd429fa97ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:35.969396 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet b63fe5b6d4db4750800ecd429fa97ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:35.969965 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet b63fe5b6d4db4750800ecd429fa97ed1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b63fe5b6d4db4750800ecd429fa97ed1 already in progress: creating tablet
I20240502 08:16:35.970264 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 71c0c2f3def643dfa99ff4efe65f6acf (table default.Table_68bf91a14ffa42cdbb10a1d7ca56cb08 [id=59d945a73daa4918ba17063eae0b9b93]) successfully deleted
I20240502 08:16:35.970587 24180 tablet_replica.cc:330] T 9628134656eb4be1b3e1419e5c1accbe P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
W20240502 08:16:35.962518 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2468b65097c244298ba963e7aa155353 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:35.970832 24180 raft_consensus.cc:2229] T 9628134656eb4be1b3e1419e5c1accbe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:35.971005 24180 raft_consensus.cc:2258] T 9628134656eb4be1b3e1419e5c1accbe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:35.972230 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab70513985b54416856a29b32cecae4d consensus state has no leader
W20240502 08:16:35.973793 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 9b1eb762780c450bbaf380d17db564b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b1eb762780c450bbaf380d17db564b6 already in progress: creating tablet
W20240502 08:16:35.974575 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet b63fe5b6d4db4750800ecd429fa97ed1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b63fe5b6d4db4750800ecd429fa97ed1 already in progress: creating tablet
I20240502 08:16:35.976997 24180 ts_tablet_manager.cc:1875] T 9628134656eb4be1b3e1419e5c1accbe P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:35.978339 24173 tablet_bootstrap.cc:492] T 9628134656eb4be1b3e1419e5c1accbe P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:35.978536 24173 ts_tablet_manager.cc:1367] T 9628134656eb4be1b3e1419e5c1accbe P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.210s user 0.022s sys 0.028s
I20240502 08:16:35.978943 23891 tablet_service.cc:1467] Processing CreateTablet for tablet ce758c4a9258427987e8e83f579a2aa2 (DEFAULT_TABLE table=default.Table_cb0ea06f5e0b47e5978fc9a5ed8880ce [id=9965057f98614e8284d007156af6fdd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.979352 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce758c4a9258427987e8e83f579a2aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.979244 24173 raft_consensus.cc:348] T 9628134656eb4be1b3e1419e5c1accbe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.979441 24173 raft_consensus.cc:374] T 9628134656eb4be1b3e1419e5c1accbe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.979512 24173 raft_consensus.cc:729] T 9628134656eb4be1b3e1419e5c1accbe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.979725 24173 consensus_queue.cc:260] T 9628134656eb4be1b3e1419e5c1accbe P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.982286 24173 ts_tablet_manager.cc:1398] T 9628134656eb4be1b3e1419e5c1accbe P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:35.984274 24173 tablet_bootstrap.cc:492] T 3052510d77f344bf87decb9c2e5ec0bd P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:35.985836 24173 tablet_bootstrap.cc:654] T 3052510d77f344bf87decb9c2e5ec0bd P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.988479 24038 tablet_service.cc:1467] Processing CreateTablet for tablet ce758c4a9258427987e8e83f579a2aa2 (DEFAULT_TABLE table=default.Table_cb0ea06f5e0b47e5978fc9a5ed8880ce [id=9965057f98614e8284d007156af6fdd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:35.988903 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce758c4a9258427987e8e83f579a2aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.994660 24173 tablet_bootstrap.cc:492] T 3052510d77f344bf87decb9c2e5ec0bd P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:35.994812 24173 ts_tablet_manager.cc:1367] T 3052510d77f344bf87decb9c2e5ec0bd P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20240502 08:16:35.995626 24173 raft_consensus.cc:348] T 3052510d77f344bf87decb9c2e5ec0bd P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.995815 24173 raft_consensus.cc:374] T 3052510d77f344bf87decb9c2e5ec0bd P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.995909 24173 raft_consensus.cc:729] T 3052510d77f344bf87decb9c2e5ec0bd P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.996110 24173 consensus_queue.cc:260] T 3052510d77f344bf87decb9c2e5ec0bd P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:35.997501 24180 ts_tablet_manager.cc:1888] T 9628134656eb4be1b3e1419e5c1accbe P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.000504 24173 ts_tablet_manager.cc:1398] T 3052510d77f344bf87decb9c2e5ec0bd P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:16:36.000675 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet 9628134656eb4be1b3e1419e5c1accbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.000761 24180 log.cc:1196] T 9628134656eb4be1b3e1419e5c1accbe P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/9628134656eb4be1b3e1419e5c1accbe
I20240502 08:16:36.001682 24173 tablet_bootstrap.cc:492] T 774487be9bc644cab4e420c3d33b3cad P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:36.002053 24181 tablet_replica.cc:330] T 9628134656eb4be1b3e1419e5c1accbe P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:36.002254 24181 raft_consensus.cc:2229] T 9628134656eb4be1b3e1419e5c1accbe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.002446 24181 raft_consensus.cc:2258] T 9628134656eb4be1b3e1419e5c1accbe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.002841 24180 ts_tablet_manager.cc:1909] T 9628134656eb4be1b3e1419e5c1accbe P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:36.003528 24181 ts_tablet_manager.cc:1875] T 9628134656eb4be1b3e1419e5c1accbe P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:36.004287 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 9628134656eb4be1b3e1419e5c1accbe (table default.Table_e077b6d407694d3eaa7f0c34e3f8b917 [id=9f2f530c5f91424d91ec92a42602b781]) successfully deleted
W20240502 08:16:36.004611 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 9b1eb762780c450bbaf380d17db564b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b1eb762780c450bbaf380d17db564b6 already in progress: creating tablet
I20240502 08:16:36.005669 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet 3052510d77f344bf87decb9c2e5ec0bd with delete_type TABLET_DATA_DELETED (replaced by tablet 23ce83c8001d4b2988b88c0ed9bf948c) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.006583 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 203ed05d0bdc4b1f99207423b7fc848b
I20240502 08:16:36.007678 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet 203ed05d0bdc4b1f99207423b7fc848b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.007876 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 6788ffd2eb2248c3b84e474de56c37a9 (DEFAULT_TABLE table=default.Table_59462d2ccf514f0da864562120217931 [id=62273209104a4a8e9f24a9a77d2a90f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.008369 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6788ffd2eb2248c3b84e474de56c37a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.009785 24181 ts_tablet_manager.cc:1888] T 9628134656eb4be1b3e1419e5c1accbe P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.009943 24181 log.cc:1196] T 9628134656eb4be1b3e1419e5c1accbe P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/9628134656eb4be1b3e1419e5c1accbe
W20240502 08:16:36.009693 24241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:36.010195 24241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20240502 08:16:36.010288 24181 ts_tablet_manager.cc:1909] T 9628134656eb4be1b3e1419e5c1accbe P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
W20240502 08:16:36.010409 24241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:36.010571 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 203ed05d0bdc4b1f99207423b7fc848b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 203ed05d0bdc4b1f99207423b7fc848b already in progress: creating tablet
I20240502 08:16:36.010947 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet 203ed05d0bdc4b1f99207423b7fc848b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.011261 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 9628134656eb4be1b3e1419e5c1accbe (table default.Table_e077b6d407694d3eaa7f0c34e3f8b917 [id=9f2f530c5f91424d91ec92a42602b781]) successfully deleted
I20240502 08:16:36.012924 24173 tablet_bootstrap.cc:654] T 774487be9bc644cab4e420c3d33b3cad P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.011165 24181 tablet_replica.cc:330] T 3052510d77f344bf87decb9c2e5ec0bd P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:36.013236 24181 raft_consensus.cc:2229] T 3052510d77f344bf87decb9c2e5ec0bd P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.013446 24181 raft_consensus.cc:2258] T 3052510d77f344bf87decb9c2e5ec0bd P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.014689 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 6788ffd2eb2248c3b84e474de56c37a9 (DEFAULT_TABLE table=default.Table_59462d2ccf514f0da864562120217931 [id=62273209104a4a8e9f24a9a77d2a90f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.015332 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6788ffd2eb2248c3b84e474de56c37a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.015599 24181 ts_tablet_manager.cc:1875] T 3052510d77f344bf87decb9c2e5ec0bd P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:36.016860 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet b63fe5b6d4db4750800ecd429fa97ed1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b63fe5b6d4db4750800ecd429fa97ed1 already in progress: creating tablet
I20240502 08:16:36.017915 23762 tablet_service.cc:1467] Processing CreateTablet for tablet ce758c4a9258427987e8e83f579a2aa2 (DEFAULT_TABLE table=default.Table_cb0ea06f5e0b47e5978fc9a5ed8880ce [id=9965057f98614e8284d007156af6fdd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.018324 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce758c4a9258427987e8e83f579a2aa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.020982 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 203ed05d0bdc4b1f99207423b7fc848b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:36.022789 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 203ed05d0bdc4b1f99207423b7fc848b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 203ed05d0bdc4b1f99207423b7fc848b already in progress: creating tablet
I20240502 08:16:36.025065 24173 tablet_bootstrap.cc:492] T 774487be9bc644cab4e420c3d33b3cad P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:36.025266 24173 ts_tablet_manager.cc:1367] T 774487be9bc644cab4e420c3d33b3cad P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240502 08:16:36.026031 24173 raft_consensus.cc:348] T 774487be9bc644cab4e420c3d33b3cad P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.026257 24173 raft_consensus.cc:374] T 774487be9bc644cab4e420c3d33b3cad P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.026329 24173 raft_consensus.cc:729] T 774487be9bc644cab4e420c3d33b3cad P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.026648 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 1065732d7333413d984ed56275fd61a9
I20240502 08:16:36.027823 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet 1065732d7333413d984ed56275fd61a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.027837 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 1065732d7333413d984ed56275fd61a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.028610 24181 ts_tablet_manager.cc:1888] T 3052510d77f344bf87decb9c2e5ec0bd P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.028641 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 1065732d7333413d984ed56275fd61a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.028746 24181 log.cc:1196] T 3052510d77f344bf87decb9c2e5ec0bd P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/3052510d77f344bf87decb9c2e5ec0bd
I20240502 08:16:36.029101 24181 ts_tablet_manager.cc:1909] T 3052510d77f344bf87decb9c2e5ec0bd P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
W20240502 08:16:36.030373 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 1065732d7333413d984ed56275fd61a9 because the tablet was not found. No further retry: Not found: Tablet not found: 1065732d7333413d984ed56275fd61a9
I20240502 08:16:36.030400 24173 consensus_queue.cc:260] T 774487be9bc644cab4e420c3d33b3cad P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.030882 24173 ts_tablet_manager.cc:1398] T 774487be9bc644cab4e420c3d33b3cad P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
W20240502 08:16:36.032249 23597 catalog_manager.cc:4797] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 1065732d7333413d984ed56275fd61a9 because the tablet was not found. No further retry: Not found: Tablet not found: 1065732d7333413d984ed56275fd61a9
I20240502 08:16:36.033010 24173 tablet_bootstrap.cc:492] T 71c0c2f3def643dfa99ff4efe65f6acf P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:36.033133 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 3052510d77f344bf87decb9c2e5ec0bd (table default.Table_623eac7101f94d959fb83b6f51d8fe0a [id=54454efbe9394170822f237dc8e6115e]) successfully deleted
W20240502 08:16:36.033355 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 203ed05d0bdc4b1f99207423b7fc848b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 203ed05d0bdc4b1f99207423b7fc848b already in progress: creating tablet
W20240502 08:16:36.033715 23597 catalog_manager.cc:4797] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 1065732d7333413d984ed56275fd61a9 because the tablet was not found. No further retry: Not found: Tablet not found: 1065732d7333413d984ed56275fd61a9
I20240502 08:16:36.034659 24173 tablet_bootstrap.cc:654] T 71c0c2f3def643dfa99ff4efe65f6acf P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.036360 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 774487be9bc644cab4e420c3d33b3cad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.036665 24181 tablet_replica.cc:330] T 774487be9bc644cab4e420c3d33b3cad P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:36.036896 24181 raft_consensus.cc:2229] T 774487be9bc644cab4e420c3d33b3cad P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.037127 24181 raft_consensus.cc:2258] T 774487be9bc644cab4e420c3d33b3cad P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.037884 24181 ts_tablet_manager.cc:1875] T 774487be9bc644cab4e420c3d33b3cad P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:36.040131 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 6788ffd2eb2248c3b84e474de56c37a9 (DEFAULT_TABLE table=default.Table_59462d2ccf514f0da864562120217931 [id=62273209104a4a8e9f24a9a77d2a90f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.040597 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6788ffd2eb2248c3b84e474de56c37a9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:36.041134 24241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:16:36.041352 24241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:36.041517 24241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:16:36.041700 24241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:16:36.043412 24173 tablet_bootstrap.cc:492] T 71c0c2f3def643dfa99ff4efe65f6acf P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:36.043547 24173 ts_tablet_manager.cc:1367] T 71c0c2f3def643dfa99ff4efe65f6acf P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240502 08:16:36.044306 24173 raft_consensus.cc:348] T 71c0c2f3def643dfa99ff4efe65f6acf P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.044523 24173 raft_consensus.cc:374] T 71c0c2f3def643dfa99ff4efe65f6acf P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.044615 24173 raft_consensus.cc:729] T 71c0c2f3def643dfa99ff4efe65f6acf P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.044839 24173 consensus_queue.cc:260] T 71c0c2f3def643dfa99ff4efe65f6acf P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.045398 24173 ts_tablet_manager.cc:1398] T 71c0c2f3def643dfa99ff4efe65f6acf P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:36.045686 24173 tablet_bootstrap.cc:492] T c1c2a988a3524c8fb7410e8508bd1abc P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:36.047286 24173 tablet_bootstrap.cc:654] T c1c2a988a3524c8fb7410e8508bd1abc P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.052268 23767 tablet_service.cc:1467] Processing CreateTablet for tablet e134e90efb4a48bb82b53a80bda708b3 (DEFAULT_TABLE table=default.Table_cc80d1cedafa4e71906e5e9530fdd828 [id=47682c26a85842c7b8991bf3828a9336]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.052414 24181 ts_tablet_manager.cc:1888] T 774487be9bc644cab4e420c3d33b3cad P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.052757 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e134e90efb4a48bb82b53a80bda708b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.054888 24181 log.cc:1196] T 774487be9bc644cab4e420c3d33b3cad P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/774487be9bc644cab4e420c3d33b3cad
I20240502 08:16:36.055502 24181 ts_tablet_manager.cc:1909] T 774487be9bc644cab4e420c3d33b3cad P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:36.056928 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 774487be9bc644cab4e420c3d33b3cad (table default.Table_a0ca472d0e4748e6995f3b55e306e483 [id=70dab926ba6c4a1e8b4cf29853ba5755]) successfully deleted
I20240502 08:16:36.058274 24173 tablet_bootstrap.cc:492] T c1c2a988a3524c8fb7410e8508bd1abc P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:36.058439 24173 ts_tablet_manager.cc:1367] T c1c2a988a3524c8fb7410e8508bd1abc P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.001s
I20240502 08:16:36.059201 24173 raft_consensus.cc:348] T c1c2a988a3524c8fb7410e8508bd1abc P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.059412 24173 raft_consensus.cc:374] T c1c2a988a3524c8fb7410e8508bd1abc P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.059509 24173 raft_consensus.cc:729] T c1c2a988a3524c8fb7410e8508bd1abc P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.059739 24173 consensus_queue.cc:260] T c1c2a988a3524c8fb7410e8508bd1abc P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.060281 24173 ts_tablet_manager.cc:1398] T c1c2a988a3524c8fb7410e8508bd1abc P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:36.063150 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 774487be9bc644cab4e420c3d33b3cad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.064409 24173 tablet_bootstrap.cc:492] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:36.064668 23597 catalog_manager.cc:4797] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 774487be9bc644cab4e420c3d33b3cad because the tablet was not found. No further retry: Not found: Tablet not found: 774487be9bc644cab4e420c3d33b3cad
I20240502 08:16:36.066187 23898 tablet_service.cc:1467] Processing CreateTablet for tablet e134e90efb4a48bb82b53a80bda708b3 (DEFAULT_TABLE table=default.Table_cc80d1cedafa4e71906e5e9530fdd828 [id=47682c26a85842c7b8991bf3828a9336]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.066188 23890 tablet_service.cc:1467] Processing CreateTablet for tablet c470d420c45843a69a63d9800b7cfc97 (DEFAULT_TABLE table=default.Table_95acc6e36a0b448eb3fd6d5df86e2de4 [id=1328b5d30ef24f98bf00c4429383ce92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.066713 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e134e90efb4a48bb82b53a80bda708b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.066661 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 21832f0151ec4cf48f51f9bd2cb2b5ec (DEFAULT_TABLE table=default.Table_75cadbecaad04383945a6e0b1e226d86 [id=4c539e8625094a278029356ac44da4c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.067015 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21832f0151ec4cf48f51f9bd2cb2b5ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.067271 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c470d420c45843a69a63d9800b7cfc97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.068068 24036 tablet_service.cc:1467] Processing CreateTablet for tablet e134e90efb4a48bb82b53a80bda708b3 (DEFAULT_TABLE table=default.Table_cc80d1cedafa4e71906e5e9530fdd828 [id=47682c26a85842c7b8991bf3828a9336]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.068426 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e134e90efb4a48bb82b53a80bda708b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.069896 24034 tablet_service.cc:1467] Processing CreateTablet for tablet c470d420c45843a69a63d9800b7cfc97 (DEFAULT_TABLE table=default.Table_95acc6e36a0b448eb3fd6d5df86e2de4 [id=1328b5d30ef24f98bf00c4429383ce92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.070233 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c470d420c45843a69a63d9800b7cfc97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.075178 23767 tablet_service.cc:1467] Processing CreateTablet for tablet c470d420c45843a69a63d9800b7cfc97 (DEFAULT_TABLE table=default.Table_95acc6e36a0b448eb3fd6d5df86e2de4 [id=1328b5d30ef24f98bf00c4429383ce92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.075588 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c470d420c45843a69a63d9800b7cfc97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.075835 23766 tablet_service.cc:1467] Processing CreateTablet for tablet b6417aafbdcf416ea7bff56bce5e6ddb (DEFAULT_TABLE table=default.Table_47643e78190448bf93ad852cabe9b8ff [id=9ff8ea75a9bc4de194a70fe2088a94bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.076036 23765 tablet_service.cc:1467] Processing CreateTablet for tablet dcdd765895254a39b86b692b5d812c83 (DEFAULT_TABLE table=default.Table_dd3c838f4a014e178351d135f5d4567d [id=6be0b9c539dd40059dab4bae00b7b180]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.076124 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6417aafbdcf416ea7bff56bce5e6ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.075590 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 0be4f8d5cc09483d8d1aa1e37679b544 (DEFAULT_TABLE table=default.Table_178527d31d7044fc98fa41886d651049 [id=6dc275aed42b42118a70099d3eae3c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.076359 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcdd765895254a39b86b692b5d812c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.077207 23761 tablet_service.cc:1467] Processing CreateTablet for tablet b11f504b93574e42a9bf30fde6d08207 (DEFAULT_TABLE table=default.Table_18007012fdb34e80ad040694f0f110bc [id=7900f407998344dfa30c1468428ef42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.077610 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b11f504b93574e42a9bf30fde6d08207. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.077848 23760 tablet_service.cc:1467] Processing CreateTablet for tablet d8f7b5f018d84680ac5533bf99102d31 (DEFAULT_TABLE table=default.Table_498b372c15a04b70b87bd67b543501cf [id=38e955d6bd644d24a7c45172ab60282b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.078234 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f7b5f018d84680ac5533bf99102d31. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:36.078259 23996 consensus_peers.cc:485] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:36.077533 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 21832f0151ec4cf48f51f9bd2cb2b5ec (DEFAULT_TABLE table=default.Table_75cadbecaad04383945a6e0b1e226d86 [id=4c539e8625094a278029356ac44da4c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.078727 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21832f0151ec4cf48f51f9bd2cb2b5ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.081452 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 10adabdd31f848548ef6f2408ccc2050 (DEFAULT_TABLE table=default.Table_c45e7b653ae641b293b87c150748c864 [id=5ff5989d908243c9a58013285842dd46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.081682 23763 tablet_service.cc:1467] Processing CreateTablet for tablet dead13bba95f43cb809e17f6d7fba028 (DEFAULT_TABLE table=default.Table_dae0bce7072a4d50aa63b702ee775fcf [id=7409c7e1a3534932a787559a679e7098]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.081806 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10adabdd31f848548ef6f2408ccc2050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.082043 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dead13bba95f43cb809e17f6d7fba028. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.082131 23787 raft_consensus.cc:1261] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Refusing update from remote peer 149ce744feab49b5aa5bcaa14bd6aba5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:36.082857 24290 consensus_queue.cc:1035] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:36.084141 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be4f8d5cc09483d8d1aa1e37679b544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.085875 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 1065732d7333413d984ed56275fd61a9 (DEFAULT_TABLE table=default.Table_c422ad5c5f164908bfae0d2977635bb3 [id=3584daf64dec4be7a42a2eee267b4861]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.086226 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1065732d7333413d984ed56275fd61a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.089022 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 4302fec5c51e4c6c841710bab50b2997 (DEFAULT_TABLE table=default.Table_adde4be2323646faa59e19d7e9d189e5 [id=e3f148f833bc4d99a3ab2ec8d5258507]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.089481 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4302fec5c51e4c6c841710bab50b2997. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.091161 24039 tablet_service.cc:1467] Processing CreateTablet for tablet b11f504b93574e42a9bf30fde6d08207 (DEFAULT_TABLE table=default.Table_18007012fdb34e80ad040694f0f110bc [id=7900f407998344dfa30c1468428ef42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.091284 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 4302fec5c51e4c6c841710bab50b2997 (DEFAULT_TABLE table=default.Table_adde4be2323646faa59e19d7e9d189e5 [id=e3f148f833bc4d99a3ab2ec8d5258507]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.091524 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 0be4f8d5cc09483d8d1aa1e37679b544 (DEFAULT_TABLE table=default.Table_178527d31d7044fc98fa41886d651049 [id=6dc275aed42b42118a70099d3eae3c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.091854 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet 71c0c2f3def643dfa99ff4efe65f6acf with delete_type TABLET_DATA_DELETED (replaced by tablet 203ed05d0bdc4b1f99207423b7fc848b) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.092044 24032 tablet_service.cc:1467] Processing CreateTablet for tablet b6417aafbdcf416ea7bff56bce5e6ddb (DEFAULT_TABLE table=default.Table_47643e78190448bf93ad852cabe9b8ff [id=9ff8ea75a9bc4de194a70fe2088a94bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.092531 24037 tablet_service.cc:1467] Processing CreateTablet for tablet dead13bba95f43cb809e17f6d7fba028 (DEFAULT_TABLE table=default.Table_dae0bce7072a4d50aa63b702ee775fcf [id=7409c7e1a3534932a787559a679e7098]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.092770 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 1065732d7333413d984ed56275fd61a9 (DEFAULT_TABLE table=default.Table_c422ad5c5f164908bfae0d2977635bb3 [id=3584daf64dec4be7a42a2eee267b4861]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.092532 24035 tablet_service.cc:1467] Processing CreateTablet for tablet dcdd765895254a39b86b692b5d812c83 (DEFAULT_TABLE table=default.Table_dd3c838f4a014e178351d135f5d4567d [id=6be0b9c539dd40059dab4bae00b7b180]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.091536 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b11f504b93574e42a9bf30fde6d08207. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.094408 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 4302fec5c51e4c6c841710bab50b2997 (DEFAULT_TABLE table=default.Table_adde4be2323646faa59e19d7e9d189e5 [id=e3f148f833bc4d99a3ab2ec8d5258507]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.094530 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 0be4f8d5cc09483d8d1aa1e37679b544 (DEFAULT_TABLE table=default.Table_178527d31d7044fc98fa41886d651049 [id=6dc275aed42b42118a70099d3eae3c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.094931 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4302fec5c51e4c6c841710bab50b2997. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.095115 23892 tablet_service.cc:1467] Processing CreateTablet for tablet b6417aafbdcf416ea7bff56bce5e6ddb (DEFAULT_TABLE table=default.Table_47643e78190448bf93ad852cabe9b8ff [id=9ff8ea75a9bc4de194a70fe2088a94bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.095448 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6417aafbdcf416ea7bff56bce5e6ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.095593 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 10adabdd31f848548ef6f2408ccc2050 (DEFAULT_TABLE table=default.Table_c45e7b653ae641b293b87c150748c864 [id=5ff5989d908243c9a58013285842dd46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.096127 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10adabdd31f848548ef6f2408ccc2050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.097769 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c2a988a3524c8fb7410e8508bd1abc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.097815 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcdd765895254a39b86b692b5d812c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.097848 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 10adabdd31f848548ef6f2408ccc2050 (DEFAULT_TABLE table=default.Table_c45e7b653ae641b293b87c150748c864 [id=5ff5989d908243c9a58013285842dd46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.097903 23899 tablet_service.cc:1467] Processing CreateTablet for tablet dcdd765895254a39b86b692b5d812c83 (DEFAULT_TABLE table=default.Table_dd3c838f4a014e178351d135f5d4567d [id=6be0b9c539dd40059dab4bae00b7b180]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.098210 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10adabdd31f848548ef6f2408ccc2050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.098237 23894 tablet_service.cc:1467] Processing CreateTablet for tablet b11f504b93574e42a9bf30fde6d08207 (DEFAULT_TABLE table=default.Table_18007012fdb34e80ad040694f0f110bc [id=7900f407998344dfa30c1468428ef42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.098271 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 1065732d7333413d984ed56275fd61a9 (DEFAULT_TABLE table=default.Table_c422ad5c5f164908bfae0d2977635bb3 [id=3584daf64dec4be7a42a2eee267b4861]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.099207 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be4f8d5cc09483d8d1aa1e37679b544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.101831 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b11f504b93574e42a9bf30fde6d08207. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.102025 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be4f8d5cc09483d8d1aa1e37679b544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.103590 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4302fec5c51e4c6c841710bab50b2997. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.104887 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1065732d7333413d984ed56275fd61a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.105762 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6417aafbdcf416ea7bff56bce5e6ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.109086 23891 tablet_service.cc:1467] Processing CreateTablet for tablet dead13bba95f43cb809e17f6d7fba028 (DEFAULT_TABLE table=default.Table_dae0bce7072a4d50aa63b702ee775fcf [id=7409c7e1a3534932a787559a679e7098]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.109201 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcdd765895254a39b86b692b5d812c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.109414 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet d5bab1b7c48248a7973c27d0a865b34b from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:36.109474 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dead13bba95f43cb809e17f6d7fba028. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.109829 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dead13bba95f43cb809e17f6d7fba028. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.110065 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1065732d7333413d984ed56275fd61a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.111251 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_bcd1014009b046a7ba72d1d757e69e26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.111850 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bcd1014009b046a7ba72d1d757e69e26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.116034 24038 tablet_service.cc:1467] Processing CreateTablet for tablet d8f7b5f018d84680ac5533bf99102d31 (DEFAULT_TABLE table=default.Table_498b372c15a04b70b87bd67b543501cf [id=38e955d6bd644d24a7c45172ab60282b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.116402 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f7b5f018d84680ac5533bf99102d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.119594 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_14c4c753725c4160b1e9f57eda5f17b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.120183 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14c4c753725c4160b1e9f57eda5f17b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.123912 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_59b22bd34c884d0ebe95d6737550e50d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:36.123967 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_bbf9d7694fe942de8aa8bd4ed61336a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.124496 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59b22bd34c884d0ebe95d6737550e50d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:36.124598 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbf9d7694fe942de8aa8bd4ed61336a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.131031 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2f863e7f34fb4bb7a9770adbca87da84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:36.131188 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_40bac43800874d1789e8ee318c61f0ee" } new_table_name: "default.Table_434596ffae0941b19dee488e942d764a" modify_external_catalogs: true
W20240502 08:16:36.131666 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f863e7f34fb4bb7a9770adbca87da84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.135455 24241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.84:40725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.22.225.126:43265,127.22.225.125:36189,127.22.225.124:41745
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.126:43265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.126
--webserver_port=34035
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:36.137274 24241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:36.139801 24241 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:16:36.141963 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet 0f2ff50fa357448494434d1487ce83bc from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:36.142618 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3602f20179c64041a63ab185d9359de6" } modify_external_catalogs: true
I20240502 08:16:36.142805 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3602f20179c64041a63ab185d9359de6" } modify_external_catalogs: true
I20240502 08:16:36.150101 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet 32150c7e130f4069a2a3a02e625bfb93 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:36.153388 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet a7c3800fb2784a1dba12d01c56c226e5 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:36.156327 24174 tablet_bootstrap.cc:492] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:36.156476 24174 ts_tablet_manager.cc:1367] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.276s user 0.024s sys 0.028s
I20240502 08:16:36.157920 23890 tablet_service.cc:1467] Processing CreateTablet for tablet d8f7b5f018d84680ac5533bf99102d31 (DEFAULT_TABLE table=default.Table_498b372c15a04b70b87bd67b543501cf [id=38e955d6bd644d24a7c45172ab60282b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.158329 24173 tablet_bootstrap.cc:654] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.159907 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f7b5f018d84680ac5533bf99102d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.160352 24174 raft_consensus.cc:348] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.160573 24174 raft_consensus.cc:374] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.161243 24174 raft_consensus.cc:729] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.161681 24174 consensus_queue.cc:260] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:36.163122 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7a19310948e248d1a85479c41c61bda2 has no consensus state
I20240502 08:16:36.163748 24174 ts_tablet_manager.cc:1398] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:16:36.164629 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a286e67e441e43a1b4738cd219dbea98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.165376 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a286e67e441e43a1b4738cd219dbea98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.165072 24174 tablet_bootstrap.cc:492] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:36.168896 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e0bf76fa2b2d451eb0102e9897669b6f" } new_table_name: "default.Table_2b5aaf0ddb384715934e7eb81369ecac" modify_external_catalogs: true
I20240502 08:16:36.170600 24174 tablet_bootstrap.cc:654] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.173311 24173 tablet_bootstrap.cc:492] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:36.173527 24173 ts_tablet_manager.cc:1367] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.109s user 0.006s sys 0.000s
I20240502 08:16:36.173916 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b9d00c29b8c2498895efd283a90c12e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.174717 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9d00c29b8c2498895efd283a90c12e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.175050 24173 raft_consensus.cc:348] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.175491 24173 raft_consensus.cc:374] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.175697 24173 raft_consensus.cc:729] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.176810 24173 consensus_queue.cc:260] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.179824 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 21832f0151ec4cf48f51f9bd2cb2b5ec (DEFAULT_TABLE table=default.Table_75cadbecaad04383945a6e0b1e226d86 [id=4c539e8625094a278029356ac44da4c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.179991 24173 ts_tablet_manager.cc:1398] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:16:36.180424 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21832f0151ec4cf48f51f9bd2cb2b5ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.181526 24173 tablet_bootstrap.cc:492] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:36.192662 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_75cadbecaad04383945a6e0b1e226d86" } modify_external_catalogs: true
I20240502 08:16:36.192903 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_75cadbecaad04383945a6e0b1e226d86" } modify_external_catalogs: true
I20240502 08:16:36.193492 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_18007012fdb34e80ad040694f0f110bc" } new_table_name: "default.Table_73bf7705ba9847519b86284f595f3f10" modify_external_catalogs: true
I20240502 08:16:36.195020 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_da5e259b0d0c47cf94ab30e3e98d1540" } modify_external_catalogs: true
I20240502 08:16:36.195243 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_da5e259b0d0c47cf94ab30e3e98d1540" } modify_external_catalogs: true
I20240502 08:16:36.195863 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e54897c4ce62463f9cf7baba62b2706a" } new_table_name: "default.Table_cd9334e686454c59999b7e3632c8195f" modify_external_catalogs: true
I20240502 08:16:36.197839 24290 raft_consensus.cc:482] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:36.197993 24290 raft_consensus.cc:504] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.198688 24290 leader_election.cc:288] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:36.199338 24173 tablet_bootstrap.cc:654] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.199311 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db80a159f14e485a971fd8bf58a21c4a" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:36.199877 23996 leader_election.cc:341] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:36.201087 23993 leader_election.cc:341] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: db80a159f14e485a971fd8bf58a21c4a
I20240502 08:16:36.201239 23993 leader_election.cc:302] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:36.201781 24290 raft_consensus.cc:2726] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:36.202510 24181 tablet_replica.cc:330] T 71c0c2f3def643dfa99ff4efe65f6acf P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:36.202989 24181 raft_consensus.cc:2229] T 71c0c2f3def643dfa99ff4efe65f6acf P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.203485 24181 raft_consensus.cc:2258] T 71c0c2f3def643dfa99ff4efe65f6acf P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.203775 24173 tablet_bootstrap.cc:492] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:36.203960 24173 ts_tablet_manager.cc:1367] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240502 08:16:36.204958 24181 ts_tablet_manager.cc:1875] T 71c0c2f3def643dfa99ff4efe65f6acf P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:36.205896 24174 tablet_bootstrap.cc:492] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:36.206086 24174 ts_tablet_manager.cc:1367] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.004s
I20240502 08:16:36.206856 24174 raft_consensus.cc:348] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.207108 24174 raft_consensus.cc:374] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.207226 24174 raft_consensus.cc:729] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.207510 24174 consensus_queue.cc:260] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.208065 24174 ts_tablet_manager.cc:1398] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:36.208401 24174 tablet_bootstrap.cc:492] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:36.212460 24174 tablet_bootstrap.cc:654] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.212406 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet db80a159f14e485a971fd8bf58a21c4a with delete_type TABLET_DATA_DELETED (replaced by tablet e134e90efb4a48bb82b53a80bda708b3) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.204826 24173 raft_consensus.cc:348] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.213384 24173 raft_consensus.cc:374] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.213480 24173 raft_consensus.cc:729] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.213788 24173 consensus_queue.cc:260] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.214387 24173 ts_tablet_manager.cc:1398] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
W20240502 08:16:36.217617 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b11f504b93574e42a9bf30fde6d08207 consensus state has no leader
I20240502 08:16:36.219734 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_434596ffae0941b19dee488e942d764a" } new_table_name: "default.Table_c11fafb2f1854834893ad5c076f1ca11" modify_external_catalogs: true
I20240502 08:16:36.220717 24243 raft_consensus.cc:482] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:36.220892 24243 raft_consensus.cc:504] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:36.221385 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43e870c3a0e340d1bad1c1fa0380a762 consensus state has no leader
I20240502 08:16:36.221649 24243 leader_election.cc:288] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:36.222028 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2df84a3eb090456e8da6106c0a17131a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.222712 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2df84a3eb090456e8da6106c0a17131a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:36.222777 23723 leader_election.cc:341] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 3052510d77f344bf87decb9c2e5ec0bd
I20240502 08:16:36.226131 24032 tablet_service.cc:1467] Processing CreateTablet for tablet fd57b0357ca749c389bc61add20cd3e7 (DEFAULT_TABLE table=default.Table_f52a4fa0e0c04f19a517f946b59a850f [id=fd141979a86c45bf9648396d086f80d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.226303 24040 tablet_service.cc:1467] Processing CreateTablet for tablet e0af5f4daf3149738647a08219efb245 (DEFAULT_TABLE table=default.Table_c29e7767625846bc83fdd0c850f54966 [id=57722a7341c541b19efa213bcf48eb53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.227811 23607 master_service.cc:946] ReplaceTablet: received request to replace tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:36.228149 24293 tablet.cc:1722] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:36.229529 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0ee9c5ee2b7242c3ad3270e17a4b407a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.231027 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ee9c5ee2b7242c3ad3270e17a4b407a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.231027 24175 tablet_bootstrap.cc:492] T 175adf0ecb504e28a3d39f001f977839 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:36.231289 24175 ts_tablet_manager.cc:1367] T 175adf0ecb504e28a3d39f001f977839 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.343s user 0.019s sys 0.062s
I20240502 08:16:36.231024 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3052510d77f344bf87decb9c2e5ec0bd" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:36.231567 23891 tablet_service.cc:1467] Processing CreateTablet for tablet fd57b0357ca749c389bc61add20cd3e7 (DEFAULT_TABLE table=default.Table_f52a4fa0e0c04f19a517f946b59a850f [id=fd141979a86c45bf9648396d086f80d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.232033 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd57b0357ca749c389bc61add20cd3e7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:36.233350 23723 leader_election.cc:341] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:36.233521 23723 leader_election.cc:302] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:36.234372 24243 raft_consensus.cc:2726] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:36.235165 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 7a19310948e248d1a85479c41c61bda2 (DEFAULT_TABLE table=default.Table_434596ffae0941b19dee488e942d764a [id=8521bfc13ef040a78cfb2cc4501b7f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.235301 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 7a19310948e248d1a85479c41c61bda2 (DEFAULT_TABLE table=default.Table_434596ffae0941b19dee488e942d764a [id=8521bfc13ef040a78cfb2cc4501b7f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.235590 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a19310948e248d1a85479c41c61bda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.236222 23760 tablet_service.cc:1467] Processing CreateTablet for tablet b844458ae73f413cbf2113027a5af359 (DEFAULT_TABLE table=default.Table_cdfdcb2e5f8b4b7bbd2d75d2ea00936e [id=fd93259a378b4ea582a6661a59104abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.236627 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b844458ae73f413cbf2113027a5af359. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.237800 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 49b816ea1d474f2db94119441146ba78 (DEFAULT_TABLE table=default.Table_9ae8c17e33fc47d5b93166ab4d269138 [id=a0606a4053d04dd3814f09419a4a70c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.238020 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 01675f637a5a4645949abfd87b3d09a2 (DEFAULT_TABLE table=default.Table_c5d06caccb424872b965a751c171a0a6 [id=4411ea4d183f4694b16b6255567ab62b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.238412 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01675f637a5a4645949abfd87b3d09a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.238374 24175 raft_consensus.cc:348] T 175adf0ecb504e28a3d39f001f977839 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:36.238695 24175 raft_consensus.cc:374] T 175adf0ecb504e28a3d39f001f977839 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.238832 24175 raft_consensus.cc:729] T 175adf0ecb504e28a3d39f001f977839 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.239166 24175 consensus_queue.cc:260] T 175adf0ecb504e28a3d39f001f977839 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:36.241525 23899 tablet_service.cc:1467] Processing CreateTablet for tablet f36aadfa1d264f23bcafa91744aed36d (DEFAULT_TABLE table=default.Table_a6b10e6dd2db4b5eb3e3d639cac19f54 [id=e18870bff56747b192b4698f1ca3a8ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.242174 23766 tablet_service.cc:1467] Processing CreateTablet for tablet f36aadfa1d264f23bcafa91744aed36d (DEFAULT_TABLE table=default.Table_a6b10e6dd2db4b5eb3e3d639cac19f54 [id=e18870bff56747b192b4698f1ca3a8ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.242664 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f36aadfa1d264f23bcafa91744aed36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.242609 24175 ts_tablet_manager.cc:1398] T 175adf0ecb504e28a3d39f001f977839 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.011s user 0.001s sys 0.002s
I20240502 08:16:36.243343 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a19310948e248d1a85479c41c61bda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.244289 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49b816ea1d474f2db94119441146ba78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.245690 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 36f2d7b6f4e442ae95b2d1002d571b6a (DEFAULT_TABLE table=default.Table_66741640bb3e41808da39a2de1103d0d [id=a95a99a58cde44fe814b3edfcc643a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.246016 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36f2d7b6f4e442ae95b2d1002d571b6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.246731 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 5c95dfda9d3b4e2ca9397b77ad71c011 (DEFAULT_TABLE table=default.Table_c201704a319a4e078054899061f76615 [id=71234c2c912b4a79b7b08d87cab6fad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.247071 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c95dfda9d3b4e2ca9397b77ad71c011. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.249754 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f36aadfa1d264f23bcafa91744aed36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.244117 24175 tablet_bootstrap.cc:492] T ec4028a8e66442bfabac9e587aed00bb P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:36.255257 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 5c95dfda9d3b4e2ca9397b77ad71c011 (DEFAULT_TABLE table=default.Table_c201704a319a4e078054899061f76615 [id=71234c2c912b4a79b7b08d87cab6fad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.255575 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 36f2d7b6f4e442ae95b2d1002d571b6a (DEFAULT_TABLE table=default.Table_66741640bb3e41808da39a2de1103d0d [id=a95a99a58cde44fe814b3edfcc643a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.255899 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36f2d7b6f4e442ae95b2d1002d571b6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.256281 23765 tablet_service.cc:1467] Processing CreateTablet for tablet fd57b0357ca749c389bc61add20cd3e7 (DEFAULT_TABLE table=default.Table_f52a4fa0e0c04f19a517f946b59a850f [id=fd141979a86c45bf9648396d086f80d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.256587 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd57b0357ca749c389bc61add20cd3e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.257444 23763 tablet_service.cc:1467] Processing CreateTablet for tablet e0af5f4daf3149738647a08219efb245 (DEFAULT_TABLE table=default.Table_c29e7767625846bc83fdd0c850f54966 [id=57722a7341c541b19efa213bcf48eb53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.257879 23893 tablet_service.cc:1467] Processing CreateTablet for tablet e0af5f4daf3149738647a08219efb245 (DEFAULT_TABLE table=default.Table_c29e7767625846bc83fdd0c850f54966 [id=57722a7341c541b19efa213bcf48eb53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.258004 23608 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 175adf0ecb504e28a3d39f001f977839 (table default.Table_9530003aa5b841b683920da217d58576 [id=a6a5f4c65807479392c01ac5c955c959]). Expected version 1 got 0
I20240502 08:16:36.258188 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0af5f4daf3149738647a08219efb245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.258791 24175 tablet_bootstrap.cc:654] T ec4028a8e66442bfabac9e587aed00bb P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.260608 24173 tablet_bootstrap.cc:492] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:36.260725 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 8810b6f61db14fed8b12d36fed413dc8 (DEFAULT_TABLE table=default.Table_ce1b3e3032d84f72b573bf1049c30237 [id=4c18f80dd28145b399b53670c182821c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.260826 23608 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet ec4028a8e66442bfabac9e587aed00bb (table default.Table_62b86da21eb84bc387b64725dabda733 [id=1e590d80e3e34e1aac8a0c7908a9d813]). Expected version 3 got 0
I20240502 08:16:36.261106 24039 tablet_service.cc:1467] Processing CreateTablet for tablet b844458ae73f413cbf2113027a5af359 (DEFAULT_TABLE table=default.Table_cdfdcb2e5f8b4b7bbd2d75d2ea00936e [id=fd93259a378b4ea582a6661a59104abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.261409 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 36f2d7b6f4e442ae95b2d1002d571b6a (DEFAULT_TABLE table=default.Table_66741640bb3e41808da39a2de1103d0d [id=a95a99a58cde44fe814b3edfcc643a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.261744 24034 tablet_service.cc:1467] Processing CreateTablet for tablet f36aadfa1d264f23bcafa91744aed36d (DEFAULT_TABLE table=default.Table_a6b10e6dd2db4b5eb3e3d639cac19f54 [id=e18870bff56747b192b4698f1ca3a8ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.261787 24283 mvcc.cc:205] Tried to move back new op lower bound from 7023156412717850624 to 7023156410800529408. Current Snapshot: MvccSnapshot[applied={T|T < 7023156412717850624}]
I20240502 08:16:36.261963 23608 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 175adf0ecb504e28a3d39f001f977839 (table default.Table_9530003aa5b841b683920da217d58576 [id=a6a5f4c65807479392c01ac5c955c959]). Expected version 1 got 0
I20240502 08:16:36.262331 24296 tablet.cc:1722] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:36.263031 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 8810b6f61db14fed8b12d36fed413dc8 (DEFAULT_TABLE table=default.Table_ce1b3e3032d84f72b573bf1049c30237 [id=4c18f80dd28145b399b53670c182821c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.263176 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd57b0357ca749c389bc61add20cd3e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.263371 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8810b6f61db14fed8b12d36fed413dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.263417 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0af5f4daf3149738647a08219efb245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.263558 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 7a19310948e248d1a85479c41c61bda2 (DEFAULT_TABLE table=default.Table_434596ffae0941b19dee488e942d764a [id=8521bfc13ef040a78cfb2cc4501b7f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.263922 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a19310948e248d1a85479c41c61bda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.264636 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c95dfda9d3b4e2ca9397b77ad71c011. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.264595 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 49b816ea1d474f2db94119441146ba78 (DEFAULT_TABLE table=default.Table_9ae8c17e33fc47d5b93166ab4d269138 [id=a0606a4053d04dd3814f09419a4a70c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.264963 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49b816ea1d474f2db94119441146ba78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.265156 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b844458ae73f413cbf2113027a5af359. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.267339 24173 tablet_bootstrap.cc:654] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.269695 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36f2d7b6f4e442ae95b2d1002d571b6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.273489 24174 tablet_bootstrap.cc:492] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:36.273667 24174 ts_tablet_manager.cc:1367] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.065s user 0.006s sys 0.000s
I20240502 08:16:36.273795 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f36aadfa1d264f23bcafa91744aed36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.274497 24174 raft_consensus.cc:348] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.274729 24174 raft_consensus.cc:374] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.274832 24174 raft_consensus.cc:729] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.275045 24174 consensus_queue.cc:260] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.275329 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 21832f0151ec4cf48f51f9bd2cb2b5ec
I20240502 08:16:36.275543 24174 ts_tablet_manager.cc:1398] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:36.276091 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c201704a319a4e078054899061f76615" } modify_external_catalogs: true
I20240502 08:16:36.276248 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c201704a319a4e078054899061f76615" } modify_external_catalogs: true
I20240502 08:16:36.277509 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet f51b26043ef44b30846498cd1497607c
I20240502 08:16:36.278100 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f52a4fa0e0c04f19a517f946b59a850f" } modify_external_catalogs: true
I20240502 08:16:36.278280 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f52a4fa0e0c04f19a517f946b59a850f" } modify_external_catalogs: true
I20240502 08:16:36.279520 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet d5bab1b7c48248a7973c27d0a865b34b
I20240502 08:16:36.279953 23602 catalog_manager.cc:6554] ReplaceTablet: tablet d5bab1b7c48248a7973c27d0a865b34b deleted and replaced by tablet 7c374f42f353497fa1d745164e7fbf61
I20240502 08:16:36.279942 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8810b6f61db14fed8b12d36fed413dc8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:36.280730 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7a19310948e248d1a85479c41c61bda2 consensus state has no leader
I20240502 08:16:36.282485 23610 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 768167b977c14b53a7537879e349d354 (table default.Table_0a242b17e3d2450eb3add7dbff397394 [id=ab1669d2a12f4187809987ed41fa0379]). Expected version 1 got 0
I20240502 08:16:36.285645 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 01675f637a5a4645949abfd87b3d09a2 (DEFAULT_TABLE table=default.Table_c5d06caccb424872b965a751c171a0a6 [id=4411ea4d183f4694b16b6255567ab62b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.286160 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01675f637a5a4645949abfd87b3d09a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.286432 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0af5f4daf3149738647a08219efb245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.287406 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 21832f0151ec4cf48f51f9bd2cb2b5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.287640 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet d5bab1b7c48248a7973c27d0a865b34b with delete_type TABLET_DATA_DELETED (replaced by tablet 7c374f42f353497fa1d745164e7fbf61) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.287684 23898 tablet_service.cc:1467] Processing CreateTablet for tablet b844458ae73f413cbf2113027a5af359 (DEFAULT_TABLE table=default.Table_cdfdcb2e5f8b4b7bbd2d75d2ea00936e [id=fd93259a378b4ea582a6661a59104abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.288080 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b844458ae73f413cbf2113027a5af359. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:36.288545 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 21832f0151ec4cf48f51f9bd2cb2b5ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21832f0151ec4cf48f51f9bd2cb2b5ec already in progress: creating tablet
W20240502 08:16:36.288810 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet d5bab1b7c48248a7973c27d0a865b34b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5bab1b7c48248a7973c27d0a865b34b already in progress: creating tablet
I20240502 08:16:36.267345 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 01675f637a5a4645949abfd87b3d09a2 (DEFAULT_TABLE table=default.Table_c5d06caccb424872b965a751c171a0a6 [id=4411ea4d183f4694b16b6255567ab62b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.290027 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 9700ea9dc58442e3ab1b7835a8c1e22c (DEFAULT_TABLE table=default.Table_7b904b0af0a8449581e717c1a56c01ce [id=2b9169f285754bfda0f4f8dc7f5cb653]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.290392 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9700ea9dc58442e3ab1b7835a8c1e22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.291735 23895 tablet_service.cc:1514] Processing DeleteTablet for tablet f51b26043ef44b30846498cd1497607c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:36.292299 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet f51b26043ef44b30846498cd1497607c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f51b26043ef44b30846498cd1497607c already in progress: creating tablet
I20240502 08:16:36.292373 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 8810b6f61db14fed8b12d36fed413dc8 (DEFAULT_TABLE table=default.Table_ce1b3e3032d84f72b573bf1049c30237 [id=4c18f80dd28145b399b53670c182821c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.292852 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8810b6f61db14fed8b12d36fed413dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.296665 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet d5bab1b7c48248a7973c27d0a865b34b with delete_type TABLET_DATA_DELETED (replaced by tablet 7c374f42f353497fa1d745164e7fbf61) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.301651 24034 tablet_service.cc:1514] Processing DeleteTablet for tablet f51b26043ef44b30846498cd1497607c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.312413 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 405ea360814e4593a289d753821277ff
I20240502 08:16:36.313385 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet a7c3800fb2784a1dba12d01c56c226e5
I20240502 08:16:36.313635 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 5c95dfda9d3b4e2ca9397b77ad71c011
I20240502 08:16:36.313938 23601 catalog_manager.cc:6554] ReplaceTablet: tablet a7c3800fb2784a1dba12d01c56c226e5 deleted and replaced by tablet f4b291bdb81a4efbac7f6691e5577bbe
I20240502 08:16:36.313408 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet fd57b0357ca749c389bc61add20cd3e7
I20240502 08:16:36.314844 23610 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_62b86da21eb84bc387b64725dabda733 [id=1e590d80e3e34e1aac8a0c7908a9d813] alter complete (version 3)
I20240502 08:16:36.315876 23904 tablet_service.cc:1514] Processing DeleteTablet for tablet fd57b0357ca749c389bc61add20cd3e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.315966 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c95dfda9d3b4e2ca9397b77ad71c011 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.316321 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 4ea2dee2d5914cf0bf4b37d16865e9b0
W20240502 08:16:36.317075 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet fd57b0357ca749c389bc61add20cd3e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd57b0357ca749c389bc61add20cd3e7 already in progress: creating tablet
W20240502 08:16:36.318039 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 5c95dfda9d3b4e2ca9397b77ad71c011 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c95dfda9d3b4e2ca9397b77ad71c011 already in progress: creating tablet
I20240502 08:16:36.315884 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet a7c3800fb2784a1dba12d01c56c226e5 with delete_type TABLET_DATA_DELETED (replaced by tablet f4b291bdb81a4efbac7f6691e5577bbe) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.318746 23607 catalog_manager.cc:6554] ReplaceTablet: tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 deleted and replaced by tablet b811b81ddf2b4dc9b5e5f80331bff294
I20240502 08:16:36.319411 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet fd57b0357ca749c389bc61add20cd3e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.319478 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 with delete_type TABLET_DATA_DELETED (replaced by tablet b811b81ddf2b4dc9b5e5f80331bff294) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:36.319518 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet a7c3800fb2784a1dba12d01c56c226e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7c3800fb2784a1dba12d01c56c226e5 already in progress: creating tablet
I20240502 08:16:36.319664 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c95dfda9d3b4e2ca9397b77ad71c011 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.319906 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 9700ea9dc58442e3ab1b7835a8c1e22c (DEFAULT_TABLE table=default.Table_7b904b0af0a8449581e717c1a56c01ce [id=2b9169f285754bfda0f4f8dc7f5cb653]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:36.320194 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 already in progress: creating tablet
I20240502 08:16:36.320035 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 with delete_type TABLET_DATA_DELETED (replaced by tablet b811b81ddf2b4dc9b5e5f80331bff294) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.320463 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet a7c3800fb2784a1dba12d01c56c226e5 with delete_type TABLET_DATA_DELETED (replaced by tablet f4b291bdb81a4efbac7f6691e5577bbe) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.320644 23895 tablet_service.cc:1514] Processing DeleteTablet for tablet 405ea360814e4593a289d753821277ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.320751 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01675f637a5a4645949abfd87b3d09a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.320883 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5d5022ea23c34e4f960c366d5ac96c39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.321453 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d5022ea23c34e4f960c366d5ac96c39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:36.321499 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 405ea360814e4593a289d753821277ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 405ea360814e4593a289d753821277ff already in progress: creating tablet
I20240502 08:16:36.324256 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet 21832f0151ec4cf48f51f9bd2cb2b5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.324458 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet 405ea360814e4593a289d753821277ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.325832 24174 tablet_bootstrap.cc:492] T 26c9913f1ca146fe84bc07ca1b4315c5 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:36.326014 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9700ea9dc58442e3ab1b7835a8c1e22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.326316 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 49b816ea1d474f2db94119441146ba78 (DEFAULT_TABLE table=default.Table_9ae8c17e33fc47d5b93166ab4d269138 [id=a0606a4053d04dd3814f09419a4a70c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.326783 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 9700ea9dc58442e3ab1b7835a8c1e22c (DEFAULT_TABLE table=default.Table_7b904b0af0a8449581e717c1a56c01ce [id=2b9169f285754bfda0f4f8dc7f5cb653]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.332109 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 21832f0151ec4cf48f51f9bd2cb2b5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.332253 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_60ce09e8ee9c47a4bb1d85e40f063338"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.333015 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60ce09e8ee9c47a4bb1d85e40f063338 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.333279 24175 tablet_bootstrap.cc:492] T ec4028a8e66442bfabac9e587aed00bb P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:36.335474 24175 ts_tablet_manager.cc:1367] T ec4028a8e66442bfabac9e587aed00bb P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.092s user 0.005s sys 0.000s
I20240502 08:16:36.336630 24175 raft_consensus.cc:348] T ec4028a8e66442bfabac9e587aed00bb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.338269 24181 ts_tablet_manager.cc:1888] T 71c0c2f3def643dfa99ff4efe65f6acf P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.340010 24181 log.cc:1196] T 71c0c2f3def643dfa99ff4efe65f6acf P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/71c0c2f3def643dfa99ff4efe65f6acf
I20240502 08:16:36.342015 24181 ts_tablet_manager.cc:1909] T 71c0c2f3def643dfa99ff4efe65f6acf P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:36.331324 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 5c95dfda9d3b4e2ca9397b77ad71c011 (DEFAULT_TABLE table=default.Table_c201704a319a4e078054899061f76615 [id=71234c2c912b4a79b7b08d87cab6fad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.342298 24173 tablet_bootstrap.cc:492] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:36.345294 24173 ts_tablet_manager.cc:1367] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.131s user 0.011s sys 0.014s
I20240502 08:16:36.345628 24175 raft_consensus.cc:374] T ec4028a8e66442bfabac9e587aed00bb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.346151 24173 raft_consensus.cc:348] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.346362 24173 raft_consensus.cc:374] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.346458 24173 raft_consensus.cc:729] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.346567 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1f1d41952a4746aba7b73e42398a343a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:36.346726 24173 consensus_queue.cc:260] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:36.347164 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f1d41952a4746aba7b73e42398a343a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.347224 24173 ts_tablet_manager.cc:1398] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:36.347573 24173 tablet_bootstrap.cc:492] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:36.349079 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet f51b26043ef44b30846498cd1497607c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.349599 24173 tablet_bootstrap.cc:654] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.349681 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet a7c3800fb2784a1dba12d01c56c226e5 with delete_type TABLET_DATA_DELETED (replaced by tablet f4b291bdb81a4efbac7f6691e5577bbe) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.349886 23754 tablet_service.cc:1514] Processing DeleteTablet for tablet 405ea360814e4593a289d753821277ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.346366 24175 raft_consensus.cc:729] T ec4028a8e66442bfabac9e587aed00bb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.351518 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_947d6d5750d84f2fa644ee5f8be29746"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.352145 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_947d6d5750d84f2fa644ee5f8be29746 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.352461 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49b816ea1d474f2db94119441146ba78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.357872 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c95dfda9d3b4e2ca9397b77ad71c011 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.358155 24174 tablet_bootstrap.cc:654] T 26c9913f1ca146fe84bc07ca1b4315c5 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.356876 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet d5bab1b7c48248a7973c27d0a865b34b with delete_type TABLET_DATA_DELETED (replaced by tablet 7c374f42f353497fa1d745164e7fbf61) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.361514 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet fd57b0357ca749c389bc61add20cd3e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.363418 24059 raft_consensus.cc:1261] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:36.364043 23925 raft_consensus.cc:1261] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:36.364375 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c95dfda9d3b4e2ca9397b77ad71c011. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.365372 24243 consensus_queue.cc:1035] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:16:36.365809 24243 consensus_queue.cc:1035] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:36.378055 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_02f448a8a9844def8ecda8716fdc352e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:36.378517 24174 tablet_bootstrap.cc:492] T 26c9913f1ca146fe84bc07ca1b4315c5 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
W20240502 08:16:36.378588 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02f448a8a9844def8ecda8716fdc352e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.380554 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet 5738cd77bbe147b49215ba9639d7f547 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:36.380822 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_73bf7705ba9847519b86284f595f3f10" } new_table_name: "default.Table_6c9edf34b95e4ad29551269a83447b4a" modify_external_catalogs: true
I20240502 08:16:36.381312 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_bd9e5acccefd4757b59b162649c7747c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:36.381772 24243 raft_consensus.cc:482] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:36.381908 24243 raft_consensus.cc:504] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:36.382136 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd9e5acccefd4757b59b162649c7747c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.383491 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05cbbf9614094afaa6f7cee0af597242" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:36.383668 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05cbbf9614094afaa6f7cee0af597242" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:36.384268 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet 0be4f8d5cc09483d8d1aa1e37679b544 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:36.384328 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 with delete_type TABLET_DATA_DELETED (replaced by tablet b811b81ddf2b4dc9b5e5f80331bff294) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.384593 24243 leader_election.cc:288] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:36.384836 23595 consensus_peers.cc:485] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 -> Peer bcc8a323e6b9429eaaf4e0c1698ad077 (127.22.225.126:43265): Couldn't send request to peer bcc8a323e6b9429eaaf4e0c1698ad077. Status: Network error: Client connection negotiation failed: client connection to 127.22.225.126:43265: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:16:36.385699 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9700ea9dc58442e3ab1b7835a8c1e22c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:36.386377 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 21832f0151ec4cf48f51f9bd2cb2b5ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21832f0151ec4cf48f51f9bd2cb2b5ec already in progress: creating tablet
W20240502 08:16:36.386823 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet f51b26043ef44b30846498cd1497607c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f51b26043ef44b30846498cd1497607c already in progress: creating tablet
W20240502 08:16:36.387065 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet a7c3800fb2784a1dba12d01c56c226e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7c3800fb2784a1dba12d01c56c226e5 already in progress: creating tablet
W20240502 08:16:36.387321 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 405ea360814e4593a289d753821277ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 405ea360814e4593a289d753821277ff already in progress: creating tablet
W20240502 08:16:36.387581 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 5c95dfda9d3b4e2ca9397b77ad71c011 because the tablet was not found. No further retry: Not found: Tablet not found: 5c95dfda9d3b4e2ca9397b77ad71c011
W20240502 08:16:36.387800 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet d5bab1b7c48248a7973c27d0a865b34b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5bab1b7c48248a7973c27d0a865b34b already in progress: creating tablet
W20240502 08:16:36.388021 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet fd57b0357ca749c389bc61add20cd3e7 because the tablet was not found. No further retry: Not found: Tablet not found: fd57b0357ca749c389bc61add20cd3e7
W20240502 08:16:36.388470 23723 leader_election.cc:341] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:36.388548 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 already in progress: creating tablet
W20240502 08:16:36.388921 23723 leader_election.cc:341] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:36.389497 24174 ts_tablet_manager.cc:1367] T 26c9913f1ca146fe84bc07ca1b4315c5 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.103s user 0.007s sys 0.005s
I20240502 08:16:36.389459 23723 leader_election.cc:302] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:36.390674 24173 tablet_bootstrap.cc:492] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:36.390849 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f2ff50fa357448494434d1487ce83bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.392277 24174 raft_consensus.cc:348] T 26c9913f1ca146fe84bc07ca1b4315c5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.395290 24243 raft_consensus.cc:2726] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:36.395961 24174 raft_consensus.cc:374] T 26c9913f1ca146fe84bc07ca1b4315c5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.396093 24174 raft_consensus.cc:729] T 26c9913f1ca146fe84bc07ca1b4315c5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.390893 24173 ts_tablet_manager.cc:1367] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.002s
I20240502 08:16:36.398578 24173 raft_consensus.cc:348] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.398801 24173 raft_consensus.cc:374] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.398883 24173 raft_consensus.cc:729] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.399118 24173 consensus_queue.cc:260] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.399667 24173 ts_tablet_manager.cc:1398] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:36.399983 24173 tablet_bootstrap.cc:492] T 26c9913f1ca146fe84bc07ca1b4315c5 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:36.401540 24173 tablet_bootstrap.cc:654] T 26c9913f1ca146fe84bc07ca1b4315c5 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.397545 24174 consensus_queue.cc:260] T 26c9913f1ca146fe84bc07ca1b4315c5 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.404057 24174 ts_tablet_manager.cc:1398] T 26c9913f1ca146fe84bc07ca1b4315c5 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.014s user 0.002s sys 0.001s
I20240502 08:16:36.405438 24174 tablet_bootstrap.cc:492] T 0f2ff50fa357448494434d1487ce83bc P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:36.408066 24174 tablet_bootstrap.cc:654] T 0f2ff50fa357448494434d1487ce83bc P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.410411 24175 consensus_queue.cc:260] T ec4028a8e66442bfabac9e587aed00bb P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.411275 24175 ts_tablet_manager.cc:1398] T ec4028a8e66442bfabac9e587aed00bb P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.075s user 0.002s sys 0.000s
I20240502 08:16:36.411746 24175 tablet_bootstrap.cc:492] T 4ce3e95d5ea7435593b7cfe5a670a76a P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:36.413583 24175 tablet_bootstrap.cc:654] T 4ce3e95d5ea7435593b7cfe5a670a76a P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.414970 23607 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 050c16739efb4037a4adc78a2ffd3cba (table default.Table_2b5aaf0ddb384715934e7eb81369ecac [id=ba62a673079d424992daa0f6b0c5d6b9]). Expected version 2 got 0
I20240502 08:16:36.416242 23604 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 050c16739efb4037a4adc78a2ffd3cba (table default.Table_2b5aaf0ddb384715934e7eb81369ecac [id=ba62a673079d424992daa0f6b0c5d6b9]). Expected version 2 got 0
I20240502 08:16:36.417539 24293 tablet.cc:1722] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:36.419714 24173 tablet_bootstrap.cc:492] T 26c9913f1ca146fe84bc07ca1b4315c5 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:36.419878 24173 ts_tablet_manager.cc:1367] T 26c9913f1ca146fe84bc07ca1b4315c5 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.001s
I20240502 08:16:36.420060 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 71c0c2f3def643dfa99ff4efe65f6acf (table default.Table_68bf91a14ffa42cdbb10a1d7ca56cb08 [id=59d945a73daa4918ba17063eae0b9b93]) successfully deleted
W20240502 08:16:36.420292 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b11f504b93574e42a9bf30fde6d08207 consensus state has no leader
I20240502 08:16:36.420563 24173 raft_consensus.cc:348] T 26c9913f1ca146fe84bc07ca1b4315c5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.420750 24173 raft_consensus.cc:374] T 26c9913f1ca146fe84bc07ca1b4315c5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.420814 24173 raft_consensus.cc:729] T 26c9913f1ca146fe84bc07ca1b4315c5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.421033 24173 consensus_queue.cc:260] T 26c9913f1ca146fe84bc07ca1b4315c5 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.421576 24173 ts_tablet_manager.cc:1398] T 26c9913f1ca146fe84bc07ca1b4315c5 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:36.422205 24173 tablet_bootstrap.cc:492] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:36.422513 23606 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet ec4028a8e66442bfabac9e587aed00bb (table default.Table_62b86da21eb84bc387b64725dabda733 [id=1e590d80e3e34e1aac8a0c7908a9d813]). Expected version 3 got 0
I20240502 08:16:36.424008 24173 tablet_bootstrap.cc:654] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.425693 24181 tablet_replica.cc:330] T c1c2a988a3524c8fb7410e8508bd1abc P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:36.425947 24181 raft_consensus.cc:2229] T c1c2a988a3524c8fb7410e8508bd1abc P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.426477 24181 raft_consensus.cc:2258] T c1c2a988a3524c8fb7410e8508bd1abc P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.427374 24181 ts_tablet_manager.cc:1875] T c1c2a988a3524c8fb7410e8508bd1abc P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:36.433740 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c9913f1ca146fe84bc07ca1b4315c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:36.438305 24326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:36.446142 24338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:36.448184 24241 server_base.cc:1034] running on GCE node
W20240502 08:16:36.448760 24320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:36.449723 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet e17eec8f2c1a4190945b434794635b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.450237 24241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:36.454995 24296 tablet.cc:1722] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:36.460871 24175 tablet_bootstrap.cc:492] T 4ce3e95d5ea7435593b7cfe5a670a76a P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:36.461397 23576 master_runner.cc:186] Error getting master registration for 127.22.225.126:43265: Network error: Client connection negotiation failed: client connection to 127.22.225.126:43265: connect: Connection refused (error 111),
I20240502 08:16:36.461617 23576 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240502 08:16:36.461861 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_14c4c753725c4160b1e9f57eda5f17b1" } new_table_name: "default.Table_9663c413d5be4a049eb7a02ececc1276" modify_external_catalogs: true
I20240502 08:16:36.463239 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_4688a59d76ee4b5a931536f5deddfd4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:36.463303 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0ee9c5ee2b7242c3ad3270e17a4b407a" } new_table_name: "default.Table_f48b4cdee51644e0b3c904d772f01404" modify_external_catalogs: true
W20240502 08:16:36.464681 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4688a59d76ee4b5a931536f5deddfd4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.464994 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_bc1e9d346aa846f8abca7504185f7cdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.466032 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc1e9d346aa846f8abca7504185f7cdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.469466 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_adde4be2323646faa59e19d7e9d189e5" } new_table_name: "default.Table_006d60eaae72439f829df411c32c1514" modify_external_catalogs: true
I20240502 08:16:36.478780 24175 ts_tablet_manager.cc:1367] T 4ce3e95d5ea7435593b7cfe5a670a76a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.067s user 0.000s sys 0.004s
W20240502 08:16:36.479684 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2bfb532c9d4c41afb7acaa4831841306 has no consensus state
W20240502 08:16:36.479907 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 has no consensus state
I20240502 08:16:36.481052 24175 raft_consensus.cc:348] T 4ce3e95d5ea7435593b7cfe5a670a76a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:36.485380 24174 tablet_bootstrap.cc:492] T 0f2ff50fa357448494434d1487ce83bc P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:36.485581 24174 ts_tablet_manager.cc:1367] T 0f2ff50fa357448494434d1487ce83bc P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.081s user 0.022s sys 0.011s
I20240502 08:16:36.486312 24174 raft_consensus.cc:348] T 0f2ff50fa357448494434d1487ce83bc P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.486567 24174 raft_consensus.cc:374] T 0f2ff50fa357448494434d1487ce83bc P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.486668 24174 raft_consensus.cc:729] T 0f2ff50fa357448494434d1487ce83bc P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.486939 24174 consensus_queue.cc:260] T 0f2ff50fa357448494434d1487ce83bc P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.487594 24174 ts_tablet_manager.cc:1398] T 0f2ff50fa357448494434d1487ce83bc P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:36.487727 24241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:36.487937 24174 tablet_bootstrap.cc:492] T e17eec8f2c1a4190945b434794635b52 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:36.489547 24174 tablet_bootstrap.cc:654] T e17eec8f2c1a4190945b434794635b52 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.490434 24175 raft_consensus.cc:374] T 4ce3e95d5ea7435593b7cfe5a670a76a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.492416 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c11fafb2f1854834893ad5c076f1ca11" } new_table_name: "default.Table_aa989e0f3fca40aeb4beda0a72a1a79e" modify_external_catalogs: true
I20240502 08:16:36.492581 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1f1d41952a4746aba7b73e42398a343a" } modify_external_catalogs: true
I20240502 08:16:36.492730 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1f1d41952a4746aba7b73e42398a343a" } modify_external_catalogs: true
I20240502 08:16:36.495129 23604 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_2b5aaf0ddb384715934e7eb81369ecac [id=ba62a673079d424992daa0f6b0c5d6b9] alter complete (version 2)
I20240502 08:16:36.497239 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6f82c95f8aa44cadabb6c55876c6f241"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.498489 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f82c95f8aa44cadabb6c55876c6f241 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.500983 24174 tablet_bootstrap.cc:492] T e17eec8f2c1a4190945b434794635b52 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:36.501169 24174 ts_tablet_manager.cc:1367] T e17eec8f2c1a4190945b434794635b52 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.002s
I20240502 08:16:36.501199 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0528f5d019fe489db8bed1f71adac3cb" } modify_external_catalogs: true
I20240502 08:16:36.501375 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0528f5d019fe489db8bed1f71adac3cb" } modify_external_catalogs: true
I20240502 08:16:36.501998 24174 raft_consensus.cc:348] T e17eec8f2c1a4190945b434794635b52 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.502224 24174 raft_consensus.cc:374] T e17eec8f2c1a4190945b434794635b52 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.502285 24174 raft_consensus.cc:729] T e17eec8f2c1a4190945b434794635b52 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
W20240502 08:16:36.502410 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4302fec5c51e4c6c841710bab50b2997 consensus state has no leader
I20240502 08:16:36.502532 24174 consensus_queue.cc:260] T e17eec8f2c1a4190945b434794635b52 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.503100 24174 ts_tablet_manager.cc:1398] T e17eec8f2c1a4190945b434794635b52 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240502 08:16:36.504480 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7a19310948e248d1a85479c41c61bda2 consensus state has no leader
I20240502 08:16:36.505002 23604 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 050c16739efb4037a4adc78a2ffd3cba (table default.Table_2b5aaf0ddb384715934e7eb81369ecac [id=ba62a673079d424992daa0f6b0c5d6b9]). Expected version 2 got 0
I20240502 08:16:36.506805 23608 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 768167b977c14b53a7537879e349d354 (table default.Table_0a242b17e3d2450eb3add7dbff397394 [id=ab1669d2a12f4187809987ed41fa0379]). Expected version 1 got 0
I20240502 08:16:36.508407 24174 tablet_bootstrap.cc:492] T 32150c7e130f4069a2a3a02e625bfb93 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:36.508364 24175 raft_consensus.cc:729] T 4ce3e95d5ea7435593b7cfe5a670a76a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.509032 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 5453cc0b93ac4cbb90a0588bb2100d60 (DEFAULT_TABLE table=default.Table_2df84a3eb090456e8da6106c0a17131a [id=8c859fafcd4b49f0b4e9fc9d8961c9ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.509510 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5453cc0b93ac4cbb90a0588bb2100d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.510577 23605 master_service.cc:946] ReplaceTablet: received request to replace tablet d8f7b5f018d84680ac5533bf99102d31 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:36.510772 24174 tablet_bootstrap.cc:654] T 32150c7e130f4069a2a3a02e625bfb93 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.514329 24175 consensus_queue.cc:260] T 4ce3e95d5ea7435593b7cfe5a670a76a P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:36.491086 24341 tablet.cc:1722] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:36.517989 24174 tablet_bootstrap.cc:492] T 32150c7e130f4069a2a3a02e625bfb93 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:36.518162 24174 ts_tablet_manager.cc:1367] T 32150c7e130f4069a2a3a02e625bfb93 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240502 08:16:36.518432 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b461b560f3094407a1766253cfc2ece7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.519012 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b461b560f3094407a1766253cfc2ece7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.518959 24174 raft_consensus.cc:348] T 32150c7e130f4069a2a3a02e625bfb93 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.519163 24174 raft_consensus.cc:374] T 32150c7e130f4069a2a3a02e625bfb93 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.519256 24174 raft_consensus.cc:729] T 32150c7e130f4069a2a3a02e625bfb93 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.519474 24174 consensus_queue.cc:260] T 32150c7e130f4069a2a3a02e625bfb93 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.520020 24174 ts_tablet_manager.cc:1398] T 32150c7e130f4069a2a3a02e625bfb93 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:36.520205 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_947d6d5750d84f2fa644ee5f8be29746" } modify_external_catalogs: true
I20240502 08:16:36.520323 24174 tablet_bootstrap.cc:492] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:36.520401 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_947d6d5750d84f2fa644ee5f8be29746" } modify_external_catalogs: true
I20240502 08:16:36.521783 24174 tablet_bootstrap.cc:654] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.522451 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c45e7b653ae641b293b87c150748c864" } modify_external_catalogs: true
I20240502 08:16:36.522629 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c45e7b653ae641b293b87c150748c864" } modify_external_catalogs: true
I20240502 08:16:36.523825 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9530003aa5b841b683920da217d58576" } modify_external_catalogs: true
I20240502 08:16:36.524004 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9530003aa5b841b683920da217d58576" } modify_external_catalogs: true
I20240502 08:16:36.526909 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet 3052510d77f344bf87decb9c2e5ec0bd with delete_type TABLET_DATA_DELETED (replaced by tablet 23ce83c8001d4b2988b88c0ed9bf948c) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.528690 24174 tablet_bootstrap.cc:492] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:36.528859 24174 ts_tablet_manager.cc:1367] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 08:16:36.528996 24180 tablet_replica.cc:330] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:36.529217 24180 raft_consensus.cc:2229] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.529395 24180 raft_consensus.cc:2258] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.529744 24241 hybrid_clock.cc:648] HybridClock initialized: now 1714637796529535 us; error 169 us; skew 500 ppm
I20240502 08:16:36.530238 24241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:36.531428 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet e17eec8f2c1a4190945b434794635b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.532001 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c9913f1ca146fe84bc07ca1b4315c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.532125 24180 ts_tablet_manager.cc:1875] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:36.532135 23608 master_service.cc:946] ReplaceTablet: received request to replace tablet f36ceb0222ef473a828ca795f393174d from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:36.532547 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f2ff50fa357448494434d1487ce83bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.535701 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 05cbbf9614094afaa6f7cee0af597242 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.538277 24180 ts_tablet_manager.cc:1888] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.538462 24180 log.cc:1196] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/3052510d77f344bf87decb9c2e5ec0bd
I20240502 08:16:36.538463 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 0be4f8d5cc09483d8d1aa1e37679b544
I20240502 08:16:36.538918 24180 ts_tablet_manager.cc:1909] T 3052510d77f344bf87decb9c2e5ec0bd P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:36.539000 23602 catalog_manager.cc:6554] ReplaceTablet: tablet 0be4f8d5cc09483d8d1aa1e37679b544 deleted and replaced by tablet e686db13bb7541c69ae88f0fb7e27061
I20240502 08:16:36.540050 24180 tablet_replica.cc:330] T e17eec8f2c1a4190945b434794635b52 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:36.540030 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 0be4f8d5cc09483d8d1aa1e37679b544 with delete_type TABLET_DATA_DELETED (replaced by tablet e686db13bb7541c69ae88f0fb7e27061) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.540247 24180 raft_consensus.cc:2229] T e17eec8f2c1a4190945b434794635b52 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.540412 24180 raft_consensus.cc:2258] T e17eec8f2c1a4190945b434794635b52 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.541148 24180 ts_tablet_manager.cc:1875] T e17eec8f2c1a4190945b434794635b52 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:36.542119 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet 26c9913f1ca146fe84bc07ca1b4315c5 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:36.542233 24175 ts_tablet_manager.cc:1398] T 4ce3e95d5ea7435593b7cfe5a670a76a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.062s user 0.001s sys 0.001s
I20240502 08:16:36.543053 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce3e95d5ea7435593b7cfe5a670a76a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.543344 24179 tablet_replica.cc:330] T 4ce3e95d5ea7435593b7cfe5a670a76a P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:36.543532 24179 raft_consensus.cc:2229] T 4ce3e95d5ea7435593b7cfe5a670a76a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.543668 24179 raft_consensus.cc:2258] T 4ce3e95d5ea7435593b7cfe5a670a76a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.544163 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 485c772245fe48a39fc6ceb6377ac0b6
I20240502 08:16:36.545167 24179 ts_tablet_manager.cc:1875] T 4ce3e95d5ea7435593b7cfe5a670a76a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:36.545374 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 485c772245fe48a39fc6ceb6377ac0b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.545634 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 485c772245fe48a39fc6ceb6377ac0b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.547044 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 3052510d77f344bf87decb9c2e5ec0bd (table default.Table_623eac7101f94d959fb83b6f51d8fe0a [id=54454efbe9394170822f237dc8e6115e]) successfully deleted
I20240502 08:16:36.547351 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 0be4f8d5cc09483d8d1aa1e37679b544 with delete_type TABLET_DATA_DELETED (replaced by tablet e686db13bb7541c69ae88f0fb7e27061) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.547633 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet 0be4f8d5cc09483d8d1aa1e37679b544 with delete_type TABLET_DATA_DELETED (replaced by tablet e686db13bb7541c69ae88f0fb7e27061) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.548840 24174 raft_consensus.cc:348] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.549074 24174 raft_consensus.cc:374] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.549135 24174 raft_consensus.cc:729] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.549626 24243 raft_consensus.cc:482] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:36.550073 24174 consensus_queue.cc:260] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.550139 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 5453cc0b93ac4cbb90a0588bb2100d60 (DEFAULT_TABLE table=default.Table_2df84a3eb090456e8da6106c0a17131a [id=8c859fafcd4b49f0b4e9fc9d8961c9ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.550563 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5453cc0b93ac4cbb90a0588bb2100d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.550607 24174 ts_tablet_manager.cc:1398] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240502 08:16:36.550745 23901 tablet_service.cc:1467] Processing CreateTablet for tablet c08228fcd1304528b9446003851f6c70 (DEFAULT_TABLE table=default.Table_bbf9d7694fe942de8aa8bd4ed61336a5 [id=f1d6eb661b0143ee82e797cde61391c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.550916 24174 tablet_bootstrap.cc:492] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:36.551105 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c08228fcd1304528b9446003851f6c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.552090 24179 ts_tablet_manager.cc:1888] T 4ce3e95d5ea7435593b7cfe5a670a76a P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.552222 24179 log.cc:1196] T 4ce3e95d5ea7435593b7cfe5a670a76a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/4ce3e95d5ea7435593b7cfe5a670a76a
I20240502 08:16:36.552341 24174 tablet_bootstrap.cc:654] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.552577 24179 ts_tablet_manager.cc:1909] T 4ce3e95d5ea7435593b7cfe5a670a76a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:36.554800 24180 ts_tablet_manager.cc:1888] T e17eec8f2c1a4190945b434794635b52 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.554976 24180 log.cc:1196] T e17eec8f2c1a4190945b434794635b52 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/e17eec8f2c1a4190945b434794635b52
I20240502 08:16:36.555297 24180 ts_tablet_manager.cc:1909] T e17eec8f2c1a4190945b434794635b52 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:36.553078 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 4ce3e95d5ea7435593b7cfe5a670a76a (table default.Table_6d48e2ca21a2409fae7c972ec53adbf5 [id=d06bffc628cd4e07a8651559cbe38e09]) successfully deleted
W20240502 08:16:36.555696 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 485c772245fe48a39fc6ceb6377ac0b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 485c772245fe48a39fc6ceb6377ac0b6 already in progress: creating tablet
W20240502 08:16:36.555934 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 0be4f8d5cc09483d8d1aa1e37679b544 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0be4f8d5cc09483d8d1aa1e37679b544 already in progress: creating tablet
I20240502 08:16:36.556543 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce3e95d5ea7435593b7cfe5a670a76a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.557348 24030 tablet_service.cc:1467] Processing CreateTablet for tablet c5d28098dcbc4761a34ee65769d6fd7c (DEFAULT_TABLE table=default.Table_2f863e7f34fb4bb7a9770adbca87da84 [id=dd9c1fd15bf446fd87bd3647cb7f041d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.557641 24180 tablet_replica.cc:330] T 26c9913f1ca146fe84bc07ca1b4315c5 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:36.557711 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d28098dcbc4761a34ee65769d6fd7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.557664 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 485c772245fe48a39fc6ceb6377ac0b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.557821 24180 raft_consensus.cc:2229] T 26c9913f1ca146fe84bc07ca1b4315c5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.557981 24180 raft_consensus.cc:2258] T 26c9913f1ca146fe84bc07ca1b4315c5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.558831 23896 tablet_service.cc:1467] Processing CreateTablet for tablet c5d28098dcbc4761a34ee65769d6fd7c (DEFAULT_TABLE table=default.Table_2f863e7f34fb4bb7a9770adbca87da84 [id=dd9c1fd15bf446fd87bd3647cb7f041d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.559187 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d28098dcbc4761a34ee65769d6fd7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.559604 24180 ts_tablet_manager.cc:1875] T 26c9913f1ca146fe84bc07ca1b4315c5 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:36.560909 23597 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 4302fec5c51e4c6c841710bab50b2997 (table default.Table_006d60eaae72439f829df411c32c1514 [id=e3f148f833bc4d99a3ab2ec8d5258507], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4302fec5c51e4c6c841710bab50b2997 consensus state has no leader
I20240502 08:16:36.549762 24243 raft_consensus.cc:504] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:36.561955 23597 catalog_manager.cc:4797] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 4ce3e95d5ea7435593b7cfe5a670a76a because the tablet was not found. No further retry: Not found: Tablet not found: 4ce3e95d5ea7435593b7cfe5a670a76a
I20240502 08:16:36.562212 24243 leader_election.cc:288] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:36.563951 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet e17eec8f2c1a4190945b434794635b52 (table default.Table_b01c340249ea483683e385a7fa964a51 [id=ee0e1cf16cb34aa98699970438cc36ae]) successfully deleted
I20240502 08:16:36.564240 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 5453cc0b93ac4cbb90a0588bb2100d60 (DEFAULT_TABLE table=default.Table_2df84a3eb090456e8da6106c0a17131a [id=8c859fafcd4b49f0b4e9fc9d8961c9ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.564425 23761 tablet_service.cc:1467] Processing CreateTablet for tablet c5d28098dcbc4761a34ee65769d6fd7c (DEFAULT_TABLE table=default.Table_2f863e7f34fb4bb7a9770adbca87da84 [id=dd9c1fd15bf446fd87bd3647cb7f041d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.565045 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768167b977c14b53a7537879e349d354" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:36.565052 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768167b977c14b53a7537879e349d354" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:36.565264 24059 raft_consensus.cc:2454] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:36.565335 23758 tablet_service.cc:1467] Processing CreateTablet for tablet c08228fcd1304528b9446003851f6c70 (DEFAULT_TABLE table=default.Table_bbf9d7694fe942de8aa8bd4ed61336a5 [id=f1d6eb661b0143ee82e797cde61391c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.565626 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet 32150c7e130f4069a2a3a02e625bfb93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.565861 24181 ts_tablet_manager.cc:1888] T c1c2a988a3524c8fb7410e8508bd1abc P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.566051 24181 log.cc:1196] T c1c2a988a3524c8fb7410e8508bd1abc P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/c1c2a988a3524c8fb7410e8508bd1abc
I20240502 08:16:36.566421 24181 ts_tablet_manager.cc:1909] T c1c2a988a3524c8fb7410e8508bd1abc P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:36.567773 24181 tablet_replica.cc:330] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:36.567848 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet c1c2a988a3524c8fb7410e8508bd1abc (table default.Table_b96eb662e06846039807538794442017 [id=31ddf074089243a89bab5928860c625e]) successfully deleted
I20240502 08:16:36.565765 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5453cc0b93ac4cbb90a0588bb2100d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.569952 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d28098dcbc4761a34ee65769d6fd7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.572075 24175 tablet_bootstrap.cc:492] T 3052510d77f344bf87decb9c2e5ec0bd P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:36.572971 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c08228fcd1304528b9446003851f6c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.573967 24175 tablet_bootstrap.cc:654] T 3052510d77f344bf87decb9c2e5ec0bd P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.575085 23759 tablet_service.cc:1467] Processing CreateTablet for tablet b1fac7df16fc41d7a3297055a7258270 (DEFAULT_TABLE table=default.Table_bcd1014009b046a7ba72d1d757e69e26 [id=e6823667e67a470e870f4ed70fc58a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.575456 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fac7df16fc41d7a3297055a7258270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.580403 23892 tablet_service.cc:1467] Processing CreateTablet for tablet d1be2c29d3154d9189af3266191f0b3e (DEFAULT_TABLE table=default.Table_2468b65097c244298ba963e7aa155353 [id=6103f133bcdd4f768b92b37245bf5356]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.580830 23901 tablet_service.cc:1467] Processing CreateTablet for tablet b1fac7df16fc41d7a3297055a7258270 (DEFAULT_TABLE table=default.Table_bcd1014009b046a7ba72d1d757e69e26 [id=e6823667e67a470e870f4ed70fc58a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.580935 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1be2c29d3154d9189af3266191f0b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.581951 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 (DEFAULT_TABLE table=default.Table_9663c413d5be4a049eb7a02ececc1276 [id=ad4f0c8adc524a019beaf37e153b33a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.582365 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c03a4353c4e46e3bcdf3a36aac04bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.582798 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fac7df16fc41d7a3297055a7258270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.566704 23723 leader_election.cc:302] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:36.567924 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet 405ea360814e4593a289d753821277ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.589488 24343 raft_consensus.cc:482] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:36.579274 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 171406810543435e832a47ad5b3c80a4 (DEFAULT_TABLE table=default.Table_a286e67e441e43a1b4738cd219dbea98 [id=b56e3c0686244ad6b49afdd589c5ed34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.590243 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171406810543435e832a47ad5b3c80a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.590225 24329 raft_consensus.cc:2781] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:36.590322 24329 raft_consensus.cc:482] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:36.590684 24329 raft_consensus.cc:3037] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:36.591964 24241 webserver.cc:460] Webserver started at http://127.22.225.126:34035/ using document root <none> and password file <none>
I20240502 08:16:36.589690 24343 raft_consensus.cc:504] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.579640 24036 tablet_service.cc:1467] Processing CreateTablet for tablet d1be2c29d3154d9189af3266191f0b3e (DEFAULT_TABLE table=default.Table_2468b65097c244298ba963e7aa155353 [id=6103f133bcdd4f768b92b37245bf5356]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.593289 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1be2c29d3154d9189af3266191f0b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.593494 23757 tablet_service.cc:1467] Processing CreateTablet for tablet d1be2c29d3154d9189af3266191f0b3e (DEFAULT_TABLE table=default.Table_2468b65097c244298ba963e7aa155353 [id=6103f133bcdd4f768b92b37245bf5356]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.593528 24241 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:36.593858 24241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:36.593436 24343 leader_election.cc:288] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:36.594441 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1be2c29d3154d9189af3266191f0b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.579996 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 848a0865eb2b4828a2594a17d96869d0 (DEFAULT_TABLE table=default.Table_b9d00c29b8c2498895efd283a90c12e5 [id=7ffacb15a8a249258fb42c4312f87d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.596391 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848a0865eb2b4828a2594a17d96869d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.600536 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 848a0865eb2b4828a2594a17d96869d0 (DEFAULT_TABLE table=default.Table_b9d00c29b8c2498895efd283a90c12e5 [id=7ffacb15a8a249258fb42c4312f87d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.600795 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 0dff0d73a933427fb1f2686bac33eba1 (DEFAULT_TABLE table=default.Table_59b22bd34c884d0ebe95d6737550e50d [id=9076329a97254502a8c10b167c7f5b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.600971 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848a0865eb2b4828a2594a17d96869d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.580324 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 0dff0d73a933427fb1f2686bac33eba1 (DEFAULT_TABLE table=default.Table_59b22bd34c884d0ebe95d6737550e50d [id=9076329a97254502a8c10b167c7f5b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.601471 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dff0d73a933427fb1f2686bac33eba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.579648 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 (DEFAULT_TABLE table=default.Table_9663c413d5be4a049eb7a02ececc1276 [id=ad4f0c8adc524a019beaf37e153b33a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.605146 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c03a4353c4e46e3bcdf3a36aac04bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.609692 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 0dff0d73a933427fb1f2686bac33eba1 (DEFAULT_TABLE table=default.Table_59b22bd34c884d0ebe95d6737550e50d [id=9076329a97254502a8c10b167c7f5b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.580628 24037 tablet_service.cc:1467] Processing CreateTablet for tablet b1fac7df16fc41d7a3297055a7258270 (DEFAULT_TABLE table=default.Table_bcd1014009b046a7ba72d1d757e69e26 [id=e6823667e67a470e870f4ed70fc58a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.580806 24030 tablet_service.cc:1467] Processing CreateTablet for tablet c08228fcd1304528b9446003851f6c70 (DEFAULT_TABLE table=default.Table_bbf9d7694fe942de8aa8bd4ed61336a5 [id=f1d6eb661b0143ee82e797cde61391c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.610625 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fac7df16fc41d7a3297055a7258270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.587527 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 7c374f42f353497fa1d745164e7fbf61 (DEFAULT_TABLE table=default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.587608 24173 tablet_bootstrap.cc:492] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:36.613848 24173 ts_tablet_manager.cc:1367] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.192s user 0.011s sys 0.044s
I20240502 08:16:36.613903 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c374f42f353497fa1d745164e7fbf61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.615077 24173 raft_consensus.cc:348] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.615401 24173 raft_consensus.cc:374] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.615499 24173 raft_consensus.cc:729] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.615819 24173 consensus_queue.cc:260] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.616223 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 10adabdd31f848548ef6f2408ccc2050
I20240502 08:16:36.616529 24173 ts_tablet_manager.cc:1398] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:16:36.617018 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 175adf0ecb504e28a3d39f001f977839
I20240502 08:16:36.618391 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet d8f7b5f018d84680ac5533bf99102d31
I20240502 08:16:36.618564 24173 tablet_bootstrap.cc:492] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:36.618829 23605 catalog_manager.cc:6554] ReplaceTablet: tablet d8f7b5f018d84680ac5533bf99102d31 deleted and replaced by tablet c17e1be6ef294769a01981f4d361be21
I20240502 08:16:36.619195 23610 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet b6b7eb31f31d449b8107fb0355d378f2
I20240502 08:16:36.619372 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dff0d73a933427fb1f2686bac33eba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.619731 23604 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet b49e519bd07b4e598f3a8681a4f55d1c
I20240502 08:16:36.620098 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet f36ceb0222ef473a828ca795f393174d
I20240502 08:16:36.620517 23608 catalog_manager.cc:6554] ReplaceTablet: tablet f36ceb0222ef473a828ca795f393174d deleted and replaced by tablet df9402cf8abc4b4d893d42f9a3a99016
I20240502 08:16:36.620759 24173 tablet_bootstrap.cc:654] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.620630 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c575f9ccf4c4495390078283ae331c7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.621202 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c575f9ccf4c4495390078283ae331c7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.623410 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 10adabdd31f848548ef6f2408ccc2050 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.623626 24034 tablet_service.cc:1514] Processing DeleteTablet for tablet 175adf0ecb504e28a3d39f001f977839 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.623910 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f7b5f018d84680ac5533bf99102d31 with delete_type TABLET_DATA_DELETED (replaced by tablet c17e1be6ef294769a01981f4d361be21) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.624253 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet f36ceb0222ef473a828ca795f393174d with delete_type TABLET_DATA_DELETED (replaced by tablet df9402cf8abc4b4d893d42f9a3a99016) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.623405 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c08228fcd1304528b9446003851f6c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.630779 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2849e243f1a94b139d3234f0c8e3c755"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.631409 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2849e243f1a94b139d3234f0c8e3c755 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.632647 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5b4e617d1c8949daa34aa358c1528f88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.633101 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b4e617d1c8949daa34aa358c1528f88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.633816 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_becb097785284ffba69cdbe40e6480ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:36.634179 24173 tablet_bootstrap.cc:492] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
W20240502 08:16:36.634258 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_becb097785284ffba69cdbe40e6480ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.634663 24173 ts_tablet_manager.cc:1367] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.001s
I20240502 08:16:36.635723 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_60ce09e8ee9c47a4bb1d85e40f063338" } new_table_name: "default.Table_a0ebd3d8bd13425ea80ffd0c428e20da" modify_external_catalogs: true
I20240502 08:16:36.635744 24173 raft_consensus.cc:348] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.635973 24173 raft_consensus.cc:374] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.636072 24173 raft_consensus.cc:729] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.636368 24173 consensus_queue.cc:260] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.636940 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0542f22f24e14c5887f78b1456fa361b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:36.637212 24173 ts_tablet_manager.cc:1398] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240502 08:16:36.637336 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0542f22f24e14c5887f78b1456fa361b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.637645 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_22e9411445aa459286b4f9038d84651b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.638042 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22e9411445aa459286b4f9038d84651b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.640415 24180 ts_tablet_manager.cc:1888] T 26c9913f1ca146fe84bc07ca1b4315c5 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.640486 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 (DEFAULT_TABLE table=default.Table_9663c413d5be4a049eb7a02ececc1276 [id=ad4f0c8adc524a019beaf37e153b33a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.640588 24180 log.cc:1196] T 26c9913f1ca146fe84bc07ca1b4315c5 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/26c9913f1ca146fe84bc07ca1b4315c5
I20240502 08:16:36.640532 24241 fs_manager.cc:709] Time spent opening directory manager: real 0.043s user 0.008s sys 0.000s
I20240502 08:16:36.640892 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c03a4353c4e46e3bcdf3a36aac04bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.640969 24180 ts_tablet_manager.cc:1909] T 26c9913f1ca146fe84bc07ca1b4315c5 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:36.641177 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 171406810543435e832a47ad5b3c80a4 (DEFAULT_TABLE table=default.Table_a286e67e441e43a1b4738cd219dbea98 [id=b56e3c0686244ad6b49afdd589c5ed34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.641505 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171406810543435e832a47ad5b3c80a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.641883 24180 tablet_replica.cc:330] T 0f2ff50fa357448494434d1487ce83bc P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:36.642076 24180 raft_consensus.cc:2229] T 0f2ff50fa357448494434d1487ce83bc P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.642235 24180 raft_consensus.cc:2258] T 0f2ff50fa357448494434d1487ce83bc P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.643090 24180 ts_tablet_manager.cc:1875] T 0f2ff50fa357448494434d1487ce83bc P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:36.648216 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 7c374f42f353497fa1d745164e7fbf61 (DEFAULT_TABLE table=default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:36.648451 23723 leader_election.cc:341] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:36.649755 24173 tablet_bootstrap.cc:492] T ab70513985b54416856a29b32cecae4d P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:36.649794 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 171406810543435e832a47ad5b3c80a4 (DEFAULT_TABLE table=default.Table_a286e67e441e43a1b4738cd219dbea98 [id=b56e3c0686244ad6b49afdd589c5ed34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.650043 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dff0d73a933427fb1f2686bac33eba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.651180 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 848a0865eb2b4828a2594a17d96869d0 (DEFAULT_TABLE table=default.Table_b9d00c29b8c2498895efd283a90c12e5 [id=7ffacb15a8a249258fb42c4312f87d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.652065 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 10adabdd31f848548ef6f2408ccc2050 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.654583 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848a0865eb2b4828a2594a17d96869d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.660673 23603 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 1ce5ccfd29724dc49afd0892b99f3169 (table default.Table_bdbe807258a5454795eaff8c27bb17dd [id=3a9d7e8fe2c84cb4942fee9053a012b0]). Expected version 1 got 0
I20240502 08:16:36.661746 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 32150c7e130f4069a2a3a02e625bfb93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.661800 24173 tablet_bootstrap.cc:654] T ab70513985b54416856a29b32cecae4d P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.663272 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c374f42f353497fa1d745164e7fbf61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.663686 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171406810543435e832a47ad5b3c80a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.664285 23895 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f7b5f018d84680ac5533bf99102d31 with delete_type TABLET_DATA_DELETED (replaced by tablet c17e1be6ef294769a01981f4d361be21) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.665158 24175 tablet_bootstrap.cc:492] T 3052510d77f344bf87decb9c2e5ec0bd P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
W20240502 08:16:36.665329 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 10adabdd31f848548ef6f2408ccc2050 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10adabdd31f848548ef6f2408ccc2050 already in progress: creating tablet
I20240502 08:16:36.663386 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 7c374f42f353497fa1d745164e7fbf61 (DEFAULT_TABLE table=default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.666095 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c374f42f353497fa1d745164e7fbf61. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:36.668557 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet d8f7b5f018d84680ac5533bf99102d31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8f7b5f018d84680ac5533bf99102d31 already in progress: creating tablet
I20240502 08:16:36.670214 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet f36ceb0222ef473a828ca795f393174d with delete_type TABLET_DATA_DELETED (replaced by tablet df9402cf8abc4b4d893d42f9a3a99016) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.672262 24175 ts_tablet_manager.cc:1367] T 3052510d77f344bf87decb9c2e5ec0bd P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.093s user 0.004s sys 0.008s
I20240502 08:16:36.673377 24175 raft_consensus.cc:348] T 3052510d77f344bf87decb9c2e5ec0bd P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:36.674698 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet f36ceb0222ef473a828ca795f393174d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f36ceb0222ef473a828ca795f393174d already in progress: creating tablet
I20240502 08:16:36.675611 24175 raft_consensus.cc:374] T 3052510d77f344bf87decb9c2e5ec0bd P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.675654 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 175adf0ecb504e28a3d39f001f977839 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.675729 24175 raft_consensus.cc:729] T 3052510d77f344bf87decb9c2e5ec0bd P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.676682 24173 tablet_bootstrap.cc:492] T ab70513985b54416856a29b32cecae4d P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:36.676900 24173 ts_tablet_manager.cc:1367] T ab70513985b54416856a29b32cecae4d P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.001s
I20240502 08:16:36.676826 24175 consensus_queue.cc:260] T 3052510d77f344bf87decb9c2e5ec0bd P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.677973 24179 tablet_replica.cc:330] T 175adf0ecb504e28a3d39f001f977839 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:36.677873 24173 raft_consensus.cc:348] T ab70513985b54416856a29b32cecae4d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.678164 24179 raft_consensus.cc:2229] T 175adf0ecb504e28a3d39f001f977839 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.681361 24353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.682314 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_59b22bd34c884d0ebe95d6737550e50d" } new_table_name: "default.Table_6ccfa5bc466242b2a325aa1ce6c85e9c" modify_external_catalogs: true
I20240502 08:16:36.682562 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet 175adf0ecb504e28a3d39f001f977839 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.682767 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 10adabdd31f848548ef6f2408ccc2050 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.683668 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_623eac7101f94d959fb83b6f51d8fe0a" } new_table_name: "default.Table_190eae2a8c64440f8d19842f8b6ac254" modify_external_catalogs: true
I20240502 08:16:36.686053 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet f36ceb0222ef473a828ca795f393174d with delete_type TABLET_DATA_DELETED (replaced by tablet df9402cf8abc4b4d893d42f9a3a99016) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.686053 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 26c9913f1ca146fe84bc07ca1b4315c5 (table default.Table_f8286fd5bd9f4a1ca39afc1e8b57eae2 [id=93515920607e433ca1e2b5573176dc0c]) successfully deleted
W20240502 08:16:36.671284 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 46537.4us
I20240502 08:16:36.686594 24241 fs_manager.cc:725] Time spent opening block manager: real 0.027s user 0.004s sys 0.000s
I20240502 08:16:36.698808 24175 ts_tablet_manager.cc:1398] T 3052510d77f344bf87decb9c2e5ec0bd P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20240502 08:16:36.699605 24179 raft_consensus.cc:2258] T 175adf0ecb504e28a3d39f001f977839 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.699895 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "405ea360814e4593a289d753821277ff" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:36.700915 24179 ts_tablet_manager.cc:1875] T 175adf0ecb504e28a3d39f001f977839 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:36.700915 24175 tablet_bootstrap.cc:492] T c1c2a988a3524c8fb7410e8508bd1abc P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:36.702754 24180 ts_tablet_manager.cc:1888] T 0f2ff50fa357448494434d1487ce83bc P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.703581 24180 log.cc:1196] T 0f2ff50fa357448494434d1487ce83bc P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/0f2ff50fa357448494434d1487ce83bc
I20240502 08:16:36.704905 24180 ts_tablet_manager.cc:1909] T 0f2ff50fa357448494434d1487ce83bc P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:36.706809 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_102eeba031aa4389820d28985935dd4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.707434 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_102eeba031aa4389820d28985935dd4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.708938 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6b8552ddc1814ad5a20c72190929a5d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.709448 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b8552ddc1814ad5a20c72190929a5d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:36.710649 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 643ce929e74c499da835fbe9b63b8880 has no consensus state
I20240502 08:16:36.711839 24175 tablet_bootstrap.cc:654] T c1c2a988a3524c8fb7410e8508bd1abc P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:36.712909 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0dff0d73a933427fb1f2686bac33eba1 consensus state has no leader
I20240502 08:16:36.716321 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9ae8c17e33fc47d5b93166ab4d269138" } new_table_name: "default.Table_20df1b504848439c869ae3739ac54185" modify_external_catalogs: true
I20240502 08:16:36.717707 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_aa989e0f3fca40aeb4beda0a72a1a79e" } new_table_name: "default.Table_785d5ddac96746379d1117bc8379566b" modify_external_catalogs: true
I20240502 08:16:36.719767 24329 raft_consensus.cc:504] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.722370 24180 tablet_replica.cc:330] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:36.722553 24180 raft_consensus.cc:2229] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.722802 24180 raft_consensus.cc:2258] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.723415 24180 ts_tablet_manager.cc:1875] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:36.724439 24329 leader_election.cc:288] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:36.724857 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 0f2ff50fa357448494434d1487ce83bc (table default.Table_065c3fa8292242d59b5fbebbfa233fdc [id=efa730295c3c4bb69e64ca3fa324abb1]) successfully deleted
W20240502 08:16:36.726212 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 23ce83c8001d4b2988b88c0ed9bf948c consensus state has no leader
I20240502 08:16:36.731567 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet 3052510d77f344bf87decb9c2e5ec0bd with delete_type TABLET_DATA_DELETED (replaced by tablet 23ce83c8001d4b2988b88c0ed9bf948c) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.735985 24179 ts_tablet_manager.cc:1888] T 175adf0ecb504e28a3d39f001f977839 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.736167 24179 log.cc:1196] T 175adf0ecb504e28a3d39f001f977839 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/175adf0ecb504e28a3d39f001f977839
I20240502 08:16:36.736572 24179 ts_tablet_manager.cc:1909] T 175adf0ecb504e28a3d39f001f977839 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:36.736768 24175 tablet_bootstrap.cc:492] T c1c2a988a3524c8fb7410e8508bd1abc P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:36.736905 24175 ts_tablet_manager.cc:1367] T c1c2a988a3524c8fb7410e8508bd1abc P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.002s
I20240502 08:16:36.738588 24175 raft_consensus.cc:348] T c1c2a988a3524c8fb7410e8508bd1abc P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.738785 24175 raft_consensus.cc:374] T c1c2a988a3524c8fb7410e8508bd1abc P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.739156 24175 raft_consensus.cc:729] T c1c2a988a3524c8fb7410e8508bd1abc P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.739395 24175 consensus_queue.cc:260] T c1c2a988a3524c8fb7410e8508bd1abc P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.741997 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f7b5f018d84680ac5533bf99102d31 with delete_type TABLET_DATA_DELETED (replaced by tablet c17e1be6ef294769a01981f4d361be21) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.742254 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 175adf0ecb504e28a3d39f001f977839 (table default.Table_9530003aa5b841b683920da217d58576 [id=a6a5f4c65807479392c01ac5c955c959]) successfully deleted
W20240502 08:16:36.743088 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 49b816ea1d474f2db94119441146ba78 consensus state has no leader
I20240502 08:16:36.743147 24175 ts_tablet_manager.cc:1398] T c1c2a988a3524c8fb7410e8508bd1abc P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:16:36.743335 24179 tablet_replica.cc:330] T 3052510d77f344bf87decb9c2e5ec0bd P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
W20240502 08:16:36.743779 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7a19310948e248d1a85479c41c61bda2 consensus state has no leader
I20240502 08:16:36.747116 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_59462d2ccf514f0da864562120217931" } modify_external_catalogs: true
I20240502 08:16:36.747370 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_59462d2ccf514f0da864562120217931" } modify_external_catalogs: true
I20240502 08:16:36.748085 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_24d1bd1b4dcc4882884e13873cf896ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.748633 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24d1bd1b4dcc4882884e13873cf896ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.678215 24173 raft_consensus.cc:374] T ab70513985b54416856a29b32cecae4d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.749743 24173 raft_consensus.cc:729] T ab70513985b54416856a29b32cecae4d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.687471 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "405ea360814e4593a289d753821277ff" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:36.750007 24173 consensus_queue.cc:260] T ab70513985b54416856a29b32cecae4d P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.750645 24175 tablet_bootstrap.cc:492] T db80a159f14e485a971fd8bf58a21c4a P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:36.751879 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768167b977c14b53a7537879e349d354" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:36.752064 24059 raft_consensus.cc:3037] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:36.686830 24241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data,/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/wal
uuid: "bcc8a323e6b9429eaaf4e0c1698ad077"
format_stamp: "Formatted at 2024-05-02 08:16:30 on dist-test-slave-m44h"
I20240502 08:16:36.753943 24241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:36.754652 24059 raft_consensus.cc:2454] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
W20240502 08:16:36.758531 23723 leader_election.cc:341] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:36.758870 23723 leader_election.cc:302] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:36.759183 24329 raft_consensus.cc:2781] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
W20240502 08:16:36.759229 23723 leader_election.cc:341] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:36.759336 24329 raft_consensus.cc:686] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:36.759552 24329 consensus_queue.cc:237] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.696841 24211 raft_consensus.cc:482] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:36.760022 24211 raft_consensus.cc:504] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.760641 24211 leader_election.cc:288] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:36.761868 23723 leader_election.cc:302] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:36.689766 24181 raft_consensus.cc:2229] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.762418 24343 raft_consensus.cc:2726] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:36.762472 24181 raft_consensus.cc:2258] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:36.763082 23993 leader_election.cc:341] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: e17eec8f2c1a4190945b434794635b52
I20240502 08:16:36.763247 24181 ts_tablet_manager.cc:1875] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:36.765740 24179 raft_consensus.cc:2229] T 3052510d77f344bf87decb9c2e5ec0bd P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.768729 24180 ts_tablet_manager.cc:1888] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.768877 24180 log.cc:1196] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/05cbbf9614094afaa6f7cee0af597242
I20240502 08:16:36.769256 24180 ts_tablet_manager.cc:1909] T 05cbbf9614094afaa6f7cee0af597242 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:36.770313 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 05cbbf9614094afaa6f7cee0af597242 (table default.Table_6c583799e352452a87de6faa19d88490 [id=80701c77b9b04253bba1d4438e6f2a78]) successfully deleted
I20240502 08:16:36.773212 23610 master_service.cc:946] ReplaceTablet: received request to replace tablet e0af5f4daf3149738647a08219efb245 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:36.774550 24175 tablet_bootstrap.cc:654] T db80a159f14e485a971fd8bf58a21c4a P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.776480 24173 ts_tablet_manager.cc:1398] T ab70513985b54416856a29b32cecae4d P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.099s user 0.004s sys 0.000s
I20240502 08:16:36.776805 24173 tablet_bootstrap.cc:492] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:36.778545 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 0be4f8d5cc09483d8d1aa1e37679b544 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0be4f8d5cc09483d8d1aa1e37679b544 already in progress: creating tablet
W20240502 08:16:36.779523 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 485c772245fe48a39fc6ceb6377ac0b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 485c772245fe48a39fc6ceb6377ac0b6 already in progress: creating tablet
I20240502 08:16:36.780128 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17eec8f2c1a4190945b434794635b52" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:36.780732 23996 leader_election.cc:341] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:36.780867 23996 leader_election.cc:302] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:36.774966 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768167b977c14b53a7537879e349d354" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:36.781275 24173 tablet_bootstrap.cc:654] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.782178 24179 raft_consensus.cc:2258] T 3052510d77f344bf87decb9c2e5ec0bd P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.782194 24355 raft_consensus.cc:2726] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:36.782891 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet fd57b0357ca749c389bc61add20cd3e7 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:36.781777 24174 tablet_bootstrap.cc:492] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:36.784096 24179 ts_tablet_manager.cc:1875] T 3052510d77f344bf87decb9c2e5ec0bd P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:36.784298 24174 ts_tablet_manager.cc:1367] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.233s user 0.051s sys 0.104s
W20240502 08:16:36.783603 23723 leader_election.cc:341] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:36.789302 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c2a988a3524c8fb7410e8508bd1abc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.782634 24180 tablet_replica.cc:330] T 32150c7e130f4069a2a3a02e625bfb93 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:36.791777 24173 tablet_bootstrap.cc:492] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:36.793427 24180 raft_consensus.cc:2229] T 32150c7e130f4069a2a3a02e625bfb93 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.794126 24180 raft_consensus.cc:2258] T 32150c7e130f4069a2a3a02e625bfb93 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.794901 24173 ts_tablet_manager.cc:1367] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.003s
I20240502 08:16:36.793406 24174 raft_consensus.cc:348] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.795094 24174 raft_consensus.cc:374] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.795218 24174 raft_consensus.cc:729] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.795559 24174 consensus_queue.cc:260] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.796761 24173 raft_consensus.cc:348] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.797962 24174 ts_tablet_manager.cc:1398] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 08:16:36.798243 24174 tablet_bootstrap.cc:492] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:36.799559 24173 raft_consensus.cc:374] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.799875 24174 tablet_bootstrap.cc:654] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.800072 24173 raft_consensus.cc:729] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.800078 23925 raft_consensus.cc:3037] T ec4028a8e66442bfabac9e587aed00bb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:36.801664 24175 tablet_bootstrap.cc:492] T db80a159f14e485a971fd8bf58a21c4a P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:36.801818 24175 ts_tablet_manager.cc:1367] T db80a159f14e485a971fd8bf58a21c4a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.009s
I20240502 08:16:36.802589 24175 raft_consensus.cc:348] T db80a159f14e485a971fd8bf58a21c4a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.802898 24175 raft_consensus.cc:374] T db80a159f14e485a971fd8bf58a21c4a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.802973 24175 raft_consensus.cc:729] T db80a159f14e485a971fd8bf58a21c4a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.803218 24175 consensus_queue.cc:260] T db80a159f14e485a971fd8bf58a21c4a P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.797413 24181 ts_tablet_manager.cc:1888] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.804338 24181 log.cc:1196] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/db80a159f14e485a971fd8bf58a21c4a
I20240502 08:16:36.804399 24173 consensus_queue.cc:260] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.804754 24181 ts_tablet_manager.cc:1909] T db80a159f14e485a971fd8bf58a21c4a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:36.805449 24179 ts_tablet_manager.cc:1888] T 3052510d77f344bf87decb9c2e5ec0bd P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.805614 24173 ts_tablet_manager.cc:1398] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20240502 08:16:36.806116 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet db80a159f14e485a971fd8bf58a21c4a (table default.Table_cc80d1cedafa4e71906e5e9530fdd828 [id=47682c26a85842c7b8991bf3828a9336]) successfully deleted
I20240502 08:16:36.805903 24173 tablet_bootstrap.cc:492] T a15ff3120b1943a69ddb609d0575ae1a P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:36.806946 24355 raft_consensus.cc:482] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:36.807106 24355 raft_consensus.cc:504] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:36.807674 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet d5bab1b7c48248a7973c27d0a865b34b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5bab1b7c48248a7973c27d0a865b34b already in progress: creating tablet
I20240502 08:16:36.807832 24355 leader_election.cc:288] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
W20240502 08:16:36.808497 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet f51b26043ef44b30846498cd1497607c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f51b26043ef44b30846498cd1497607c already in progress: creating tablet
I20240502 08:16:36.808835 24355 raft_consensus.cc:482] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:36.808823 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32150c7e130f4069a2a3a02e625bfb93" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:36.808960 24355 raft_consensus.cc:504] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.809136 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32150c7e130f4069a2a3a02e625bfb93" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:36.809384 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet fd57b0357ca749c389bc61add20cd3e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd57b0357ca749c389bc61add20cd3e7 already in progress: creating tablet
I20240502 08:16:36.809522 24355 leader_election.cc:288] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:36.809711 23996 leader_election.cc:341] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:36.810101 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 5c95dfda9d3b4e2ca9397b77ad71c011 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c95dfda9d3b4e2ca9397b77ad71c011 already in progress: creating tablet
I20240502 08:16:36.810124 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2ff50fa357448494434d1487ce83bc" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:36.810349 23993 leader_election.cc:341] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:36.810545 23996 leader_election.cc:341] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:36.810513 23993 leader_election.cc:302] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:36.810674 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 already in progress: creating tablet
I20240502 08:16:36.811120 24355 raft_consensus.cc:2726] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:36.811394 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet a7c3800fb2784a1dba12d01c56c226e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7c3800fb2784a1dba12d01c56c226e5 already in progress: creating tablet
W20240502 08:16:36.811944 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 21832f0151ec4cf48f51f9bd2cb2b5ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21832f0151ec4cf48f51f9bd2cb2b5ec already in progress: creating tablet
I20240502 08:16:36.812451 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet 6bb2d0a4898544f182ab8070d29a7940 from {username='slave'} at 127.0.0.1:36668
W20240502 08:16:36.812584 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 405ea360814e4593a289d753821277ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 405ea360814e4593a289d753821277ff already in progress: creating tablet
I20240502 08:16:36.812912 24181 tablet_replica.cc:330] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:36.813115 24181 raft_consensus.cc:2229] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.813361 24181 raft_consensus.cc:2258] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.813452 24179 log.cc:1196] T 3052510d77f344bf87decb9c2e5ec0bd P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/3052510d77f344bf87decb9c2e5ec0bd
I20240502 08:16:36.813853 24179 ts_tablet_manager.cc:1909] T 3052510d77f344bf87decb9c2e5ec0bd P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:36.814337 24181 ts_tablet_manager.cc:1875] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:36.814779 24179 tablet_replica.cc:330] T c1c2a988a3524c8fb7410e8508bd1abc P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:36.814980 24179 raft_consensus.cc:2229] T c1c2a988a3524c8fb7410e8508bd1abc P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.815168 24179 raft_consensus.cc:2258] T c1c2a988a3524c8fb7410e8508bd1abc P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.815837 24179 ts_tablet_manager.cc:1875] T c1c2a988a3524c8fb7410e8508bd1abc P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:36.816663 23993 leader_election.cc:341] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 0f2ff50fa357448494434d1487ce83bc
I20240502 08:16:36.816814 23993 leader_election.cc:302] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:36.816987 23993 raft_consensus.cc:2670] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:16:36.817255 24173 tablet_bootstrap.cc:654] T a15ff3120b1943a69ddb609d0575ae1a P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.818019 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 3052510d77f344bf87decb9c2e5ec0bd (table default.Table_190eae2a8c64440f8d19842f8b6ac254 [id=54454efbe9394170822f237dc8e6115e]) successfully deleted
I20240502 08:16:36.819110 24175 ts_tablet_manager.cc:1398] T db80a159f14e485a971fd8bf58a21c4a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240502 08:16:36.819597 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_95acc6e36a0b448eb3fd6d5df86e2de4" } modify_external_catalogs: true
I20240502 08:16:36.819715 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d9b6d863d79341ebb994770c34ae2174" } modify_external_catalogs: true
I20240502 08:16:36.819926 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d9b6d863d79341ebb994770c34ae2174" } modify_external_catalogs: true
I20240502 08:16:36.819782 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_95acc6e36a0b448eb3fd6d5df86e2de4" } modify_external_catalogs: true
I20240502 08:16:36.820369 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet db80a159f14e485a971fd8bf58a21c4a with delete_type TABLET_DATA_DELETED (replaced by tablet e134e90efb4a48bb82b53a80bda708b3) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.821743 24175 tablet_bootstrap.cc:492] T e17eec8f2c1a4190945b434794635b52 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:36.823356 24175 tablet_bootstrap.cc:654] T e17eec8f2c1a4190945b434794635b52 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.825721 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c5d06caccb424872b965a751c171a0a6" } modify_external_catalogs: true
I20240502 08:16:36.825842 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c5d06caccb424872b965a751c171a0a6" } modify_external_catalogs: true
I20240502 08:16:36.827270 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_20c5ace604d64b6cabead6037cdc04e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.827880 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20c5ace604d64b6cabead6037cdc04e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.828835 24341 tablet.cc:1722] T ec4028a8e66442bfabac9e587aed00bb P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:36.829223 23608 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet ec4028a8e66442bfabac9e587aed00bb (table default.Table_62b86da21eb84bc387b64725dabda733 [id=1e590d80e3e34e1aac8a0c7908a9d813]). Expected version 3 got 0
I20240502 08:16:36.825459 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f48b4cdee51644e0b3c904d772f01404" } modify_external_catalogs: true
I20240502 08:16:36.829556 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f48b4cdee51644e0b3c904d772f01404" } modify_external_catalogs: true
I20240502 08:16:36.833221 23606 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet ab70513985b54416856a29b32cecae4d (table default.Table_d9b6d863d79341ebb994770c34ae2174 [id=259cb14e2ef6427f8dfba46f188050dd]). Expected version 1 got 0
I20240502 08:16:36.837221 23767 tablet_service.cc:1467] Processing CreateTablet for tablet bcfacc302ff04178bfffc48111cd7a3f (DEFAULT_TABLE table=default.Table_6f82c95f8aa44cadabb6c55876c6f241 [id=fa5100e581a7430ab8f0c19b2daae994]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.837659 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcfacc302ff04178bfffc48111cd7a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.837841 23896 tablet_service.cc:1467] Processing CreateTablet for tablet bcfacc302ff04178bfffc48111cd7a3f (DEFAULT_TABLE table=default.Table_6f82c95f8aa44cadabb6c55876c6f241 [id=fa5100e581a7430ab8f0c19b2daae994]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.838217 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcfacc302ff04178bfffc48111cd7a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.841881 24030 tablet_service.cc:1467] Processing CreateTablet for tablet bcfacc302ff04178bfffc48111cd7a3f (DEFAULT_TABLE table=default.Table_6f82c95f8aa44cadabb6c55876c6f241 [id=fa5100e581a7430ab8f0c19b2daae994]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.842013 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 71aa916247c449fea718b35869a6df20 (DEFAULT_TABLE table=default.Table_4688a59d76ee4b5a931536f5deddfd4e [id=5cee878b3a31426c879a1bf95045bb09]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.842155 24175 tablet_bootstrap.cc:492] T e17eec8f2c1a4190945b434794635b52 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:36.842298 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcfacc302ff04178bfffc48111cd7a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.842355 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71aa916247c449fea718b35869a6df20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.842284 24175 ts_tablet_manager.cc:1367] T e17eec8f2c1a4190945b434794635b52 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20240502 08:16:36.844414 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 6788ffd2eb2248c3b84e474de56c37a9
I20240502 08:16:36.844612 24175 raft_consensus.cc:348] T e17eec8f2c1a4190945b434794635b52 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.844803 24175 raft_consensus.cc:374] T e17eec8f2c1a4190945b434794635b52 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.844894 24175 raft_consensus.cc:729] T e17eec8f2c1a4190945b434794635b52 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.845185 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 01675f637a5a4645949abfd87b3d09a2
I20240502 08:16:36.845144 24175 consensus_queue.cc:260] T e17eec8f2c1a4190945b434794635b52 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.845754 24175 ts_tablet_manager.cc:1398] T e17eec8f2c1a4190945b434794635b52 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:36.845875 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 6788ffd2eb2248c3b84e474de56c37a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.846038 24175 tablet_bootstrap.cc:492] T 774487be9bc644cab4e420c3d33b3cad P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:36.847102 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 6788ffd2eb2248c3b84e474de56c37a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.847391 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 01675f637a5a4645949abfd87b3d09a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.847681 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 01675f637a5a4645949abfd87b3d09a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.848392 24175 tablet_bootstrap.cc:654] T 774487be9bc644cab4e420c3d33b3cad P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.849095 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 6bb2d0a4898544f182ab8070d29a7940
I20240502 08:16:36.849856 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb2d0a4898544f182ab8070d29a7940 with delete_type TABLET_DATA_DELETED (replaced by tablet 1cd8338500f54604abcd6d42702c2eca) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.849885 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 2bfb532c9d4c41afb7acaa4831841306
I20240502 08:16:36.850450 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bfb532c9d4c41afb7acaa4831841306 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.851164 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bfb532c9d4c41afb7acaa4831841306 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.851629 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb2d0a4898544f182ab8070d29a7940 with delete_type TABLET_DATA_DELETED (replaced by tablet 1cd8338500f54604abcd6d42702c2eca) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.851846 23609 catalog_manager.cc:6554] ReplaceTablet: tablet 6bb2d0a4898544f182ab8070d29a7940 deleted and replaced by tablet 1cd8338500f54604abcd6d42702c2eca
I20240502 08:16:36.851979 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet 01675f637a5a4645949abfd87b3d09a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.852209 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bfb532c9d4c41afb7acaa4831841306 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.852401 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb2d0a4898544f182ab8070d29a7940 with delete_type TABLET_DATA_DELETED (replaced by tablet 1cd8338500f54604abcd6d42702c2eca) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.852644 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 6788ffd2eb2248c3b84e474de56c37a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.856817 24173 tablet_bootstrap.cc:492] T a15ff3120b1943a69ddb609d0575ae1a P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:36.857005 24173 ts_tablet_manager.cc:1367] T a15ff3120b1943a69ddb609d0575ae1a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.001s
I20240502 08:16:36.857364 24175 tablet_bootstrap.cc:492] T 774487be9bc644cab4e420c3d33b3cad P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:36.857527 24175 ts_tablet_manager.cc:1367] T 774487be9bc644cab4e420c3d33b3cad P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240502 08:16:36.857858 24173 raft_consensus.cc:348] T a15ff3120b1943a69ddb609d0575ae1a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.858119 24173 raft_consensus.cc:374] T a15ff3120b1943a69ddb609d0575ae1a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.858217 24173 raft_consensus.cc:729] T a15ff3120b1943a69ddb609d0575ae1a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.858191 24175 raft_consensus.cc:348] T 774487be9bc644cab4e420c3d33b3cad P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.858386 24175 raft_consensus.cc:374] T 774487be9bc644cab4e420c3d33b3cad P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.858528 24175 raft_consensus.cc:729] T 774487be9bc644cab4e420c3d33b3cad P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.858543 24173 consensus_queue.cc:260] T a15ff3120b1943a69ddb609d0575ae1a P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.858778 24175 consensus_queue.cc:260] T 774487be9bc644cab4e420c3d33b3cad P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.859315 24173 ts_tablet_manager.cc:1398] T a15ff3120b1943a69ddb609d0575ae1a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:36.859890 24173 tablet_bootstrap.cc:492] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:36.861657 24173 tablet_bootstrap.cc:654] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.862478 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet ab70513985b54416856a29b32cecae4d
I20240502 08:16:36.863559 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet ab70513985b54416856a29b32cecae4d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.863713 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet e17eec8f2c1a4190945b434794635b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.864071 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet ab70513985b54416856a29b32cecae4d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.864264 24175 ts_tablet_manager.cc:1398] T 774487be9bc644cab4e420c3d33b3cad P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 08:16:36.864612 24175 tablet_bootstrap.cc:492] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:36.868283 24175 tablet_bootstrap.cc:654] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.868297 24173 tablet_bootstrap.cc:492] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:36.868464 24173 ts_tablet_manager.cc:1367] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20240502 08:16:36.868955 24174 tablet_bootstrap.cc:492] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:36.869156 24174 ts_tablet_manager.cc:1367] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.003s
I20240502 08:16:36.869127 24173 raft_consensus.cc:348] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.869318 24173 raft_consensus.cc:374] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.869412 24173 raft_consensus.cc:729] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.869799 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 774487be9bc644cab4e420c3d33b3cad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.869827 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet a15ff3120b1943a69ddb609d0575ae1a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.870692 24174 raft_consensus.cc:348] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.871027 24174 raft_consensus.cc:374] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.871127 24174 raft_consensus.cc:729] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.871409 24174 consensus_queue.cc:260] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.872135 24174 ts_tablet_manager.cc:1398] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:16:36.872438 24174 tablet_bootstrap.cc:492] T c7aa03e73acd4a9da5574b9ba93415ae P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:36.873530 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet ab70513985b54416856a29b32cecae4d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.875511 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 71aa916247c449fea718b35869a6df20 (DEFAULT_TABLE table=default.Table_4688a59d76ee4b5a931536f5deddfd4e [id=5cee878b3a31426c879a1bf95045bb09]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.875851 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71aa916247c449fea718b35869a6df20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.876296 24175 tablet_bootstrap.cc:492] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:36.876349 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 0882b7509a74475780eb700818aab398 (DEFAULT_TABLE table=default.Table_bc1e9d346aa846f8abca7504185f7cdb [id=8d27878c6c8f4234908a2c78cb8bbd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.876425 24175 ts_tablet_manager.cc:1367] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20240502 08:16:36.876701 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0882b7509a74475780eb700818aab398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.877244 24175 raft_consensus.cc:348] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.877429 24175 raft_consensus.cc:374] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.877535 24175 raft_consensus.cc:729] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.877693 24355 raft_consensus.cc:482] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:36.877830 24355 raft_consensus.cc:504] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.877789 24175 consensus_queue.cc:260] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.878453 24355 leader_election.cc:288] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:36.878800 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 0882b7509a74475780eb700818aab398 (DEFAULT_TABLE table=default.Table_bc1e9d346aa846f8abca7504185f7cdb [id=8d27878c6c8f4234908a2c78cb8bbd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.879177 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0882b7509a74475780eb700818aab398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.879328 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce5ccfd29724dc49afd0892b99f3169" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:36.879956 23601 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 768167b977c14b53a7537879e349d354 (table default.Table_0a242b17e3d2450eb3add7dbff397394 [id=ab1669d2a12f4187809987ed41fa0379]). Expected version 1 got 0
I20240502 08:16:36.880036 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce5ccfd29724dc49afd0892b99f3169" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:36.880589 23993 leader_election.cc:341] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:36.880988 23996 leader_election.cc:341] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:36.881098 23996 leader_election.cc:302] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:36.881745 24175 ts_tablet_manager.cc:1398] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:16:36.881836 24032 tablet_service.cc:1467] Processing CreateTablet for tablet a3043c5ed98c401b82c6e967ca3f3613 (DEFAULT_TABLE table=default.Table_bd9e5acccefd4757b59b162649c7747c [id=d1a317371a5b43f185eb6453861e112a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.882078 24175 tablet_bootstrap.cc:492] T 71c0c2f3def643dfa99ff4efe65f6acf P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:36.882203 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3043c5ed98c401b82c6e967ca3f3613. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.882282 23899 tablet_service.cc:1467] Processing CreateTablet for tablet a3043c5ed98c401b82c6e967ca3f3613 (DEFAULT_TABLE table=default.Table_bd9e5acccefd4757b59b162649c7747c [id=d1a317371a5b43f185eb6453861e112a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.883191 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3043c5ed98c401b82c6e967ca3f3613. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.883831 23601 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 768167b977c14b53a7537879e349d354 (table default.Table_0a242b17e3d2450eb3add7dbff397394 [id=ab1669d2a12f4187809987ed41fa0379]). Expected version 1 got 0
I20240502 08:16:36.884390 24175 tablet_bootstrap.cc:654] T 71c0c2f3def643dfa99ff4efe65f6acf P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.886170 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 0882b7509a74475780eb700818aab398 (DEFAULT_TABLE table=default.Table_bc1e9d346aa846f8abca7504185f7cdb [id=8d27878c6c8f4234908a2c78cb8bbd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.886509 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0882b7509a74475780eb700818aab398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.887777 24179 ts_tablet_manager.cc:1888] T c1c2a988a3524c8fb7410e8508bd1abc P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.887897 24179 log.cc:1196] T c1c2a988a3524c8fb7410e8508bd1abc P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/c1c2a988a3524c8fb7410e8508bd1abc
I20240502 08:16:36.888296 24179 ts_tablet_manager.cc:1909] T c1c2a988a3524c8fb7410e8508bd1abc P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:36.888736 24179 tablet_replica.cc:330] T db80a159f14e485a971fd8bf58a21c4a P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:36.888862 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet c1c2a988a3524c8fb7410e8508bd1abc (table default.Table_b96eb662e06846039807538794442017 [id=31ddf074089243a89bab5928860c625e]) successfully deleted
I20240502 08:16:36.888931 24179 raft_consensus.cc:2229] T db80a159f14e485a971fd8bf58a21c4a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.889101 24179 raft_consensus.cc:2258] T db80a159f14e485a971fd8bf58a21c4a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.889317 23758 tablet_service.cc:1467] Processing CreateTablet for tablet a3043c5ed98c401b82c6e967ca3f3613 (DEFAULT_TABLE table=default.Table_bd9e5acccefd4757b59b162649c7747c [id=d1a317371a5b43f185eb6453861e112a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.889670 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3043c5ed98c401b82c6e967ca3f3613. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.892297 24174 tablet_bootstrap.cc:654] T c7aa03e73acd4a9da5574b9ba93415ae P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.892851 24180 ts_tablet_manager.cc:1875] T 32150c7e130f4069a2a3a02e625bfb93 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:36.894229 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 71aa916247c449fea718b35869a6df20 (DEFAULT_TABLE table=default.Table_4688a59d76ee4b5a931536f5deddfd4e [id=5cee878b3a31426c879a1bf95045bb09]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.894591 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71aa916247c449fea718b35869a6df20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.895916 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet 171406810543435e832a47ad5b3c80a4 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:36.896363 24181 ts_tablet_manager.cc:1888] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.896515 24181 log.cc:1196] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/0f2ff50fa357448494434d1487ce83bc
I20240502 08:16:36.896857 24181 ts_tablet_manager.cc:1909] T 0f2ff50fa357448494434d1487ce83bc P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:36.897856 24355 raft_consensus.cc:2726] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:36.897877 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 0f2ff50fa357448494434d1487ce83bc (table default.Table_065c3fa8292242d59b5fbebbfa233fdc [id=efa730295c3c4bb69e64ca3fa324abb1]) successfully deleted
I20240502 08:16:36.902968 24179 ts_tablet_manager.cc:1875] T db80a159f14e485a971fd8bf58a21c4a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:36.904095 23606 master_service.cc:946] ReplaceTablet: received request to replace tablet ba3d4b4584dd45b58ff0daa4f1c09a0a from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:36.905938 24181 tablet_replica.cc:330] T 26c9913f1ca146fe84bc07ca1b4315c5 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:36.906183 24181 raft_consensus.cc:2229] T 26c9913f1ca146fe84bc07ca1b4315c5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.906437 24181 raft_consensus.cc:2258] T 26c9913f1ca146fe84bc07ca1b4315c5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.908478 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 27346a7eae6a488fa3415646262a4b88 (DEFAULT_TABLE table=default.Table_5d5022ea23c34e4f960c366d5ac96c39 [id=999c7344afec4d3c858dc29a986a8426]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.909793 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27346a7eae6a488fa3415646262a4b88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.912246 24174 tablet_bootstrap.cc:492] T c7aa03e73acd4a9da5574b9ba93415ae P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:36.912484 24174 ts_tablet_manager.cc:1367] T c7aa03e73acd4a9da5574b9ba93415ae P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20240502 08:16:36.913502 24174 raft_consensus.cc:348] T c7aa03e73acd4a9da5574b9ba93415ae P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.913879 24174 raft_consensus.cc:374] T c7aa03e73acd4a9da5574b9ba93415ae P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.914031 24174 raft_consensus.cc:729] T c7aa03e73acd4a9da5574b9ba93415ae P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.914355 24174 consensus_queue.cc:260] T c7aa03e73acd4a9da5574b9ba93415ae P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.915356 24174 ts_tablet_manager.cc:1398] T c7aa03e73acd4a9da5574b9ba93415ae P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:36.915710 24174 tablet_bootstrap.cc:492] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:36.916671 24179 ts_tablet_manager.cc:1888] T db80a159f14e485a971fd8bf58a21c4a P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.916838 24179 log.cc:1196] T db80a159f14e485a971fd8bf58a21c4a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/db80a159f14e485a971fd8bf58a21c4a
I20240502 08:16:36.917222 24179 ts_tablet_manager.cc:1909] T db80a159f14e485a971fd8bf58a21c4a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:36.917865 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6c9edf34b95e4ad29551269a83447b4a" } new_table_name: "default.Table_ec5e04e87fba492ab9a96d4034db240a" modify_external_catalogs: true
I20240502 08:16:36.918480 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8c4dfc5c68e84b6ab815e9f333143678"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.919117 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c4dfc5c68e84b6ab815e9f333143678 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.919353 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1f6435dfbba64b358601dea132f0e1f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.919945 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f6435dfbba64b358601dea132f0e1f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.920708 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet db80a159f14e485a971fd8bf58a21c4a (table default.Table_cc80d1cedafa4e71906e5e9530fdd828 [id=47682c26a85842c7b8991bf3828a9336]) successfully deleted
W20240502 08:16:36.920956 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 6788ffd2eb2248c3b84e474de56c37a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6788ffd2eb2248c3b84e474de56c37a9 already in progress: creating tablet
I20240502 08:16:36.921133 24179 tablet_replica.cc:330] T e17eec8f2c1a4190945b434794635b52 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
W20240502 08:16:36.921211 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 01675f637a5a4645949abfd87b3d09a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01675f637a5a4645949abfd87b3d09a2 already in progress: creating tablet
I20240502 08:16:36.921130 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_725d048135a04e32bd016e60ac974715"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:36.921311 24179 raft_consensus.cc:2229] T e17eec8f2c1a4190945b434794635b52 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.921334 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet c470d420c45843a69a63d9800b7cfc97
W20240502 08:16:36.921447 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 6bb2d0a4898544f182ab8070d29a7940 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bb2d0a4898544f182ab8070d29a7940 already in progress: creating tablet
W20240502 08:16:36.921624 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_725d048135a04e32bd016e60ac974715 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.922127 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b9d00c29b8c2498895efd283a90c12e5" } new_table_name: "default.Table_7bae786c8c2e44bd9e893ff90696d216" modify_external_catalogs: true
I20240502 08:16:36.921487 24179 raft_consensus.cc:2258] T e17eec8f2c1a4190945b434794635b52 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.922569 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet c470d420c45843a69a63d9800b7cfc97 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.922739 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_bc9b016da1ce42b6a0509fbb664c8548"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:36.923035 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7c04460336ea48e79db571d95cdb1bcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.923180 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc9b016da1ce42b6a0509fbb664c8548 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.923280 24179 ts_tablet_manager.cc:1875] T e17eec8f2c1a4190945b434794635b52 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:36.923461 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c04460336ea48e79db571d95cdb1bcc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:36.924796 23597 catalog_manager.cc:4797] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 2bfb532c9d4c41afb7acaa4831841306 because the tablet was not found. No further retry: Not found: Tablet not found: 2bfb532c9d4c41afb7acaa4831841306
W20240502 08:16:36.925069 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet ab70513985b54416856a29b32cecae4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab70513985b54416856a29b32cecae4d already in progress: creating tablet
I20240502 08:16:36.926395 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 643ce929e74c499da835fbe9b63b8880 (DEFAULT_TABLE table=default.Table_a0ebd3d8bd13425ea80ffd0c428e20da [id=1c9043e3bf434bae83ea36aaef230b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.926800 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 643ce929e74c499da835fbe9b63b8880. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.929518 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet c470d420c45843a69a63d9800b7cfc97 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.930320 24179 ts_tablet_manager.cc:1888] T e17eec8f2c1a4190945b434794635b52 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.930424 24179 log.cc:1196] T e17eec8f2c1a4190945b434794635b52 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/e17eec8f2c1a4190945b434794635b52
I20240502 08:16:36.930757 24179 ts_tablet_manager.cc:1909] T e17eec8f2c1a4190945b434794635b52 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:36.931644 24179 tablet_replica.cc:330] T 774487be9bc644cab4e420c3d33b3cad P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:36.931859 24179 raft_consensus.cc:2229] T 774487be9bc644cab4e420c3d33b3cad P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.932016 24179 raft_consensus.cc:2258] T 774487be9bc644cab4e420c3d33b3cad P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.932749 24179 ts_tablet_manager.cc:1875] T 774487be9bc644cab4e420c3d33b3cad P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:36.933459 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet e17eec8f2c1a4190945b434794635b52 (table default.Table_b01c340249ea483683e385a7fa964a51 [id=ee0e1cf16cb34aa98699970438cc36ae]) successfully deleted
I20240502 08:16:36.935472 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e0af5f4daf3149738647a08219efb245
I20240502 08:16:36.935531 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet ba3d4b4584dd45b58ff0daa4f1c09a0a
I20240502 08:16:36.935930 23606 catalog_manager.cc:6554] ReplaceTablet: tablet ba3d4b4584dd45b58ff0daa4f1c09a0a deleted and replaced by tablet bd4a31e5aa834adeb9f59267247c6980
I20240502 08:16:36.935968 23610 catalog_manager.cc:6554] ReplaceTablet: tablet e0af5f4daf3149738647a08219efb245 deleted and replaced by tablet b8b1fa17dad143f0a7f1751e651eb81a
I20240502 08:16:36.936300 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2df84a3eb090456e8da6106c0a17131a" } new_table_name: "default.Table_a92f0fc8bc664448a6acb94dec9daa14" modify_external_catalogs: true
I20240502 08:16:36.937383 24175 tablet_bootstrap.cc:492] T 71c0c2f3def643dfa99ff4efe65f6acf P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:36.937546 24175 ts_tablet_manager.cc:1367] T 71c0c2f3def643dfa99ff4efe65f6acf P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.012s
I20240502 08:16:36.938187 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b3278c5d6d0b4593ac33bc9977f5c3b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:36.938271 24175 raft_consensus.cc:348] T 71c0c2f3def643dfa99ff4efe65f6acf P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.938436 24175 raft_consensus.cc:374] T 71c0c2f3def643dfa99ff4efe65f6acf P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.938521 24175 raft_consensus.cc:729] T 71c0c2f3def643dfa99ff4efe65f6acf P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
W20240502 08:16:36.938644 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3278c5d6d0b4593ac33bc9977f5c3b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.938728 24175 consensus_queue.cc:260] T 71c0c2f3def643dfa99ff4efe65f6acf P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.934348 24174 tablet_bootstrap.cc:654] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.939186 24175 ts_tablet_manager.cc:1398] T 71c0c2f3def643dfa99ff4efe65f6acf P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:16:36.937320 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet e0af5f4daf3149738647a08219efb245 with delete_type TABLET_DATA_DELETED (replaced by tablet b8b1fa17dad143f0a7f1751e651eb81a) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.939476 24175 tablet_bootstrap.cc:492] T 0f2ff50fa357448494434d1487ce83bc P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:36.936645 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet ba3d4b4584dd45b58ff0daa4f1c09a0a with delete_type TABLET_DATA_DELETED (replaced by tablet bd4a31e5aa834adeb9f59267247c6980) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.940407 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 171406810543435e832a47ad5b3c80a4
I20240502 08:16:36.940894 23601 catalog_manager.cc:6554] ReplaceTablet: tablet 171406810543435e832a47ad5b3c80a4 deleted and replaced by tablet 1e5de793e74247dd9e73b7b1d58027d9
I20240502 08:16:36.941955 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 171406810543435e832a47ad5b3c80a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e5de793e74247dd9e73b7b1d58027d9) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:36.942488 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet e0af5f4daf3149738647a08219efb245 with delete_type TABLET_DATA_DELETED (replaced by tablet b8b1fa17dad143f0a7f1751e651eb81a) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.942600 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet c470d420c45843a69a63d9800b7cfc97 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.942747 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet 171406810543435e832a47ad5b3c80a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e5de793e74247dd9e73b7b1d58027d9) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.942973 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet ba3d4b4584dd45b58ff0daa4f1c09a0a with delete_type TABLET_DATA_DELETED (replaced by tablet bd4a31e5aa834adeb9f59267247c6980) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.942963 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 27346a7eae6a488fa3415646262a4b88 (DEFAULT_TABLE table=default.Table_5d5022ea23c34e4f960c366d5ac96c39 [id=999c7344afec4d3c858dc29a986a8426]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.943346 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27346a7eae6a488fa3415646262a4b88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.942492 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 27346a7eae6a488fa3415646262a4b88 (DEFAULT_TABLE table=default.Table_5d5022ea23c34e4f960c366d5ac96c39 [id=999c7344afec4d3c858dc29a986a8426]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.944371 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27346a7eae6a488fa3415646262a4b88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.945472 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet e0af5f4daf3149738647a08219efb245 with delete_type TABLET_DATA_DELETED (replaced by tablet b8b1fa17dad143f0a7f1751e651eb81a) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.945523 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 71c0c2f3def643dfa99ff4efe65f6acf with delete_type TABLET_DATA_DELETED (replaced by tablet 203ed05d0bdc4b1f99207423b7fc848b) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.945679 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 171406810543435e832a47ad5b3c80a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e5de793e74247dd9e73b7b1d58027d9) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:36.946444 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 643ce929e74c499da835fbe9b63b8880 (DEFAULT_TABLE table=default.Table_a0ebd3d8bd13425ea80ffd0c428e20da [id=1c9043e3bf434bae83ea36aaef230b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.946889 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 643ce929e74c499da835fbe9b63b8880. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.946870 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 643ce929e74c499da835fbe9b63b8880 (DEFAULT_TABLE table=default.Table_a0ebd3d8bd13425ea80ffd0c428e20da [id=1c9043e3bf434bae83ea36aaef230b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.947328 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 643ce929e74c499da835fbe9b63b8880. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.947858 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 2bfb532c9d4c41afb7acaa4831841306 (DEFAULT_TABLE table=default.Table_f48b4cdee51644e0b3c904d772f01404 [id=35259dda872249a7bdddd93783103db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.948235 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bfb532c9d4c41afb7acaa4831841306. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.948736 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cd9334e686454c59999b7e3632c8195f" } new_table_name: "default.Table_ae8d973c3d53448d9068ba92779c32df" modify_external_catalogs: true
I20240502 08:16:36.949966 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 2bfb532c9d4c41afb7acaa4831841306 (DEFAULT_TABLE table=default.Table_f48b4cdee51644e0b3c904d772f01404 [id=35259dda872249a7bdddd93783103db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.950639 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bfb532c9d4c41afb7acaa4831841306. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.951927 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet ba3d4b4584dd45b58ff0daa4f1c09a0a with delete_type TABLET_DATA_DELETED (replaced by tablet bd4a31e5aa834adeb9f59267247c6980) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:36.955320 24175 tablet_bootstrap.cc:654] T 0f2ff50fa357448494434d1487ce83bc P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.961557 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 2bfb532c9d4c41afb7acaa4831841306 (DEFAULT_TABLE table=default.Table_f48b4cdee51644e0b3c904d772f01404 [id=35259dda872249a7bdddd93783103db8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.961982 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bfb532c9d4c41afb7acaa4831841306. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.961962 23893 tablet_service.cc:1467] Processing CreateTablet for tablet f4b291bdb81a4efbac7f6691e5577bbe (DEFAULT_TABLE table=default.Table_295f41167e6e481ea2443d5f8d02a998 [id=73559755c4594bcfa591d054964154ae]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:36.962020 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b11f504b93574e42a9bf30fde6d08207 consensus state has no leader
I20240502 08:16:36.962354 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b291bdb81a4efbac7f6691e5577bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.963163 23608 catalog_manager.cc:5484] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:36.963606 23608 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 768167b977c14b53a7537879e349d354 (table default.Table_0a242b17e3d2450eb3add7dbff397394 [id=ab1669d2a12f4187809987ed41fa0379]). Expected version 1 got 0
I20240502 08:16:36.967659 24030 tablet_service.cc:1467] Processing CreateTablet for tablet f4b291bdb81a4efbac7f6691e5577bbe (DEFAULT_TABLE table=default.Table_295f41167e6e481ea2443d5f8d02a998 [id=73559755c4594bcfa591d054964154ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.970703 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1ce01bea1dbd40d18ca5073c848b506d" } new_table_name: "default.Table_50e65b10059b451aa2a131c796cb3844" modify_external_catalogs: true
I20240502 08:16:36.968370 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b291bdb81a4efbac7f6691e5577bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.973201 24179 ts_tablet_manager.cc:1888] T 774487be9bc644cab4e420c3d33b3cad P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:36.973678 24180 ts_tablet_manager.cc:1888] T 32150c7e130f4069a2a3a02e625bfb93 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240502 08:16:36.975976 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 848a0865eb2b4828a2594a17d96869d0 consensus state has no leader
I20240502 08:16:36.976106 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_78c0d4912be14173a79ee9f2cab2288b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:36.976258 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e00515baeaae4ce9ad93000143f2a53a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.976642 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78c0d4912be14173a79ee9f2cab2288b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:36.976706 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e00515baeaae4ce9ad93000143f2a53a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.974829 24175 tablet_bootstrap.cc:492] T 0f2ff50fa357448494434d1487ce83bc P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:36.978471 24175 ts_tablet_manager.cc:1367] T 0f2ff50fa357448494434d1487ce83bc P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.002s
I20240502 08:16:36.979256 24175 raft_consensus.cc:348] T 0f2ff50fa357448494434d1487ce83bc P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.979480 24175 raft_consensus.cc:374] T 0f2ff50fa357448494434d1487ce83bc P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.979588 24175 raft_consensus.cc:729] T 0f2ff50fa357448494434d1487ce83bc P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.979844 24175 consensus_queue.cc:260] T 0f2ff50fa357448494434d1487ce83bc P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.979930 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0b7c860d457f421294ca1d37248e40b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:36.981776 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b7c860d457f421294ca1d37248e40b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:36.980397 24175 ts_tablet_manager.cc:1398] T 0f2ff50fa357448494434d1487ce83bc P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:36.982905 24175 tablet_bootstrap.cc:492] T 26c9913f1ca146fe84bc07ca1b4315c5 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:36.987613 24175 tablet_bootstrap.cc:654] T 26c9913f1ca146fe84bc07ca1b4315c5 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.988574 24179 log.cc:1196] T 774487be9bc644cab4e420c3d33b3cad P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/774487be9bc644cab4e420c3d33b3cad
I20240502 08:16:36.988958 24179 ts_tablet_manager.cc:1909] T 774487be9bc644cab4e420c3d33b3cad P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:36.990399 24179 tablet_replica.cc:330] T 71c0c2f3def643dfa99ff4efe65f6acf P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:36.990581 24181 ts_tablet_manager.cc:1875] T 26c9913f1ca146fe84bc07ca1b4315c5 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:36.990638 24179 raft_consensus.cc:2229] T 71c0c2f3def643dfa99ff4efe65f6acf P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:36.991034 24179 raft_consensus.cc:2258] T 71c0c2f3def643dfa99ff4efe65f6acf P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:36.992034 24179 ts_tablet_manager.cc:1875] T 71c0c2f3def643dfa99ff4efe65f6acf P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:36.975636 24174 tablet_bootstrap.cc:492] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:36.990434 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1736b496354e2dadd844242d3867f8 (DEFAULT_TABLE table=default.Table_02f448a8a9844def8ecda8716fdc352e [id=79a03d22b7e34f76bb3242c640129a13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.993060 24174 ts_tablet_manager.cc:1367] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.077s user 0.003s sys 0.001s
I20240502 08:16:36.993475 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1736b496354e2dadd844242d3867f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.869640 24173 consensus_queue.cc:260] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.994139 24174 raft_consensus.cc:348] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.994386 24174 raft_consensus.cc:374] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:36.994504 24174 raft_consensus.cc:729] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:36.994846 24174 consensus_queue.cc:260] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:36.995460 24174 ts_tablet_manager.cc:1398] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:36.995875 24174 tablet_bootstrap.cc:492] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:36.997351 24174 tablet_bootstrap.cc:654] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:36.990855 23768 tablet_service.cc:1467] Processing CreateTablet for tablet b811b81ddf2b4dc9b5e5f80331bff294 (DEFAULT_TABLE table=default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.991189 23761 tablet_service.cc:1467] Processing CreateTablet for tablet e686db13bb7541c69ae88f0fb7e27061 (DEFAULT_TABLE table=default.Table_178527d31d7044fc98fa41886d651049 [id=6dc275aed42b42118a70099d3eae3c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.998481 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1736b496354e2dadd844242d3867f8 (DEFAULT_TABLE table=default.Table_02f448a8a9844def8ecda8716fdc352e [id=79a03d22b7e34f76bb3242c640129a13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:36.998869 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1736b496354e2dadd844242d3867f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:36.999681 24175 tablet_bootstrap.cc:492] T 26c9913f1ca146fe84bc07ca1b4315c5 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:36.999934 24175 ts_tablet_manager.cc:1367] T 26c9913f1ca146fe84bc07ca1b4315c5 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.003s
I20240502 08:16:37.000924 24175 raft_consensus.cc:348] T 26c9913f1ca146fe84bc07ca1b4315c5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.001231 24175 raft_consensus.cc:374] T 26c9913f1ca146fe84bc07ca1b4315c5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.001353 24175 raft_consensus.cc:729] T 26c9913f1ca146fe84bc07ca1b4315c5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.001721 24175 consensus_queue.cc:260] T 26c9913f1ca146fe84bc07ca1b4315c5 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.002599 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 774487be9bc644cab4e420c3d33b3cad (table default.Table_a0ca472d0e4748e6995f3b55e306e483 [id=70dab926ba6c4a1e8b4cf29853ba5755]) successfully deleted
I20240502 08:16:37.003057 24175 ts_tablet_manager.cc:1398] T 26c9913f1ca146fe84bc07ca1b4315c5 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:37.003772 23896 tablet_service.cc:1467] Processing CreateTablet for tablet b811b81ddf2b4dc9b5e5f80331bff294 (DEFAULT_TABLE table=default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.004177 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b811b81ddf2b4dc9b5e5f80331bff294. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.003386 24175 tablet_bootstrap.cc:492] T 9628134656eb4be1b3e1419e5c1accbe P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:37.005371 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet c470d420c45843a69a63d9800b7cfc97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c470d420c45843a69a63d9800b7cfc97 already in progress: creating tablet
W20240502 08:16:37.005787 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet ba3d4b4584dd45b58ff0daa4f1c09a0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba3d4b4584dd45b58ff0daa4f1c09a0a already in progress: creating tablet
W20240502 08:16:37.006076 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e0af5f4daf3149738647a08219efb245 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0af5f4daf3149738647a08219efb245 already in progress: creating tablet
I20240502 08:16:37.006516 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b811b81ddf2b4dc9b5e5f80331bff294. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.008191 24173 ts_tablet_manager.cc:1398] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.126s user 0.002s sys 0.000s
I20240502 08:16:37.008606 24173 tablet_bootstrap.cc:492] T 223c92ae8f63402f8643e03d914e8e7e P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:36.992735 23758 tablet_service.cc:1467] Processing CreateTablet for tablet f4b291bdb81a4efbac7f6691e5577bbe (DEFAULT_TABLE table=default.Table_295f41167e6e481ea2443d5f8d02a998 [id=73559755c4594bcfa591d054964154ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.009749 24033 tablet_service.cc:1467] Processing CreateTablet for tablet e686db13bb7541c69ae88f0fb7e27061 (DEFAULT_TABLE table=default.Table_178527d31d7044fc98fa41886d651049 [id=6dc275aed42b42118a70099d3eae3c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.009958 23899 tablet_service.cc:1467] Processing CreateTablet for tablet e686db13bb7541c69ae88f0fb7e27061 (DEFAULT_TABLE table=default.Table_178527d31d7044fc98fa41886d651049 [id=6dc275aed42b42118a70099d3eae3c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.010124 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e686db13bb7541c69ae88f0fb7e27061. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.011469 24030 tablet_service.cc:1467] Processing CreateTablet for tablet b811b81ddf2b4dc9b5e5f80331bff294 (DEFAULT_TABLE table=default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.011839 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b811b81ddf2b4dc9b5e5f80331bff294. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.013774 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e686db13bb7541c69ae88f0fb7e27061. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:37.016762 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5453cc0b93ac4cbb90a0588bb2100d60 consensus state has no leader
I20240502 08:16:37.016983 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1736b496354e2dadd844242d3867f8 (DEFAULT_TABLE table=default.Table_02f448a8a9844def8ecda8716fdc352e [id=79a03d22b7e34f76bb3242c640129a13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.017354 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1736b496354e2dadd844242d3867f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.017678 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e686db13bb7541c69ae88f0fb7e27061. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.018872 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f7518b1e36147a9b9f0c58bca17061a with delete_type TABLET_DATA_DELETED (replaced by tablet d8f7b5f018d84680ac5533bf99102d31) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.020797 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b291bdb81a4efbac7f6691e5577bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.022689 24173 tablet_bootstrap.cc:654] T 223c92ae8f63402f8643e03d914e8e7e P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.029639 24180 log.cc:1196] T 32150c7e130f4069a2a3a02e625bfb93 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/32150c7e130f4069a2a3a02e625bfb93
I20240502 08:16:37.031181 24181 ts_tablet_manager.cc:1888] T 26c9913f1ca146fe84bc07ca1b4315c5 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:37.031322 24181 log.cc:1196] T 26c9913f1ca146fe84bc07ca1b4315c5 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/26c9913f1ca146fe84bc07ca1b4315c5
I20240502 08:16:37.031713 24181 ts_tablet_manager.cc:1909] T 26c9913f1ca146fe84bc07ca1b4315c5 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:37.035694 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c9913f1ca146fe84bc07ca1b4315c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:37.036567 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43e870c3a0e340d1bad1c1fa0380a762 consensus state has no leader
I20240502 08:16:37.045634 24179 ts_tablet_manager.cc:1888] T 71c0c2f3def643dfa99ff4efe65f6acf P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:37.045861 24179 log.cc:1196] T 71c0c2f3def643dfa99ff4efe65f6acf P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/71c0c2f3def643dfa99ff4efe65f6acf
I20240502 08:16:37.047463 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_62b86da21eb84bc387b64725dabda733" } modify_external_catalogs: true
I20240502 08:16:37.047650 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_62b86da21eb84bc387b64725dabda733" } modify_external_catalogs: true
I20240502 08:16:37.048563 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f2ff50fa357448494434d1487ce83bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:37.049000 24175 tablet_bootstrap.cc:654] T 9628134656eb4be1b3e1419e5c1accbe P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.050190 24179 ts_tablet_manager.cc:1909] T 71c0c2f3def643dfa99ff4efe65f6acf P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:37.050805 23606 master_service.cc:946] ReplaceTablet: received request to replace tablet e0af5f4daf3149738647a08219efb245 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:37.050966 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5782cc56973c43cfac93b2560c62c688"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:37.051492 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5782cc56973c43cfac93b2560c62c688 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:37.052003 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_178527d31d7044fc98fa41886d651049" } modify_external_catalogs: true
I20240502 08:16:37.052175 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_178527d31d7044fc98fa41886d651049" } modify_external_catalogs: true
I20240502 08:16:37.052737 24179 tablet_replica.cc:330] T 26c9913f1ca146fe84bc07ca1b4315c5 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:37.053035 24179 raft_consensus.cc:2229] T 26c9913f1ca146fe84bc07ca1b4315c5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.053244 24179 raft_consensus.cc:2258] T 26c9913f1ca146fe84bc07ca1b4315c5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:37.053437 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_006d60eaae72439f829df411c32c1514" } modify_external_catalogs: true
I20240502 08:16:37.053596 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_006d60eaae72439f829df411c32c1514" } modify_external_catalogs: true
I20240502 08:16:37.059813 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 71c0c2f3def643dfa99ff4efe65f6acf (table default.Table_68bf91a14ffa42cdbb10a1d7ca56cb08 [id=59d945a73daa4918ba17063eae0b9b93]) successfully deleted
W20240502 08:16:37.060092 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 171406810543435e832a47ad5b3c80a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 171406810543435e832a47ad5b3c80a4 already in progress: creating tablet
I20240502 08:16:37.064378 24179 ts_tablet_manager.cc:1875] T 26c9913f1ca146fe84bc07ca1b4315c5 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.066078 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ce1b3e3032d84f72b573bf1049c30237" } modify_external_catalogs: true
I20240502 08:16:37.066229 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ce1b3e3032d84f72b573bf1049c30237" } modify_external_catalogs: true
I20240502 08:16:37.067693 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 26c9913f1ca146fe84bc07ca1b4315c5 (table default.Table_f8286fd5bd9f4a1ca39afc1e8b57eae2 [id=93515920607e433ca1e2b5573176dc0c]) successfully deleted
I20240502 08:16:37.070508 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet 8633864e248f416cb651da3b0c2aa936 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:37.073190 24179 ts_tablet_manager.cc:1888] T 26c9913f1ca146fe84bc07ca1b4315c5 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240502 08:16:37.077010 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8633864e248f416cb651da3b0c2aa936 consensus state has no leader
I20240502 08:16:37.078043 24181 tablet_replica.cc:330] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:37.078557 24181 raft_consensus.cc:2229] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.079073 24181 raft_consensus.cc:2258] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:37.079855 24181 ts_tablet_manager.cc:1875] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.082697 24173 tablet_bootstrap.cc:492] T 223c92ae8f63402f8643e03d914e8e7e P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.082967 24173 ts_tablet_manager.cc:1367] T 223c92ae8f63402f8643e03d914e8e7e P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.005s
I20240502 08:16:37.083851 24173 raft_consensus.cc:348] T 223c92ae8f63402f8643e03d914e8e7e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.084179 24173 raft_consensus.cc:374] T 223c92ae8f63402f8643e03d914e8e7e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.084307 24173 raft_consensus.cc:729] T 223c92ae8f63402f8643e03d914e8e7e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.084656 24173 consensus_queue.cc:260] T 223c92ae8f63402f8643e03d914e8e7e P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.086067 24173 ts_tablet_manager.cc:1398] T 223c92ae8f63402f8643e03d914e8e7e P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:16:37.087299 24173 tablet_bootstrap.cc:492] T 9b1eb762780c450bbaf380d17db564b6 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:37.088516 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet 223c92ae8f63402f8643e03d914e8e7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.098183 24173 tablet_bootstrap.cc:654] T 9b1eb762780c450bbaf380d17db564b6 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.100086 24173 tablet_bootstrap.cc:492] T 9b1eb762780c450bbaf380d17db564b6 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.100277 24173 ts_tablet_manager.cc:1367] T 9b1eb762780c450bbaf380d17db564b6 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 08:16:37.101154 24173 raft_consensus.cc:348] T 9b1eb762780c450bbaf380d17db564b6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.101433 24173 raft_consensus.cc:374] T 9b1eb762780c450bbaf380d17db564b6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.101524 24173 raft_consensus.cc:729] T 9b1eb762780c450bbaf380d17db564b6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.101776 24173 consensus_queue.cc:260] T 9b1eb762780c450bbaf380d17db564b6 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.104694 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet ec4028a8e66442bfabac9e587aed00bb
I20240502 08:16:37.109107 24173 ts_tablet_manager.cc:1398] T 9b1eb762780c450bbaf380d17db564b6 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.009s user 0.000s sys 0.003s
I20240502 08:16:37.110090 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet ec4028a8e66442bfabac9e587aed00bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.110308 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b1eb762780c450bbaf380d17db564b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.114398 24173 tablet_bootstrap.cc:492] T 485c772245fe48a39fc6ceb6377ac0b6 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:37.115907 24181 ts_tablet_manager.cc:1888] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:37.116009 24173 tablet_bootstrap.cc:654] T 485c772245fe48a39fc6ceb6377ac0b6 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.116065 24181 log.cc:1196] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/e17eec8f2c1a4190945b434794635b52
I20240502 08:16:37.116444 24181 ts_tablet_manager.cc:1909] T e17eec8f2c1a4190945b434794635b52 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:37.118925 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e686db13bb7541c69ae88f0fb7e27061
I20240502 08:16:37.114060 24175 tablet_bootstrap.cc:492] T 9628134656eb4be1b3e1419e5c1accbe P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:37.119431 24179 log.cc:1196] T 26c9913f1ca146fe84bc07ca1b4315c5 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/26c9913f1ca146fe84bc07ca1b4315c5
I20240502 08:16:37.119518 24175 ts_tablet_manager.cc:1367] T 9628134656eb4be1b3e1419e5c1accbe P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.116s user 0.015s sys 0.024s
I20240502 08:16:37.119583 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 8633864e248f416cb651da3b0c2aa936
I20240502 08:16:37.119854 24179 ts_tablet_manager.cc:1909] T 26c9913f1ca146fe84bc07ca1b4315c5 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:37.119994 23604 catalog_manager.cc:6554] ReplaceTablet: tablet 8633864e248f416cb651da3b0c2aa936 deleted and replaced by tablet 9761df0bbdee44af90135bffb8b5828f
I20240502 08:16:37.120725 24179 tablet_replica.cc:330] T 0f2ff50fa357448494434d1487ce83bc P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:37.121115 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 8810b6f61db14fed8b12d36fed413dc8
I20240502 08:16:37.122471 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 26c9913f1ca146fe84bc07ca1b4315c5 (table default.Table_f8286fd5bd9f4a1ca39afc1e8b57eae2 [id=93515920607e433ca1e2b5573176dc0c]) successfully deleted
I20240502 08:16:37.122671 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet 8633864e248f416cb651da3b0c2aa936 with delete_type TABLET_DATA_DELETED (replaced by tablet 9761df0bbdee44af90135bffb8b5828f) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.122922 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet e686db13bb7541c69ae88f0fb7e27061 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.123623 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 8810b6f61db14fed8b12d36fed413dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.124140 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6ccfa5bc466242b2a325aa1ce6c85e9c" } modify_external_catalogs: true
I20240502 08:16:37.124351 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6ccfa5bc466242b2a325aa1ce6c85e9c" } modify_external_catalogs: true
I20240502 08:16:37.126332 24173 tablet_bootstrap.cc:492] T 485c772245fe48a39fc6ceb6377ac0b6 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.126518 24173 ts_tablet_manager.cc:1367] T 485c772245fe48a39fc6ceb6377ac0b6 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240502 08:16:37.127382 24173 raft_consensus.cc:348] T 485c772245fe48a39fc6ceb6377ac0b6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.127619 24173 raft_consensus.cc:374] T 485c772245fe48a39fc6ceb6377ac0b6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.127737 24173 raft_consensus.cc:729] T 485c772245fe48a39fc6ceb6377ac0b6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.128085 24173 consensus_queue.cc:260] T 485c772245fe48a39fc6ceb6377ac0b6 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.131827 24173 ts_tablet_manager.cc:1398] T 485c772245fe48a39fc6ceb6377ac0b6 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:16:37.120443 24175 raft_consensus.cc:348] T 9628134656eb4be1b3e1419e5c1accbe P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.132531 24175 raft_consensus.cc:374] T 9628134656eb4be1b3e1419e5c1accbe P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.132638 24175 raft_consensus.cc:729] T 9628134656eb4be1b3e1419e5c1accbe P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.132859 24175 consensus_queue.cc:260] T 9628134656eb4be1b3e1419e5c1accbe P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.133360 24175 ts_tablet_manager.cc:1398] T 9628134656eb4be1b3e1419e5c1accbe P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 08:16:37.123428 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 8810b6f61db14fed8b12d36fed413dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:37.123502 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 8633864e248f416cb651da3b0c2aa936 with delete_type TABLET_DATA_DELETED (replaced by tablet 9761df0bbdee44af90135bffb8b5828f) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:37.123428 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet e686db13bb7541c69ae88f0fb7e27061 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:37.134184 24179 raft_consensus.cc:2229] T 0f2ff50fa357448494434d1487ce83bc P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.134363 24179 raft_consensus.cc:2258] T 0f2ff50fa357448494434d1487ce83bc P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:37.135149 24179 ts_tablet_manager.cc:1875] T 0f2ff50fa357448494434d1487ce83bc P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.136091 24173 tablet_bootstrap.cc:492] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:37.136214 24175 tablet_bootstrap.cc:492] T 05cbbf9614094afaa6f7cee0af597242 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:37.137722 24175 tablet_bootstrap.cc:654] T 05cbbf9614094afaa6f7cee0af597242 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.138631 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet 9628134656eb4be1b3e1419e5c1accbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:37.138976 24355 raft_consensus.cc:482] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.139132 24355 raft_consensus.cc:504] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.139850 24355 leader_election.cc:288] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:37.140117 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7aa03e73acd4a9da5574b9ba93415ae" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:37.141554 23996 leader_election.cc:341] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:37.142937 24173 tablet_bootstrap.cc:654] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.143612 24179 ts_tablet_manager.cc:1888] T 0f2ff50fa357448494434d1487ce83bc P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:37.144872 24179 log.cc:1196] T 0f2ff50fa357448494434d1487ce83bc P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/0f2ff50fa357448494434d1487ce83bc
I20240502 08:16:37.145004 24175 tablet_bootstrap.cc:492] T 05cbbf9614094afaa6f7cee0af597242 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:37.145335 24175 ts_tablet_manager.cc:1367] T 05cbbf9614094afaa6f7cee0af597242 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20240502 08:16:37.146394 24175 raft_consensus.cc:348] T 05cbbf9614094afaa6f7cee0af597242 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.146780 24175 raft_consensus.cc:374] T 05cbbf9614094afaa6f7cee0af597242 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.146942 24175 raft_consensus.cc:729] T 05cbbf9614094afaa6f7cee0af597242 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.147328 24175 consensus_queue.cc:260] T 05cbbf9614094afaa6f7cee0af597242 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.148075 24175 ts_tablet_manager.cc:1398] T 05cbbf9614094afaa6f7cee0af597242 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:16:37.148523 24175 tablet_bootstrap.cc:492] T c7aa03e73acd4a9da5574b9ba93415ae P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:37.149997 24175 tablet_bootstrap.cc:654] T c7aa03e73acd4a9da5574b9ba93415ae P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.152833 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet e17eec8f2c1a4190945b434794635b52 (table default.Table_b01c340249ea483683e385a7fa964a51 [id=ee0e1cf16cb34aa98699970438cc36ae]) successfully deleted
I20240502 08:16:37.153307 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet 05cbbf9614094afaa6f7cee0af597242 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:37.153528 24179 ts_tablet_manager.cc:1909] T 0f2ff50fa357448494434d1487ce83bc P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:37.154142 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 0f2ff50fa357448494434d1487ce83bc (table default.Table_065c3fa8292242d59b5fbebbfa233fdc [id=efa730295c3c4bb69e64ca3fa324abb1]) successfully deleted
I20240502 08:16:37.154278 24181 tablet_replica.cc:330] T 485c772245fe48a39fc6ceb6377ac0b6 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:37.154503 24181 raft_consensus.cc:2229] T 485c772245fe48a39fc6ceb6377ac0b6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.154560 24179 tablet_replica.cc:330] T 9628134656eb4be1b3e1419e5c1accbe P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:37.154680 24181 raft_consensus.cc:2258] T 485c772245fe48a39fc6ceb6377ac0b6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:37.154759 24179 raft_consensus.cc:2229] T 9628134656eb4be1b3e1419e5c1accbe P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.154938 24179 raft_consensus.cc:2258] T 9628134656eb4be1b3e1419e5c1accbe P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:37.155575 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 8810b6f61db14fed8b12d36fed413dc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8810b6f61db14fed8b12d36fed413dc8 already in progress: creating tablet
W20240502 08:16:37.155879 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 8633864e248f416cb651da3b0c2aa936 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8633864e248f416cb651da3b0c2aa936 already in progress: creating tablet
W20240502 08:16:37.156153 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e686db13bb7541c69ae88f0fb7e27061 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e686db13bb7541c69ae88f0fb7e27061 already in progress: creating tablet
I20240502 08:16:37.156836 24181 ts_tablet_manager.cc:1875] T 485c772245fe48a39fc6ceb6377ac0b6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.160205 23606 master_service.cc:946] ReplaceTablet: received request to replace tablet 9700ea9dc58442e3ab1b7835a8c1e22c from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:37.162874 24179 ts_tablet_manager.cc:1875] T 9628134656eb4be1b3e1419e5c1accbe P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.167325 23607 master_service.cc:946] ReplaceTablet: received request to replace tablet 485c772245fe48a39fc6ceb6377ac0b6 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:37.168306 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0542f22f24e14c5887f78b1456fa361b" } new_table_name: "default.Table_e66f5d1a72b347f7a7b39bddbf74b263" modify_external_catalogs: true
I20240502 08:16:37.169584 24181 ts_tablet_manager.cc:1888] T 485c772245fe48a39fc6ceb6377ac0b6 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:37.172248 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8b1436ec53e349658e5e39d3e10da3e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:37.172809 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b1436ec53e349658e5e39d3e10da3e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:37.175833 24179 ts_tablet_manager.cc:1888] T 9628134656eb4be1b3e1419e5c1accbe P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:37.177062 24181 log.cc:1196] T 485c772245fe48a39fc6ceb6377ac0b6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/485c772245fe48a39fc6ceb6377ac0b6
I20240502 08:16:37.177835 24181 ts_tablet_manager.cc:1909] T 485c772245fe48a39fc6ceb6377ac0b6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:37.179351 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 485c772245fe48a39fc6ceb6377ac0b6 (table default.Table_0528f5d019fe489db8bed1f71adac3cb [id=7dd2b25a6676488d8f3a8e2e6b65046e]) successfully deleted
I20240502 08:16:37.179432 24181 tablet_replica.cc:330] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:37.179630 24181 raft_consensus.cc:2229] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.179811 24181 raft_consensus.cc:2258] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:37.179888 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 10adabdd31f848548ef6f2408ccc2050 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10adabdd31f848548ef6f2408ccc2050 already in progress: creating tablet
I20240502 08:16:37.179836 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_02ea392fafea47a28cf6565e8e60dc5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:37.180372 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02ea392fafea47a28cf6565e8e60dc5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:37.180385 24181 ts_tablet_manager.cc:1875] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.182006 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a757855f2b244c548d8c2add77baaff7" } new_table_name: "default.Table_241ebfbdbdc94f27a8ff9035c037c15d" modify_external_catalogs: true
I20240502 08:16:37.181977 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7e12ddab0f5241b4ac31de16f1467292"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:37.182538 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e12ddab0f5241b4ac31de16f1467292 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:37.183223 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7d5158c57d7342388436ec0d390ab50c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:37.183692 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d5158c57d7342388436ec0d390ab50c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:37.186930 24179 log.cc:1196] T 9628134656eb4be1b3e1419e5c1accbe P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/9628134656eb4be1b3e1419e5c1accbe
I20240502 08:16:37.187151 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_cb7682ca6f0f4d93922aa35e427c9cfa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:37.187405 24179 ts_tablet_manager.cc:1909] T 9628134656eb4be1b3e1419e5c1accbe P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
W20240502 08:16:37.187584 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb7682ca6f0f4d93922aa35e427c9cfa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:37.188004 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 9628134656eb4be1b3e1419e5c1accbe (table default.Table_e077b6d407694d3eaa7f0c34e3f8b917 [id=9f2f530c5f91424d91ec92a42602b781]) successfully deleted
I20240502 08:16:37.128194 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet ec4028a8e66442bfabac9e587aed00bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:37.190517 24241 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:37.191530 24241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:37.191864 24241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:37.193702 24179 tablet_replica.cc:330] T 05cbbf9614094afaa6f7cee0af597242 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:37.194294 24179 raft_consensus.cc:2229] T 05cbbf9614094afaa6f7cee0af597242 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.194811 24179 raft_consensus.cc:2258] T 05cbbf9614094afaa6f7cee0af597242 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:37.196028 24179 ts_tablet_manager.cc:1875] T 05cbbf9614094afaa6f7cee0af597242 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:37.199409 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7a97427e819b4826b1ed89e6d021c8d4 has no consensus state
I20240502 08:16:37.206728 24181 ts_tablet_manager.cc:1888] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:37.207782 24181 log.cc:1196] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/175adf0ecb504e28a3d39f001f977839
I20240502 08:16:37.208449 24181 ts_tablet_manager.cc:1909] T 175adf0ecb504e28a3d39f001f977839 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:37.210008 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 175adf0ecb504e28a3d39f001f977839 (table default.Table_9530003aa5b841b683920da217d58576 [id=a6a5f4c65807479392c01ac5c955c959]) successfully deleted
W20240502 08:16:37.213047 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet d8f7b5f018d84680ac5533bf99102d31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8f7b5f018d84680ac5533bf99102d31 already in progress: creating tablet
W20240502 08:16:37.214248 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet f36ceb0222ef473a828ca795f393174d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f36ceb0222ef473a828ca795f393174d already in progress: creating tablet
I20240502 08:16:37.214910 24181 tablet_replica.cc:330] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:37.216311 24179 ts_tablet_manager.cc:1888] T 05cbbf9614094afaa6f7cee0af597242 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:37.216765 24181 raft_consensus.cc:2229] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.217283 24181 raft_consensus.cc:2258] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:37.218541 24181 ts_tablet_manager.cc:1875] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.219413 24179 log.cc:1196] T 05cbbf9614094afaa6f7cee0af597242 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/05cbbf9614094afaa6f7cee0af597242
I20240502 08:16:37.220835 24179 ts_tablet_manager.cc:1909] T 05cbbf9614094afaa6f7cee0af597242 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:37.221499 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet f36ceb0222ef473a828ca795f393174d from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:37.222105 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 05cbbf9614094afaa6f7cee0af597242 (table default.Table_6c583799e352452a87de6faa19d88490 [id=80701c77b9b04253bba1d4438e6f2a78]) successfully deleted
I20240502 08:16:37.222712 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4688a59d76ee4b5a931536f5deddfd4e" } new_table_name: "default.Table_0dc68fd335024f2dbce6b1fcc698f1be" modify_external_catalogs: true
I20240502 08:16:37.223858 24179 tablet_replica.cc:330] T ec4028a8e66442bfabac9e587aed00bb P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:37.224321 24179 raft_consensus.cc:2229] T ec4028a8e66442bfabac9e587aed00bb P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.225123 24179 raft_consensus.cc:2258] T ec4028a8e66442bfabac9e587aed00bb P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:37.230257 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 71aa916247c449fea718b35869a6df20 consensus state has no leader
I20240502 08:16:37.230507 24179 ts_tablet_manager.cc:1875] T ec4028a8e66442bfabac9e587aed00bb P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.231730 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e87bd15719d642bfbc0c9ab785ff62d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:37.232523 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e87bd15719d642bfbc0c9ab785ff62d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:37.238090 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7d5158c57d7342388436ec0d390ab50c" } new_table_name: "default.Table_2e0d27237b2f4eb89acb7eb245578f1f" modify_external_catalogs: true
I20240502 08:16:37.238325 24181 ts_tablet_manager.cc:1888] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:37.242848 24179 ts_tablet_manager.cc:1888] T ec4028a8e66442bfabac9e587aed00bb P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:16:37.244845 24181 log.cc:1196] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/32150c7e130f4069a2a3a02e625bfb93
I20240502 08:16:37.245543 24181 ts_tablet_manager.cc:1909] T 32150c7e130f4069a2a3a02e625bfb93 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:37.247632 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 32150c7e130f4069a2a3a02e625bfb93 (table default.Table_90cefac5d31b4c449140db4b7772a0d2 [id=a895b3f375374ad6bb44a6f87fc7b98d]) successfully deleted
I20240502 08:16:37.247818 24180 ts_tablet_manager.cc:1909] T 32150c7e130f4069a2a3a02e625bfb93 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
W20240502 08:16:37.249640 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 6788ffd2eb2248c3b84e474de56c37a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6788ffd2eb2248c3b84e474de56c37a9 already in progress: creating tablet
W20240502 08:16:37.250471 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 01675f637a5a4645949abfd87b3d09a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01675f637a5a4645949abfd87b3d09a2 already in progress: creating tablet
W20240502 08:16:37.251304 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 2bfb532c9d4c41afb7acaa4831841306 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2bfb532c9d4c41afb7acaa4831841306 already in progress: creating tablet
I20240502 08:16:37.251783 24179 log.cc:1196] T ec4028a8e66442bfabac9e587aed00bb P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/ec4028a8e66442bfabac9e587aed00bb
W20240502 08:16:37.252326 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 6bb2d0a4898544f182ab8070d29a7940 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bb2d0a4898544f182ab8070d29a7940 already in progress: creating tablet
I20240502 08:16:37.253089 24179 ts_tablet_manager.cc:1909] T ec4028a8e66442bfabac9e587aed00bb P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:37.253197 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet e686db13bb7541c69ae88f0fb7e27061 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:37.253422 24181 tablet_replica.cc:330] T ab70513985b54416856a29b32cecae4d P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:37.253492 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet ec4028a8e66442bfabac9e587aed00bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:37.253722 24329 raft_consensus.cc:482] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.253898 24181 raft_consensus.cc:2229] T ab70513985b54416856a29b32cecae4d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.253885 24329 raft_consensus.cc:504] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.254086 24181 raft_consensus.cc:2258] T ab70513985b54416856a29b32cecae4d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:37.254734 24181 ts_tablet_manager.cc:1875] T ab70513985b54416856a29b32cecae4d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.255749 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c316e1aeab224634b760641a4ce188c8" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:37.255779 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c316e1aeab224634b760641a4ce188c8" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:37.258721 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet ec4028a8e66442bfabac9e587aed00bb (table default.Table_62b86da21eb84bc387b64725dabda733 [id=1e590d80e3e34e1aac8a0c7908a9d813]) successfully deleted
I20240502 08:16:37.259884 24329 leader_election.cc:288] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:37.261267 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet 8633864e248f416cb651da3b0c2aa936 with delete_type TABLET_DATA_DELETED (replaced by tablet 9761df0bbdee44af90135bffb8b5828f) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:37.261638 24343 raft_consensus.cc:482] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.261806 24343 raft_consensus.cc:504] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.262755 24343 leader_election.cc:288] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:37.263285 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7aa03e73acd4a9da5574b9ba93415ae" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:37.263523 23787 raft_consensus.cc:2454] T c7aa03e73acd4a9da5574b9ba93415ae P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:37.264140 23993 leader_election.cc:302] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:37.264546 24355 raft_consensus.cc:2781] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:37.264678 24355 raft_consensus.cc:482] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:37.264776 24355 raft_consensus.cc:3037] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:37.265460 23723 leader_election.cc:341] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:37.265951 23723 leader_election.cc:341] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:37.266734 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab70513985b54416856a29b32cecae4d" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:37.267758 23723 leader_election.cc:302] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:37.268088 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab70513985b54416856a29b32cecae4d" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:37.268316 23723 leader_election.cc:341] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: Tablet not RUNNING: STOPPED
I20240502 08:16:37.268654 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet 8810b6f61db14fed8b12d36fed413dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:37.267637 24355 raft_consensus.cc:504] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.269414 24355 leader_election.cc:288] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
W20240502 08:16:37.269722 23723 leader_election.cc:341] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:37.269732 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7aa03e73acd4a9da5574b9ba93415ae" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:37.269829 23723 leader_election.cc:302] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
W20240502 08:16:37.270176 23996 leader_election.cc:341] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:37.271106 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7aa03e73acd4a9da5574b9ba93415ae" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:37.271319 23787 raft_consensus.cc:3037] T c7aa03e73acd4a9da5574b9ba93415ae P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.273506 24180 tablet_replica.cc:330] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:37.273792 24180 raft_consensus.cc:2229] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.273912 24181 ts_tablet_manager.cc:1888] T ab70513985b54416856a29b32cecae4d P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:37.273998 24180 raft_consensus.cc:2258] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:37.274034 24181 log.cc:1196] T ab70513985b54416856a29b32cecae4d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/ab70513985b54416856a29b32cecae4d
I20240502 08:16:37.274461 24181 ts_tablet_manager.cc:1909] T ab70513985b54416856a29b32cecae4d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:37.274698 24180 ts_tablet_manager.cc:1875] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.275671 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet ab70513985b54416856a29b32cecae4d (table default.Table_d9b6d863d79341ebb994770c34ae2174 [id=259cb14e2ef6427f8dfba46f188050dd]) successfully deleted
I20240502 08:16:37.276757 24343 raft_consensus.cc:2726] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:37.277070 24329 raft_consensus.cc:2726] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:37.277678 24355 raft_consensus.cc:482] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.277817 24355 raft_consensus.cc:504] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.278391 24355 leader_election.cc:288] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:37.278797 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f7518b1e36147a9b9f0c58bca17061a" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:37.279137 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f7518b1e36147a9b9f0c58bca17061a" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:37.279269 23996 leader_election.cc:341] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:37.279642 23993 leader_election.cc:341] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:37.279639 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7b904b0af0a8449581e717c1a56c01ce" } new_table_name: "default.Table_9741459a96e649fb8151cc28b642523e" modify_external_catalogs: true
I20240502 08:16:37.279754 23993 leader_election.cc:302] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:37.280337 24181 tablet_replica.cc:330] T a15ff3120b1943a69ddb609d0575ae1a P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:37.280903 24181 raft_consensus.cc:2229] T a15ff3120b1943a69ddb609d0575ae1a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.281455 24181 raft_consensus.cc:2258] T a15ff3120b1943a69ddb609d0575ae1a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:37.281950 24355 raft_consensus.cc:2726] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:37.283191 24181 ts_tablet_manager.cc:1875] T a15ff3120b1943a69ddb609d0575ae1a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.285859 23787 raft_consensus.cc:2454] T c7aa03e73acd4a9da5574b9ba93415ae P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:37.286604 23993 leader_election.cc:302] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:37.289709 24355 raft_consensus.cc:2781] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:37.296574 24181 ts_tablet_manager.cc:1888] T a15ff3120b1943a69ddb609d0575ae1a P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:37.299031 24181 log.cc:1196] T a15ff3120b1943a69ddb609d0575ae1a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/a15ff3120b1943a69ddb609d0575ae1a
I20240502 08:16:37.298935 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_97709af0716042cebdc01ed2d06dcf20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:37.299645 24181 ts_tablet_manager.cc:1909] T a15ff3120b1943a69ddb609d0575ae1a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
W20240502 08:16:37.299665 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97709af0716042cebdc01ed2d06dcf20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:37.300815 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet a15ff3120b1943a69ddb609d0575ae1a (table default.Table_bc6b3019e44547cda285ac0021565619 [id=c08bf22592064123bb9cab28935d81a5]) successfully deleted
I20240502 08:16:37.301311 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_df744f06850c416f98d94d2df132a03b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:37.301888 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df744f06850c416f98d94d2df132a03b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:37.302815 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e0af5f4daf3149738647a08219efb245 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0af5f4daf3149738647a08219efb245 already in progress: creating tablet
W20240502 08:16:37.304661 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet c470d420c45843a69a63d9800b7cfc97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c470d420c45843a69a63d9800b7cfc97 already in progress: creating tablet
W20240502 08:16:37.306238 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 171406810543435e832a47ad5b3c80a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 171406810543435e832a47ad5b3c80a4 already in progress: creating tablet
W20240502 08:16:37.307291 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet ba3d4b4584dd45b58ff0daa4f1c09a0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba3d4b4584dd45b58ff0daa4f1c09a0a already in progress: creating tablet
I20240502 08:16:37.307757 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 32150c7e130f4069a2a3a02e625bfb93 (table default.Table_90cefac5d31b4c449140db4b7772a0d2 [id=a895b3f375374ad6bb44a6f87fc7b98d]) successfully deleted
I20240502 08:16:37.308106 24181 tablet_replica.cc:330] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:37.308519 24181 raft_consensus.cc:2229] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.309070 24181 raft_consensus.cc:2258] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:37.310308 24181 ts_tablet_manager.cc:1875] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.313735 24180 ts_tablet_manager.cc:1888] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:37.316927 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a89a2cdb74684df5bcbe15389816c50b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:37.317965 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a89a2cdb74684df5bcbe15389816c50b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:37.320096 23434 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 102454us
I20240502 08:16:37.326934 24181 ts_tablet_manager.cc:1888] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:37.327520 24181 log.cc:1196] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/6f7518b1e36147a9b9f0c58bca17061a
I20240502 08:16:37.328253 24181 ts_tablet_manager.cc:1909] T 6f7518b1e36147a9b9f0c58bca17061a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:37.330161 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 6f7518b1e36147a9b9f0c58bca17061a (table default.Table_498b372c15a04b70b87bd67b543501cf [id=38e955d6bd644d24a7c45172ab60282b]) successfully deleted
I20240502 08:16:37.331652 24181 tablet_replica.cc:330] T 223c92ae8f63402f8643e03d914e8e7e P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:37.332191 24181 raft_consensus.cc:2229] T 223c92ae8f63402f8643e03d914e8e7e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.332716 24181 raft_consensus.cc:2258] T 223c92ae8f63402f8643e03d914e8e7e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:37.334051 24181 ts_tablet_manager.cc:1875] T 223c92ae8f63402f8643e03d914e8e7e P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.338919 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b461b560f3094407a1766253cfc2ece7" } modify_external_catalogs: true
I20240502 08:16:37.339244 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b461b560f3094407a1766253cfc2ece7" } modify_external_catalogs: true
I20240502 08:16:37.346621 24181 ts_tablet_manager.cc:1888] T 223c92ae8f63402f8643e03d914e8e7e P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:37.349781 24183 raft_consensus.cc:482] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.349997 24183 raft_consensus.cc:504] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.350909 24183 leader_election.cc:288] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:37.351869 24181 log.cc:1196] T 223c92ae8f63402f8643e03d914e8e7e P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/223c92ae8f63402f8643e03d914e8e7e
I20240502 08:16:37.351817 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768167b977c14b53a7537879e349d354" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:37.352417 24059 raft_consensus.cc:2379] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:16:37.352604 24181 ts_tablet_manager.cc:1909] T 223c92ae8f63402f8643e03d914e8e7e P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:37.355199 24181 tablet_replica.cc:330] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:37.355672 24181 raft_consensus.cc:2229] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:37.356374 24181 raft_consensus.cc:2258] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:37.357411 24181 ts_tablet_manager.cc:1875] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.363808 24181 ts_tablet_manager.cc:1888] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:16:37.369899 24181 log.cc:1196] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/ec4028a8e66442bfabac9e587aed00bb
I20240502 08:16:37.370533 24181 ts_tablet_manager.cc:1909] T ec4028a8e66442bfabac9e587aed00bb P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:37.354218 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 223c92ae8f63402f8643e03d914e8e7e (table default.Table_00eb74cd24954c52b5484751f035f4c1 [id=b42e15a2350f4c9a8097ccb0cf1aaddc]) successfully deleted
I20240502 08:16:37.372108 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet ec4028a8e66442bfabac9e587aed00bb (table default.Table_62b86da21eb84bc387b64725dabda733 [id=1e590d80e3e34e1aac8a0c7908a9d813]) successfully deleted
I20240502 08:16:37.373385 24181 tablet_replica.cc:330] T 9b1eb762780c450bbaf380d17db564b6 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:37.374017 24181 raft_consensus.cc:2229] T 9b1eb762780c450bbaf380d17db564b6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.374608 24181 raft_consensus.cc:2258] T 9b1eb762780c450bbaf380d17db564b6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:37.376052 24181 ts_tablet_manager.cc:1875] T 9b1eb762780c450bbaf380d17db564b6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.378126 24355 raft_consensus.cc:686] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:37.378463 24355 consensus_queue.cc:237] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.315547 24174 tablet_bootstrap.cc:492] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:37.351401 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "768167b977c14b53a7537879e349d354" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:37.379578 24180 log.cc:1196] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/405ea360814e4593a289d753821277ff
I20240502 08:16:37.379653 24174 ts_tablet_manager.cc:1367] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.384s user 0.072s sys 0.096s
I20240502 08:16:37.380069 24180 ts_tablet_manager.cc:1909] T 405ea360814e4593a289d753821277ff P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:37.380188 23858 leader_election.cc:302] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:37.380504 24183 raft_consensus.cc:3037] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.380542 24174 raft_consensus.cc:348] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.380728 24174 raft_consensus.cc:374] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.380831 24174 raft_consensus.cc:729] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.381078 24174 consensus_queue.cc:260] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.381690 24174 ts_tablet_manager.cc:1398] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:37.382030 24174 tablet_bootstrap.cc:492] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:37.383287 24183 raft_consensus.cc:2726] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:37.384212 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 405ea360814e4593a289d753821277ff (table default.Table_3602f20179c64041a63ab185d9359de6 [id=6d4996e259e34d51b7ee13c8efa7e2bd]) successfully deleted
I20240502 08:16:37.385458 24174 tablet_bootstrap.cc:654] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.408218 24175 tablet_bootstrap.cc:492] T c7aa03e73acd4a9da5574b9ba93415ae P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:37.408427 24175 ts_tablet_manager.cc:1367] T c7aa03e73acd4a9da5574b9ba93415ae P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.260s user 0.040s sys 0.080s
I20240502 08:16:37.408584 24181 ts_tablet_manager.cc:1888] T 9b1eb762780c450bbaf380d17db564b6 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:37.408924 24181 log.cc:1196] T 9b1eb762780c450bbaf380d17db564b6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/9b1eb762780c450bbaf380d17db564b6
I20240502 08:16:37.409646 24173 tablet_bootstrap.cc:492] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.409832 24173 ts_tablet_manager.cc:1367] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.274s user 0.037s sys 0.055s
I20240502 08:16:37.410243 24174 tablet_bootstrap.cc:492] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:37.410421 24174 ts_tablet_manager.cc:1367] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.002s
I20240502 08:16:37.410789 24173 raft_consensus.cc:348] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.411072 24173 raft_consensus.cc:374] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.411206 24173 raft_consensus.cc:729] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.411180 24174 raft_consensus.cc:348] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.411401 24174 raft_consensus.cc:374] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.411513 24174 raft_consensus.cc:729] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.411700 24173 consensus_queue.cc:260] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.411754 24174 consensus_queue.cc:260] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.412338 24173 ts_tablet_manager.cc:1398] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:37.412350 24174 ts_tablet_manager.cc:1398] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:37.412707 24173 tablet_bootstrap.cc:492] T c316e1aeab224634b760641a4ce188c8 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:37.415706 24173 tablet_bootstrap.cc:654] T c316e1aeab224634b760641a4ce188c8 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.417594 24181 ts_tablet_manager.cc:1909] T 9b1eb762780c450bbaf380d17db564b6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:37.418756 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 9b1eb762780c450bbaf380d17db564b6 (table default.Table_4dd22967bcf147388ea4ceb9cd24b065 [id=ba56850714c2422eb5dbbc0f49df7ce7]) successfully deleted
W20240502 08:16:37.419914 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 8633864e248f416cb651da3b0c2aa936 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8633864e248f416cb651da3b0c2aa936 already in progress: creating tablet
W20240502 08:16:37.420738 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e686db13bb7541c69ae88f0fb7e27061 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e686db13bb7541c69ae88f0fb7e27061 already in progress: creating tablet
W20240502 08:16:37.421422 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 8810b6f61db14fed8b12d36fed413dc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8810b6f61db14fed8b12d36fed413dc8 already in progress: creating tablet
I20240502 08:16:37.421648 24174 tablet_bootstrap.cc:492] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:37.423648 24174 tablet_bootstrap.cc:654] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.424412 24173 tablet_bootstrap.cc:492] T c316e1aeab224634b760641a4ce188c8 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.424679 24173 ts_tablet_manager.cc:1367] T c316e1aeab224634b760641a4ce188c8 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20240502 08:16:37.425288 24175 raft_consensus.cc:348] T c7aa03e73acd4a9da5574b9ba93415ae P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.425535 24175 raft_consensus.cc:374] T c7aa03e73acd4a9da5574b9ba93415ae P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.425680 24175 raft_consensus.cc:729] T c7aa03e73acd4a9da5574b9ba93415ae P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.425680 24173 raft_consensus.cc:348] T c316e1aeab224634b760641a4ce188c8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.426061 24173 raft_consensus.cc:374] T c316e1aeab224634b760641a4ce188c8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.426175 24173 raft_consensus.cc:729] T c316e1aeab224634b760641a4ce188c8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.426501 24173 consensus_queue.cc:260] T c316e1aeab224634b760641a4ce188c8 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.425937 24175 consensus_queue.cc:260] T c7aa03e73acd4a9da5574b9ba93415ae P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.427706 24175 ts_tablet_manager.cc:1398] T c7aa03e73acd4a9da5574b9ba93415ae P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20240502 08:16:37.428135 24175 tablet_bootstrap.cc:492] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:37.429352 24173 ts_tablet_manager.cc:1398] T c316e1aeab224634b760641a4ce188c8 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:16:37.430539 24173 tablet_bootstrap.cc:492] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:37.431454 24180 tablet_replica.cc:330] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
W20240502 08:16:37.431795 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9700ea9dc58442e3ab1b7835a8c1e22c consensus state has no leader
I20240502 08:16:37.432044 24175 tablet_bootstrap.cc:654] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.432386 24173 tablet_bootstrap.cc:654] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.435552 24174 tablet_bootstrap.cc:492] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
W20240502 08:16:37.435550 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 128c88af54d24ffd895e9de19cdb4c8a has no consensus state
I20240502 08:16:37.436746 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet c08228fcd1304528b9446003851f6c70 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:37.437405 24173 tablet_bootstrap.cc:492] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.437549 24173 ts_tablet_manager.cc:1367] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 08:16:37.438346 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_78c0d4912be14173a79ee9f2cab2288b" } modify_external_catalogs: true
I20240502 08:16:37.438321 24173 raft_consensus.cc:348] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.438618 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_78c0d4912be14173a79ee9f2cab2288b" } modify_external_catalogs: true
I20240502 08:16:37.438505 24173 raft_consensus.cc:374] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.438941 24173 raft_consensus.cc:729] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.439203 24173 consensus_queue.cc:260] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.439826 24174 ts_tablet_manager.cc:1367] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240502 08:16:37.440554 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_3def35abf8ad4ff49c5ae77acb410e01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:37.441118 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3def35abf8ad4ff49c5ae77acb410e01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:37.442348 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cb0ea06f5e0b47e5978fc9a5ed8880ce" } modify_external_catalogs: true
I20240502 08:16:37.442579 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cb0ea06f5e0b47e5978fc9a5ed8880ce" } modify_external_catalogs: true
I20240502 08:16:37.443125 24174 raft_consensus.cc:348] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.443428 24174 raft_consensus.cc:374] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.443562 24174 raft_consensus.cc:729] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.443843 24174 consensus_queue.cc:260] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.431742 24180 raft_consensus.cc:2229] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.445367 24180 raft_consensus.cc:2258] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:37.446059 24180 ts_tablet_manager.cc:1875] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.446974 24175 tablet_bootstrap.cc:492] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:37.447172 24175 ts_tablet_manager.cc:1367] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.001s
I20240502 08:16:37.447172 24173 ts_tablet_manager.cc:1398] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240502 08:16:37.448457 24173 tablet_bootstrap.cc:492] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:37.449995 24173 tablet_bootstrap.cc:654] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.449923 24175 raft_consensus.cc:348] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:37.450531 23434 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 89537.1us
I20240502 08:16:37.456784 24173 tablet_bootstrap.cc:492] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.456934 24173 ts_tablet_manager.cc:1367] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 08:16:37.457615 24173 raft_consensus.cc:348] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.457818 24173 raft_consensus.cc:374] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.457885 24173 raft_consensus.cc:729] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.458077 24175 raft_consensus.cc:374] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.458143 24173 consensus_queue.cc:260] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.458309 24175 raft_consensus.cc:729] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.458619 24173 ts_tablet_manager.cc:1398] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:37.458577 24175 consensus_queue.cc:260] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.460165 24173 tablet_bootstrap.cc:492] T d8324e9b796047428e2fe918f07f9b32 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:37.460520 24175 ts_tablet_manager.cc:1398] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240502 08:16:37.460839 24175 tablet_bootstrap.cc:492] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:37.462363 24175 tablet_bootstrap.cc:654] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.462368 24173 tablet_bootstrap.cc:654] T d8324e9b796047428e2fe918f07f9b32 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.469507 24417 tablet.cc:1722] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:37.448401 24174 ts_tablet_manager.cc:1398] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:16:37.451951 24343 raft_consensus.cc:482] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.470695 24174 tablet_bootstrap.cc:492] T a15ff3120b1943a69ddb609d0575ae1a P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:37.470568 24343 raft_consensus.cc:504] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.470629 24410 tablet.cc:1722] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:37.471271 24343 leader_election.cc:288] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:37.455504 24408 tablet.cc:1722] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:37.472440 24174 tablet_bootstrap.cc:654] T a15ff3120b1943a69ddb609d0575ae1a P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.482128 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160fda2643e4b80bc401bc48f97eff9" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:37.482789 23723 leader_election.cc:341] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:37.485751 24173 tablet_bootstrap.cc:492] T d8324e9b796047428e2fe918f07f9b32 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.485958 24173 ts_tablet_manager.cc:1367] T d8324e9b796047428e2fe918f07f9b32 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.003s
I20240502 08:16:37.486670 24173 raft_consensus.cc:348] T d8324e9b796047428e2fe918f07f9b32 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.486922 24173 raft_consensus.cc:374] T d8324e9b796047428e2fe918f07f9b32 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.487046 24173 raft_consensus.cc:729] T d8324e9b796047428e2fe918f07f9b32 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.486954 23610 catalog_manager.cc:5484] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67). New cstate: current_term: 1 leader_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:37.487319 24173 consensus_queue.cc:260] T d8324e9b796047428e2fe918f07f9b32 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.487939 24173 ts_tablet_manager.cc:1398] T d8324e9b796047428e2fe918f07f9b32 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:37.488036 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 4a57ed188315493198e95cefb6907ab6 (DEFAULT_TABLE table=default.Table_8c4dfc5c68e84b6ab815e9f333143678 [id=58ae4f5729f44a9c8be960be8a1a13a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.488232 24173 tablet_bootstrap.cc:492] T 5160fda2643e4b80bc401bc48f97eff9 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:37.488312 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 77c49da1e0f0435ba241a6a66526e790 (DEFAULT_TABLE table=default.Table_1f6435dfbba64b358601dea132f0e1f7 [id=0de18f9c7e194e698b4f37603bfd4780]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.489909 24173 tablet_bootstrap.cc:654] T 5160fda2643e4b80bc401bc48f97eff9 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.489867 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 4a57ed188315493198e95cefb6907ab6 (DEFAULT_TABLE table=default.Table_8c4dfc5c68e84b6ab815e9f333143678 [id=58ae4f5729f44a9c8be960be8a1a13a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.490255 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 4a57ed188315493198e95cefb6907ab6 (DEFAULT_TABLE table=default.Table_8c4dfc5c68e84b6ab815e9f333143678 [id=58ae4f5729f44a9c8be960be8a1a13a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.490661 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a57ed188315493198e95cefb6907ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.490890 24174 tablet_bootstrap.cc:492] T a15ff3120b1943a69ddb609d0575ae1a P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:37.491083 24174 ts_tablet_manager.cc:1367] T a15ff3120b1943a69ddb609d0575ae1a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.003s
I20240502 08:16:37.491957 24174 raft_consensus.cc:348] T a15ff3120b1943a69ddb609d0575ae1a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.492211 24174 raft_consensus.cc:374] T a15ff3120b1943a69ddb609d0575ae1a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.492300 24174 raft_consensus.cc:729] T a15ff3120b1943a69ddb609d0575ae1a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.492621 24174 consensus_queue.cc:260] T a15ff3120b1943a69ddb609d0575ae1a P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.494259 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77c49da1e0f0435ba241a6a66526e790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.494765 24174 ts_tablet_manager.cc:1398] T a15ff3120b1943a69ddb609d0575ae1a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:16:37.494830 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a57ed188315493198e95cefb6907ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.495239 24174 tablet_bootstrap.cc:492] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:37.496526 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet 5160fda2643e4b80bc401bc48f97eff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:37.496623 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet c316e1aeab224634b760641a4ce188c8 with delete_type TABLET_DATA_DELETED (replaced by tablet d5bab1b7c48248a7973c27d0a865b34b) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:37.497866 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 77c49da1e0f0435ba241a6a66526e790 (DEFAULT_TABLE table=default.Table_1f6435dfbba64b358601dea132f0e1f7 [id=0de18f9c7e194e698b4f37603bfd4780]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.499784 24174 tablet_bootstrap.cc:654] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.502019 24175 tablet_bootstrap.cc:492] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:37.502211 24175 ts_tablet_manager.cc:1367] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.000s
I20240502 08:16:37.503259 24175 raft_consensus.cc:348] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.503526 24175 raft_consensus.cc:374] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.503626 24175 raft_consensus.cc:729] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.503912 24175 consensus_queue.cc:260] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.504612 24175 ts_tablet_manager.cc:1398] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:37.504737 23899 tablet_service.cc:1467] Processing CreateTablet for tablet a5768cb928e244d2914f10f4eda3c1cd (DEFAULT_TABLE table=default.Table_6b8552ddc1814ad5a20c72190929a5d9 [id=09f6a10417fb4639a9b97f5bb9efee6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.505115 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 27093862fef54c5c85d810ebeff9819c (DEFAULT_TABLE table=default.Table_c575f9ccf4c4495390078283ae331c7b [id=0035c98028aa49f7a7dcea576bb23351]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.506500 23905 tablet_service.cc:1467] Processing CreateTablet for tablet d9fcc7accb1f45ad9cf55b8f64d21b91 (DEFAULT_TABLE table=default.Table_24d1bd1b4dcc4882884e13873cf896ff [id=9d91876e10fd42f883117eb1434d3a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.507275 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 27093862fef54c5c85d810ebeff9819c (DEFAULT_TABLE table=default.Table_c575f9ccf4c4495390078283ae331c7b [id=0035c98028aa49f7a7dcea576bb23351]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.507836 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27093862fef54c5c85d810ebeff9819c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.508843 23767 tablet_service.cc:1467] Processing CreateTablet for tablet d9fcc7accb1f45ad9cf55b8f64d21b91 (DEFAULT_TABLE table=default.Table_24d1bd1b4dcc4882884e13873cf896ff [id=9d91876e10fd42f883117eb1434d3a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.509230 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9fcc7accb1f45ad9cf55b8f64d21b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.514186 23759 tablet_service.cc:1467] Processing CreateTablet for tablet a5768cb928e244d2914f10f4eda3c1cd (DEFAULT_TABLE table=default.Table_6b8552ddc1814ad5a20c72190929a5d9 [id=09f6a10417fb4639a9b97f5bb9efee6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.514669 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5768cb928e244d2914f10f4eda3c1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.515234 24035 tablet_service.cc:1467] Processing CreateTablet for tablet a5768cb928e244d2914f10f4eda3c1cd (DEFAULT_TABLE table=default.Table_6b8552ddc1814ad5a20c72190929a5d9 [id=09f6a10417fb4639a9b97f5bb9efee6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.515722 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5768cb928e244d2914f10f4eda3c1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.517890 24175 tablet_bootstrap.cc:492] T 5160fda2643e4b80bc401bc48f97eff9 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:37.518188 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 27093862fef54c5c85d810ebeff9819c (DEFAULT_TABLE table=default.Table_c575f9ccf4c4495390078283ae331c7b [id=0035c98028aa49f7a7dcea576bb23351]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.518522 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27093862fef54c5c85d810ebeff9819c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.520131 24175 tablet_bootstrap.cc:654] T 5160fda2643e4b80bc401bc48f97eff9 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.523449 24173 tablet_bootstrap.cc:492] T 5160fda2643e4b80bc401bc48f97eff9 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.523612 24173 ts_tablet_manager.cc:1367] T 5160fda2643e4b80bc401bc48f97eff9 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.035s user 0.017s sys 0.008s
I20240502 08:16:37.524533 24173 raft_consensus.cc:348] T 5160fda2643e4b80bc401bc48f97eff9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.525918 24173 raft_consensus.cc:374] T 5160fda2643e4b80bc401bc48f97eff9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.526044 24173 raft_consensus.cc:729] T 5160fda2643e4b80bc401bc48f97eff9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.526289 24173 consensus_queue.cc:260] T 5160fda2643e4b80bc401bc48f97eff9 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.526808 24030 tablet_service.cc:1467] Processing CreateTablet for tablet d9fcc7accb1f45ad9cf55b8f64d21b91 (DEFAULT_TABLE table=default.Table_24d1bd1b4dcc4882884e13873cf896ff [id=9d91876e10fd42f883117eb1434d3a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.527401 24329 raft_consensus.cc:482] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.527560 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 98a96292b28a4b0aad88fe0a96730a68 (DEFAULT_TABLE table=default.Table_2849e243f1a94b139d3234f0c8e3c755 [id=120f59a131ed47bb8f56dbc3a4ebec3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.527592 24329 raft_consensus.cc:504] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.528275 24329 leader_election.cc:288] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:37.528331 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9fcc7accb1f45ad9cf55b8f64d21b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.529769 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce5ccfd29724dc49afd0892b99f3169" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:37.530046 24060 raft_consensus.cc:2454] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:37.530440 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce5ccfd29724dc49afd0892b99f3169" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:37.530624 23925 raft_consensus.cc:2454] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:37.531152 24173 ts_tablet_manager.cc:1398] T 5160fda2643e4b80bc401bc48f97eff9 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 08:16:37.531237 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 98a96292b28a4b0aad88fe0a96730a68 (DEFAULT_TABLE table=default.Table_2849e243f1a94b139d3234f0c8e3c755 [id=120f59a131ed47bb8f56dbc3a4ebec3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.531519 24173 tablet_bootstrap.cc:492] T c709efdd2ce74b5a8ab7278b402ebfe9 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:37.531615 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98a96292b28a4b0aad88fe0a96730a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.533969 23723 leader_election.cc:302] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:37.534410 24329 raft_consensus.cc:2781] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:37.534436 24173 tablet_bootstrap.cc:654] T c709efdd2ce74b5a8ab7278b402ebfe9 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.534529 24329 raft_consensus.cc:482] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:37.534613 24329 raft_consensus.cc:3037] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.534713 24175 tablet_bootstrap.cc:492] T 5160fda2643e4b80bc401bc48f97eff9 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:37.534881 24175 ts_tablet_manager.cc:1367] T 5160fda2643e4b80bc401bc48f97eff9 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240502 08:16:37.535712 24175 raft_consensus.cc:348] T 5160fda2643e4b80bc401bc48f97eff9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.535964 24175 raft_consensus.cc:374] T 5160fda2643e4b80bc401bc48f97eff9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.536037 24175 raft_consensus.cc:729] T 5160fda2643e4b80bc401bc48f97eff9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.536298 24175 consensus_queue.cc:260] T 5160fda2643e4b80bc401bc48f97eff9 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.536914 24175 ts_tablet_manager.cc:1398] T 5160fda2643e4b80bc401bc48f97eff9 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:37.538074 24174 tablet_bootstrap.cc:492] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:37.538260 24174 ts_tablet_manager.cc:1367] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.000s
I20240502 08:16:37.539397 24175 tablet_bootstrap.cc:492] T 6f7518b1e36147a9b9f0c58bca17061a P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:37.539947 24174 raft_consensus.cc:348] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.540297 24174 raft_consensus.cc:374] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.540403 24174 raft_consensus.cc:729] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.540704 24174 consensus_queue.cc:260] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.541217 24174 ts_tablet_manager.cc:1398] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:37.540016 24329 raft_consensus.cc:504] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.546713 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce5ccfd29724dc49afd0892b99f3169" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:37.546911 23925 raft_consensus.cc:3037] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.549739 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce5ccfd29724dc49afd0892b99f3169" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:37.550089 24060 raft_consensus.cc:3037] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.551007 23925 raft_consensus.cc:2454] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:37.551936 24175 tablet_bootstrap.cc:654] T 6f7518b1e36147a9b9f0c58bca17061a P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.552775 24060 raft_consensus.cc:2454] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:37.554366 24173 tablet_bootstrap.cc:492] T c709efdd2ce74b5a8ab7278b402ebfe9 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.554563 24173 ts_tablet_manager.cc:1367] T c709efdd2ce74b5a8ab7278b402ebfe9 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.005s
I20240502 08:16:37.555585 24173 raft_consensus.cc:348] T c709efdd2ce74b5a8ab7278b402ebfe9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.555836 24173 raft_consensus.cc:374] T c709efdd2ce74b5a8ab7278b402ebfe9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.555950 24173 raft_consensus.cc:729] T c709efdd2ce74b5a8ab7278b402ebfe9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.556233 24173 consensus_queue.cc:260] T c709efdd2ce74b5a8ab7278b402ebfe9 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.557538 24173 ts_tablet_manager.cc:1398] T c709efdd2ce74b5a8ab7278b402ebfe9 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:37.557699 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 98a96292b28a4b0aad88fe0a96730a68 (DEFAULT_TABLE table=default.Table_2849e243f1a94b139d3234f0c8e3c755 [id=120f59a131ed47bb8f56dbc3a4ebec3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.558053 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98a96292b28a4b0aad88fe0a96730a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.558646 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 1e5de793e74247dd9e73b7b1d58027d9 (DEFAULT_TABLE table=default.Table_a286e67e441e43a1b4738cd219dbea98 [id=b56e3c0686244ad6b49afdd589c5ed34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.558974 24173 tablet_bootstrap.cc:492] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:37.560642 24173 tablet_bootstrap.cc:654] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.560691 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 1e5de793e74247dd9e73b7b1d58027d9 (DEFAULT_TABLE table=default.Table_a286e67e441e43a1b4738cd219dbea98 [id=b56e3c0686244ad6b49afdd589c5ed34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.561116 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e5de793e74247dd9e73b7b1d58027d9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:37.561460 23595 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet da4be1d381a64b7baeb1c60cb17d924a (table default.Table_1e7f2516f34b483d913cf5080a8febec [id=d5c976b2debc472fa1d7c28ae73ca32a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet da4be1d381a64b7baeb1c60cb17d924a consensus state has no leader
I20240502 08:16:37.564142 24173 tablet_bootstrap.cc:492] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.564296 24173 ts_tablet_manager.cc:1367] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:16:37.565088 24173 raft_consensus.cc:348] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.565362 24173 raft_consensus.cc:374] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.565444 24173 raft_consensus.cc:729] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.565726 24173 consensus_queue.cc:260] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.566021 24175 tablet_bootstrap.cc:492] T 6f7518b1e36147a9b9f0c58bca17061a P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:37.566148 24175 ts_tablet_manager.cc:1367] T 6f7518b1e36147a9b9f0c58bca17061a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.002s
I20240502 08:16:37.566924 24175 raft_consensus.cc:348] T 6f7518b1e36147a9b9f0c58bca17061a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.567116 24175 raft_consensus.cc:374] T 6f7518b1e36147a9b9f0c58bca17061a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.567216 24175 raft_consensus.cc:729] T 6f7518b1e36147a9b9f0c58bca17061a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.567363 24173 ts_tablet_manager.cc:1398] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:37.567917 24173 tablet_bootstrap.cc:492] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:37.567840 24175 consensus_queue.cc:260] T 6f7518b1e36147a9b9f0c58bca17061a P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.568375 24175 ts_tablet_manager.cc:1398] T 6f7518b1e36147a9b9f0c58bca17061a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:16:37.568701 24175 tablet_bootstrap.cc:492] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:37.569701 24355 raft_consensus.cc:482] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.569851 24355 raft_consensus.cc:504] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.570333 24175 tablet_bootstrap.cc:654] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.570463 24355 leader_election.cc:288] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:37.572731 23996 leader_election.cc:341] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Not found: Tablet not found: 05cbbf9614094afaa6f7cee0af597242
W20240502 08:16:37.573072 23993 leader_election.cc:341] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 05cbbf9614094afaa6f7cee0af597242
I20240502 08:16:37.573216 23993 leader_election.cc:302] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:37.574095 24355 raft_consensus.cc:2726] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:37.576983 24173 tablet_bootstrap.cc:654] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.553002 24174 tablet_bootstrap.cc:492] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:37.567633 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 1e5de793e74247dd9e73b7b1d58027d9 (DEFAULT_TABLE table=default.Table_a286e67e441e43a1b4738cd219dbea98 [id=b56e3c0686244ad6b49afdd589c5ed34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.567698 23757 tablet_service.cc:1467] Processing CreateTablet for tablet b8b1fa17dad143f0a7f1751e651eb81a (DEFAULT_TABLE table=default.Table_c29e7767625846bc83fdd0c850f54966 [id=57722a7341c541b19efa213bcf48eb53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.548003 24329 leader_election.cc:288] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:37.553025 23723 leader_election.cc:302] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:37.578603 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e5de793e74247dd9e73b7b1d58027d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.583827 24175 tablet_bootstrap.cc:492] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:37.583985 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b1fa17dad143f0a7f1751e651eb81a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.584013 24175 ts_tablet_manager.cc:1367] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.001s
I20240502 08:16:37.584805 24175 raft_consensus.cc:348] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.585038 24175 raft_consensus.cc:374] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.585129 24175 raft_consensus.cc:729] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.585412 24175 consensus_queue.cc:260] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.586071 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160fda2643e4b80bc401bc48f97eff9" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:37.586282 24059 raft_consensus.cc:2454] T 5160fda2643e4b80bc401bc48f97eff9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:37.586665 24175 ts_tablet_manager.cc:1398] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 08:16:37.586948 24175 tablet_bootstrap.cc:492] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:37.588254 23892 tablet_service.cc:1467] Processing CreateTablet for tablet b8b1fa17dad143f0a7f1751e651eb81a (DEFAULT_TABLE table=default.Table_c29e7767625846bc83fdd0c850f54966 [id=57722a7341c541b19efa213bcf48eb53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.588351 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 9c072eb45e1b41eeb5aafa6d5a786944 (DEFAULT_TABLE table=default.Table_20c5ace604d64b6cabead6037cdc04e1 [id=2e5831a5849049c8bd445667b01b1bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.589192 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 9c072eb45e1b41eeb5aafa6d5a786944 (DEFAULT_TABLE table=default.Table_20c5ace604d64b6cabead6037cdc04e1 [id=2e5831a5849049c8bd445667b01b1bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.589581 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c072eb45e1b41eeb5aafa6d5a786944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.591370 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a57ed188315493198e95cefb6907ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.591624 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c072eb45e1b41eeb5aafa6d5a786944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.591821 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5768cb928e244d2914f10f4eda3c1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.592036 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e5de793e74247dd9e73b7b1d58027d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.593672 24175 tablet_bootstrap.cc:654] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.595904 24035 tablet_service.cc:1467] Processing CreateTablet for tablet e2126bf2d4ab44628b52481dc2ec3a99 (DEFAULT_TABLE table=default.Table_5b4e617d1c8949daa34aa358c1528f88 [id=92cbf164660d48bd974bf8a26b8dfb0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.596273 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2126bf2d4ab44628b52481dc2ec3a99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.597287 24173 tablet_bootstrap.cc:492] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.597481 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9fcc7accb1f45ad9cf55b8f64d21b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.597519 24173 ts_tablet_manager.cc:1367] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.003s
I20240502 08:16:37.598361 24173 raft_consensus.cc:348] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.597709 24174 tablet_bootstrap.cc:654] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.598547 24173 raft_consensus.cc:374] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.598621 24173 raft_consensus.cc:729] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.598877 24173 consensus_queue.cc:260] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.599493 24173 ts_tablet_manager.cc:1398] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:16:37.599730 24173 tablet_bootstrap.cc:492] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:37.598837 23891 tablet_service.cc:1467] Processing CreateTablet for tablet e2126bf2d4ab44628b52481dc2ec3a99 (DEFAULT_TABLE table=default.Table_5b4e617d1c8949daa34aa358c1528f88 [id=92cbf164660d48bd974bf8a26b8dfb0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.602082 24030 tablet_service.cc:1467] Processing CreateTablet for tablet b8b1fa17dad143f0a7f1751e651eb81a (DEFAULT_TABLE table=default.Table_c29e7767625846bc83fdd0c850f54966 [id=57722a7341c541b19efa213bcf48eb53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.603330 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b1fa17dad143f0a7f1751e651eb81a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.603389 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b1fa17dad143f0a7f1751e651eb81a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.603585 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98a96292b28a4b0aad88fe0a96730a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.597630 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 77c49da1e0f0435ba241a6a66526e790 (DEFAULT_TABLE table=default.Table_1f6435dfbba64b358601dea132f0e1f7 [id=0de18f9c7e194e698b4f37603bfd4780]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.597993 23723 leader_election.cc:302] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:37.603955 24329 raft_consensus.cc:2781] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:37.604470 24173 tablet_bootstrap.cc:654] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.610747 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77c49da1e0f0435ba241a6a66526e790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.611095 24329 raft_consensus.cc:686] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:37.611229 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2126bf2d4ab44628b52481dc2ec3a99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.611497 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27093862fef54c5c85d810ebeff9819c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.612244 24343 raft_consensus.cc:2781] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:37.616039 24329 consensus_queue.cc:237] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.616050 24343 raft_consensus.cc:482] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:37.617143 23759 tablet_service.cc:1467] Processing CreateTablet for tablet e2126bf2d4ab44628b52481dc2ec3a99 (DEFAULT_TABLE table=default.Table_5b4e617d1c8949daa34aa358c1528f88 [id=92cbf164660d48bd974bf8a26b8dfb0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.618002 24343 raft_consensus.cc:3037] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.625206 24173 tablet_bootstrap.cc:492] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.625416 24173 ts_tablet_manager.cc:1367] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.009s
I20240502 08:16:37.626250 24173 raft_consensus.cc:348] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:37.626470 24173 raft_consensus.cc:374] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.626542 24173 raft_consensus.cc:729] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.626766 24173 consensus_queue.cc:260] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:37.630147 24173 ts_tablet_manager.cc:1398] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:16:37.633249 24173 tablet_bootstrap.cc:492] T a7c3800fb2784a1dba12d01c56c226e5 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:37.635077 24173 tablet_bootstrap.cc:654] T a7c3800fb2784a1dba12d01c56c226e5 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.637557 24174 tablet_bootstrap.cc:492] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
W20240502 08:16:37.642176 23434 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 188080us
I20240502 08:16:37.644155 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 319fabf91458438ba5e40ab68b323055 (DEFAULT_TABLE table=default.Table_becb097785284ffba69cdbe40e6480ab [id=4454421c70094eed8a8f3e42dcc8a524]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.651473 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 319fabf91458438ba5e40ab68b323055 (DEFAULT_TABLE table=default.Table_becb097785284ffba69cdbe40e6480ab [id=4454421c70094eed8a8f3e42dcc8a524]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.651973 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 319fabf91458438ba5e40ab68b323055. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.652585 24173 tablet_bootstrap.cc:492] T a7c3800fb2784a1dba12d01c56c226e5 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.652735 24173 ts_tablet_manager.cc:1367] T a7c3800fb2784a1dba12d01c56c226e5 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.014s
I20240502 08:16:37.653632 24173 raft_consensus.cc:348] T a7c3800fb2784a1dba12d01c56c226e5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:37.653860 24173 raft_consensus.cc:374] T a7c3800fb2784a1dba12d01c56c226e5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.653971 24173 raft_consensus.cc:729] T a7c3800fb2784a1dba12d01c56c226e5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.654247 24173 consensus_queue.cc:260] T a7c3800fb2784a1dba12d01c56c226e5 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:37.657825 24173 ts_tablet_manager.cc:1398] T a7c3800fb2784a1dba12d01c56c226e5 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240502 08:16:37.658664 24175 tablet_bootstrap.cc:492] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:37.658824 24175 ts_tablet_manager.cc:1367] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.072s user 0.000s sys 0.003s
I20240502 08:16:37.659957 24175 raft_consensus.cc:348] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.660176 24175 raft_consensus.cc:374] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.660267 24175 raft_consensus.cc:729] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.660372 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 319fabf91458438ba5e40ab68b323055. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.660526 24175 consensus_queue.cc:260] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.660733 24173 tablet_bootstrap.cc:492] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:37.662272 24175 ts_tablet_manager.cc:1398] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:16:37.664194 24173 tablet_bootstrap.cc:654] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.668457 23605 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet b1e68513374d4c51addec439de7ae3d1
I20240502 08:16:37.670147 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1f6435dfbba64b358601dea132f0e1f7" } new_table_name: "default.Table_da5548ba323e495dafdd33eb59a9c6b7" modify_external_catalogs: true
I20240502 08:16:37.672205 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_bd4df4dff42547d6aba1825676ef7879"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:37.673234 23603 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 1ce5ccfd29724dc49afd0892b99f3169 (table default.Table_bdbe807258a5454795eaff8c27bb17dd [id=3a9d7e8fe2c84cb4942fee9053a012b0]). Expected version 1 got 0
I20240502 08:16:37.674523 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 4302fec5c51e4c6c841710bab50b2997
I20240502 08:16:37.674964 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 9700ea9dc58442e3ab1b7835a8c1e22c
I20240502 08:16:37.675789 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet c08228fcd1304528b9446003851f6c70
I20240502 08:16:37.676409 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 0dff0d73a933427fb1f2686bac33eba1
I20240502 08:16:37.676970 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet ce758c4a9258427987e8e83f579a2aa2
I20240502 08:16:37.678570 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77c49da1e0f0435ba241a6a66526e790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.679368 24343 raft_consensus.cc:504] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.680058 24343 leader_election.cc:288] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:37.682335 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0a242b17e3d2450eb3add7dbff397394" } new_table_name: "default.Table_d10cdf44509d4b4aabd9fb5c4a035f4b" modify_external_catalogs: true
I20240502 08:16:37.684473 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_dcff63ecc4f740fd999047021472be23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:37.685140 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcff63ecc4f740fd999047021472be23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:37.686295 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_dbe385cc3f6b48d39bee298ca9490248"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:37.686793 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet 4302fec5c51e4c6c841710bab50b2997 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:37.686847 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbe385cc3f6b48d39bee298ca9490248 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:37.687040 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet ce758c4a9258427987e8e83f579a2aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:37.688294 23604 catalog_manager.cc:6554] ReplaceTablet: tablet c08228fcd1304528b9446003851f6c70 deleted and replaced by tablet b59c2f93d6514c9cbf358ab96f1a06b5
I20240502 08:16:37.688834 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e1af8994d95141788ddb43bac3cc5a90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:37.689361 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1af8994d95141788ddb43bac3cc5a90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:37.689764 23606 catalog_manager.cc:6554] ReplaceTablet: tablet 9700ea9dc58442e3ab1b7835a8c1e22c deleted and replaced by tablet 5d04312bd81f46ed8c49c42b1af1aec9
I20240502 08:16:37.690135 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_bac7f229ce9541b39c0a70951431b689"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:37.690637 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bac7f229ce9541b39c0a70951431b689 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:37.691226 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dff0d73a933427fb1f2686bac33eba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:37.691668 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd4df4dff42547d6aba1825676ef7879 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:37.692536 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet c08228fcd1304528b9446003851f6c70 with delete_type TABLET_DATA_DELETED (replaced by tablet b59c2f93d6514c9cbf358ab96f1a06b5) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:37.692863 24174 ts_tablet_manager.cc:1367] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.096s user 0.006s sys 0.002s
I20240502 08:16:37.692916 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 319fabf91458438ba5e40ab68b323055 (DEFAULT_TABLE table=default.Table_becb097785284ffba69cdbe40e6480ab [id=4454421c70094eed8a8f3e42dcc8a524]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.693269 23605 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 768167b977c14b53a7537879e349d354 (table default.Table_0a242b17e3d2450eb3add7dbff397394 [id=ab1669d2a12f4187809987ed41fa0379]). Expected version 1 got 0
I20240502 08:16:37.693387 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 319fabf91458438ba5e40ab68b323055. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.693871 24174 raft_consensus.cc:348] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.694314 24174 raft_consensus.cc:374] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.694417 24174 raft_consensus.cc:729] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.694758 24174 consensus_queue.cc:260] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.696712 24174 ts_tablet_manager.cc:1398] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:37.697630 23763 tablet_service.cc:1467] Processing CreateTablet for tablet df9402cf8abc4b4d893d42f9a3a99016 (DEFAULT_TABLE table=default.Table_4b967ef2b2fc4e2caa4219afcae841a6 [id=c32fc96889554c32bdf1d4550633e91f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.697925 23605 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 1ce5ccfd29724dc49afd0892b99f3169 (table default.Table_bdbe807258a5454795eaff8c27bb17dd [id=3a9d7e8fe2c84cb4942fee9053a012b0]). Expected version 1 got 0
I20240502 08:16:37.698040 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df9402cf8abc4b4d893d42f9a3a99016. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.698207 24174 tablet_bootstrap.cc:492] T 485c772245fe48a39fc6ceb6377ac0b6 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:37.700104 24174 tablet_bootstrap.cc:654] T 485c772245fe48a39fc6ceb6377ac0b6 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.701210 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet 9700ea9dc58442e3ab1b7835a8c1e22c with delete_type TABLET_DATA_DELETED (replaced by tablet 5d04312bd81f46ed8c49c42b1af1aec9) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:37.701403 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 9c072eb45e1b41eeb5aafa6d5a786944 (DEFAULT_TABLE table=default.Table_20c5ace604d64b6cabead6037cdc04e1 [id=2e5831a5849049c8bd445667b01b1bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.701905 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 5bcc7169796e4ee0910f5e7084849012 (DEFAULT_TABLE table=default.Table_bc9b016da1ce42b6a0509fbb664c8548 [id=8c0e32da69b34dd0b2a3c68e4ba7a166]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.706247 24355 raft_consensus.cc:482] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.706416 24355 raft_consensus.cc:504] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.707029 24355 leader_election.cc:288] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:37.707918 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "405ea360814e4593a289d753821277ff" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:37.708122 23993 leader_election.cc:341] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 405ea360814e4593a289d753821277ff
W20240502 08:16:37.708642 23996 leader_election.cc:341] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:37.708775 23996 leader_election.cc:302] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:37.709225 24355 raft_consensus.cc:2726] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:37.710044 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c072eb45e1b41eeb5aafa6d5a786944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.711823 24173 tablet_bootstrap.cc:492] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.712044 24173 ts_tablet_manager.cc:1367] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.018s
I20240502 08:16:37.712793 24173 raft_consensus.cc:348] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:37.713081 24173 raft_consensus.cc:374] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.713196 24173 raft_consensus.cc:729] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.713480 24173 consensus_queue.cc:260] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:37.715898 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e549cad1b3be4b28abb0bf5af720eb52
I20240502 08:16:37.716635 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 4302fec5c51e4c6c841710bab50b2997 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:37.717567 24173 ts_tablet_manager.cc:1398] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 08:16:37.717870 24175 tablet_bootstrap.cc:492] T 223c92ae8f63402f8643e03d914e8e7e P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:37.718173 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160fda2643e4b80bc401bc48f97eff9" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:37.718237 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet ce758c4a9258427987e8e83f579a2aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:37.718367 23925 raft_consensus.cc:3037] T 5160fda2643e4b80bc401bc48f97eff9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:37.718883 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 4302fec5c51e4c6c841710bab50b2997 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4302fec5c51e4c6c841710bab50b2997 already in progress: creating tablet
W20240502 08:16:37.719133 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet ce758c4a9258427987e8e83f579a2aa2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce758c4a9258427987e8e83f579a2aa2 already in progress: creating tablet
I20240502 08:16:37.719398 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet c08228fcd1304528b9446003851f6c70 with delete_type TABLET_DATA_DELETED (replaced by tablet b59c2f93d6514c9cbf358ab96f1a06b5) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:37.719626 24175 tablet_bootstrap.cc:654] T 223c92ae8f63402f8643e03d914e8e7e P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:37.719993 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet c08228fcd1304528b9446003851f6c70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c08228fcd1304528b9446003851f6c70 already in progress: creating tablet
I20240502 08:16:37.720288 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dff0d73a933427fb1f2686bac33eba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:37.720618 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet e549cad1b3be4b28abb0bf5af720eb52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:37.721087 23925 raft_consensus.cc:2454] T 5160fda2643e4b80bc401bc48f97eff9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:37.721160 24173 tablet_bootstrap.cc:492] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:37.721129 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 0dff0d73a933427fb1f2686bac33eba1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0dff0d73a933427fb1f2686bac33eba1 already in progress: creating tablet
I20240502 08:16:37.721343 23895 tablet_service.cc:1467] Processing CreateTablet for tablet c17e1be6ef294769a01981f4d361be21 (DEFAULT_TABLE table=default.Table_498b372c15a04b70b87bd67b543501cf [id=38e955d6bd644d24a7c45172ab60282b]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:37.721436 23597 catalog_manager.cc:4797] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e549cad1b3be4b28abb0bf5af720eb52 because the tablet was not found. No further retry: Not found: Tablet not found: e549cad1b3be4b28abb0bf5af720eb52
I20240502 08:16:37.721761 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c17e1be6ef294769a01981f4d361be21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.722623 24173 tablet_bootstrap.cc:654] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.730581 24173 tablet_bootstrap.cc:492] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.730732 24173 ts_tablet_manager.cc:1367] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20240502 08:16:37.731307 24175 tablet_bootstrap.cc:492] T 223c92ae8f63402f8643e03d914e8e7e P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:37.731477 24173 raft_consensus.cc:348] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:37.731663 24173 raft_consensus.cc:374] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.731747 24173 raft_consensus.cc:729] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.732017 24173 consensus_queue.cc:260] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:37.734347 24173 ts_tablet_manager.cc:1398] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:16:37.734627 24173 tablet_bootstrap.cc:492] T 203ed05d0bdc4b1f99207423b7fc848b P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:37.735812 24175 ts_tablet_manager.cc:1367] T 223c92ae8f63402f8643e03d914e8e7e P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.006s
I20240502 08:16:37.735749 23924 raft_consensus.cc:1261] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:37.735689 23899 tablet_service.cc:1467] Processing CreateTablet for tablet df9402cf8abc4b4d893d42f9a3a99016 (DEFAULT_TABLE table=default.Table_4b967ef2b2fc4e2caa4219afcae841a6 [id=c32fc96889554c32bdf1d4550633e91f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.735780 23900 tablet_service.cc:1467] Processing CreateTablet for tablet aacdf506221044ffada0fec0c7d42d0b (DEFAULT_TABLE table=default.Table_102eeba031aa4389820d28985935dd4c [id=9480d59a76fc401e959a1e41caafc845]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.736485 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aacdf506221044ffada0fec0c7d42d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.736502 24173 tablet_bootstrap.cc:654] T 203ed05d0bdc4b1f99207423b7fc848b P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.736537 24175 raft_consensus.cc:348] T 223c92ae8f63402f8643e03d914e8e7e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.736732 24175 raft_consensus.cc:374] T 223c92ae8f63402f8643e03d914e8e7e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.736805 24175 raft_consensus.cc:729] T 223c92ae8f63402f8643e03d914e8e7e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.737447 24060 raft_consensus.cc:1261] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:37.737067 24175 consensus_queue.cc:260] T 223c92ae8f63402f8643e03d914e8e7e P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.737681 24030 tablet_service.cc:1467] Processing CreateTablet for tablet df9402cf8abc4b4d893d42f9a3a99016 (DEFAULT_TABLE table=default.Table_4b967ef2b2fc4e2caa4219afcae841a6 [id=c32fc96889554c32bdf1d4550633e91f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.738006 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df9402cf8abc4b4d893d42f9a3a99016. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.738060 24175 ts_tablet_manager.cc:1398] T 223c92ae8f63402f8643e03d914e8e7e P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:37.738363 24175 tablet_bootstrap.cc:492] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:37.739269 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5160fda2643e4b80bc401bc48f97eff9" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:37.739506 24059 raft_consensus.cc:3037] T 5160fda2643e4b80bc401bc48f97eff9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.739779 24175 tablet_bootstrap.cc:654] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.742162 24059 raft_consensus.cc:2454] T 5160fda2643e4b80bc401bc48f97eff9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:37.743310 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df9402cf8abc4b4d893d42f9a3a99016. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.744196 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bcc7169796e4ee0910f5e7084849012. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.744241 23723 leader_election.cc:302] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:37.745985 24343 raft_consensus.cc:2781] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:37.745985 24329 consensus_queue.cc:1035] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:37.746330 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 5bcc7169796e4ee0910f5e7084849012 (DEFAULT_TABLE table=default.Table_bc9b016da1ce42b6a0509fbb664c8548 [id=8c0e32da69b34dd0b2a3c68e4ba7a166]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.746779 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet 9700ea9dc58442e3ab1b7835a8c1e22c with delete_type TABLET_DATA_DELETED (replaced by tablet 5d04312bd81f46ed8c49c42b1af1aec9) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.746843 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet ce758c4a9258427987e8e83f579a2aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.747936 23761 tablet_service.cc:1467] Processing CreateTablet for tablet c17e1be6ef294769a01981f4d361be21 (DEFAULT_TABLE table=default.Table_498b372c15a04b70b87bd67b543501cf [id=38e955d6bd644d24a7c45172ab60282b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.747506 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet 4302fec5c51e4c6c841710bab50b2997 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.749533 24180 ts_tablet_manager.cc:1888] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:37.750233 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bcc7169796e4ee0910f5e7084849012. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.750800 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dff0d73a933427fb1f2686bac33eba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.751358 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet c08228fcd1304528b9446003851f6c70 with delete_type TABLET_DATA_DELETED (replaced by tablet b59c2f93d6514c9cbf358ab96f1a06b5) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.752077 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 5bcc7169796e4ee0910f5e7084849012 (DEFAULT_TABLE table=default.Table_bc9b016da1ce42b6a0509fbb664c8548 [id=8c0e32da69b34dd0b2a3c68e4ba7a166]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.752493 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bcc7169796e4ee0910f5e7084849012. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.753063 24417 tablet.cc:1722] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:37.756906 24408 tablet.cc:1722] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:37.758033 24035 tablet_service.cc:1467] Processing CreateTablet for tablet c17e1be6ef294769a01981f4d361be21 (DEFAULT_TABLE table=default.Table_498b372c15a04b70b87bd67b543501cf [id=38e955d6bd644d24a7c45172ab60282b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.758167 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet e549cad1b3be4b28abb0bf5af720eb52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.758409 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c17e1be6ef294769a01981f4d361be21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.758939 24175 tablet_bootstrap.cc:492] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:37.759071 24175 ts_tablet_manager.cc:1367] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.008s
I20240502 08:16:37.759773 24175 raft_consensus.cc:348] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.759974 24175 raft_consensus.cc:374] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.760051 24175 raft_consensus.cc:729] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.760243 24175 consensus_queue.cc:260] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.760938 24175 ts_tablet_manager.cc:1398] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:16:37.761193 24175 tablet_bootstrap.cc:492] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:37.762986 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 9700ea9dc58442e3ab1b7835a8c1e22c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9700ea9dc58442e3ab1b7835a8c1e22c already in progress: creating tablet
I20240502 08:16:37.763254 24175 tablet_bootstrap.cc:654] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:37.763558 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet ce758c4a9258427987e8e83f579a2aa2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce758c4a9258427987e8e83f579a2aa2 already in progress: creating tablet
I20240502 08:16:37.763839 23607 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 08183523bf4a4d1c9020eb693aa07878 (table default.Table_fcd8e870383b4c8f9b40db6fb2a7149b [id=0a2f8dbb62a64b0a96931ab5f8c17540]). Expected version 1 got 0
I20240502 08:16:37.764146 23607 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet da4be1d381a64b7baeb1c60cb17d924a (table default.Table_1e7f2516f34b483d913cf5080a8febec [id=d5c976b2debc472fa1d7c28ae73ca32a]). Expected version 1 got 0
W20240502 08:16:37.766695 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 4302fec5c51e4c6c841710bab50b2997 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4302fec5c51e4c6c841710bab50b2997 already in progress: creating tablet
W20240502 08:16:37.766975 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 0dff0d73a933427fb1f2686bac33eba1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0dff0d73a933427fb1f2686bac33eba1 already in progress: creating tablet
W20240502 08:16:37.767203 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet c08228fcd1304528b9446003851f6c70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c08228fcd1304528b9446003851f6c70 already in progress: creating tablet
W20240502 08:16:37.767479 23597 catalog_manager.cc:4797] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e549cad1b3be4b28abb0bf5af720eb52 because the tablet was not found. No further retry: Not found: Tablet not found: e549cad1b3be4b28abb0bf5af720eb52
I20240502 08:16:37.767561 24343 raft_consensus.cc:686] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:37.767915 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2126bf2d4ab44628b52481dc2ec3a99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.772081 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c17e1be6ef294769a01981f4d361be21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.772182 24175 tablet_bootstrap.cc:492] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:37.772315 24175 ts_tablet_manager.cc:1367] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20240502 08:16:37.773015 24175 raft_consensus.cc:348] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.773239 24175 raft_consensus.cc:374] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.773319 24175 raft_consensus.cc:729] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.773540 24175 consensus_queue.cc:260] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.774495 24173 tablet_bootstrap.cc:492] T 203ed05d0bdc4b1f99207423b7fc848b P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.774715 24173 ts_tablet_manager.cc:1367] T 203ed05d0bdc4b1f99207423b7fc848b P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.007s
I20240502 08:16:37.775619 24173 raft_consensus.cc:348] T 203ed05d0bdc4b1f99207423b7fc848b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.776211 24175 ts_tablet_manager.cc:1398] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:37.776474 24175 tablet_bootstrap.cc:492] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:37.776639 24173 raft_consensus.cc:374] T 203ed05d0bdc4b1f99207423b7fc848b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.777331 24173 raft_consensus.cc:729] T 203ed05d0bdc4b1f99207423b7fc848b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.777647 24173 consensus_queue.cc:260] T 203ed05d0bdc4b1f99207423b7fc848b P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.778311 24173 ts_tablet_manager.cc:1398] T 203ed05d0bdc4b1f99207423b7fc848b P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:37.778657 24173 tablet_bootstrap.cc:492] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:37.779731 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet e549cad1b3be4b28abb0bf5af720eb52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:37.779922 24329 consensus_queue.cc:1035] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20240502 08:16:37.779991 24343 consensus_queue.cc:237] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.780288 24175 tablet_bootstrap.cc:654] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.780427 24173 tablet_bootstrap.cc:654] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.780896 24329 raft_consensus.cc:482] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.781059 24329 raft_consensus.cc:504] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.781831 24329 leader_election.cc:288] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:37.784039 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223c92ae8f63402f8643e03d914e8e7e" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:37.784307 23924 raft_consensus.cc:2454] T 223c92ae8f63402f8643e03d914e8e7e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:37.784824 23723 leader_election.cc:302] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
W20240502 08:16:37.785144 23723 leader_election.cc:341] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 223c92ae8f63402f8643e03d914e8e7e
I20240502 08:16:37.785658 24329 raft_consensus.cc:2781] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:37.785740 24329 raft_consensus.cc:482] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:37.785835 24329 raft_consensus.cc:3037] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.786314 24175 tablet_bootstrap.cc:492] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:37.786461 24175 ts_tablet_manager.cc:1367] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 08:16:37.787081 24175 raft_consensus.cc:348] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.787297 24175 raft_consensus.cc:374] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.787397 24175 raft_consensus.cc:729] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.787608 24175 consensus_queue.cc:260] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.790222 24174 tablet_bootstrap.cc:492] T 485c772245fe48a39fc6ceb6377ac0b6 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:37.790369 24174 ts_tablet_manager.cc:1367] T 485c772245fe48a39fc6ceb6377ac0b6 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.092s user 0.003s sys 0.000s
I20240502 08:16:37.790474 24175 ts_tablet_manager.cc:1398] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:37.790710 24175 tablet_bootstrap.cc:492] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:37.791008 24174 raft_consensus.cc:348] T 485c772245fe48a39fc6ceb6377ac0b6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.791232 24174 raft_consensus.cc:374] T 485c772245fe48a39fc6ceb6377ac0b6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.791321 24174 raft_consensus.cc:729] T 485c772245fe48a39fc6ceb6377ac0b6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.792016 24180 log.cc:1196] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/175adf0ecb504e28a3d39f001f977839
I20240502 08:16:37.793243 24180 ts_tablet_manager.cc:1909] T 175adf0ecb504e28a3d39f001f977839 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:37.794080 24175 tablet_bootstrap.cc:654] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.794625 23763 tablet_service.cc:1467] Processing CreateTablet for tablet aacdf506221044ffada0fec0c7d42d0b (DEFAULT_TABLE table=default.Table_102eeba031aa4389820d28985935dd4c [id=9480d59a76fc401e959a1e41caafc845]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.794696 24329 raft_consensus.cc:504] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.795123 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aacdf506221044ffada0fec0c7d42d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.792680 24174 consensus_queue.cc:260] T 485c772245fe48a39fc6ceb6377ac0b6 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.795722 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223c92ae8f63402f8643e03d914e8e7e" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:37.795909 23924 raft_consensus.cc:3037] T 223c92ae8f63402f8643e03d914e8e7e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.796932 24174 ts_tablet_manager.cc:1398] T 485c772245fe48a39fc6ceb6377ac0b6 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:16:37.797813 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 195) took 1135 ms. Trace:
I20240502 08:16:37.797940 24180 rpcz_store.cc:276] 0502 08:16:36.662264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:36.662382 (+ 118us) service_pool.cc:225] Handling call
0502 08:16:37.431377 (+768995us) ts_tablet_manager.cc:1120] Deleting tablet 175adf0ecb504e28a3d39f001f977839
0502 08:16:37.431389 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:37.473950 (+ 42561us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:37.733745 (+259795us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:37.797755 (+ 64010us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":748664}
I20240502 08:16:37.798470 23924 raft_consensus.cc:2454] T 223c92ae8f63402f8643e03d914e8e7e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:37.798561 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 175adf0ecb504e28a3d39f001f977839 (table default.Table_9530003aa5b841b683920da217d58576 [id=a6a5f4c65807479392c01ac5c955c959]) successfully deleted
I20240502 08:16:37.798920 23723 leader_election.cc:302] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:37.799252 24343 raft_consensus.cc:2781] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:37.799394 24343 raft_consensus.cc:686] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
W20240502 08:16:37.799572 23723 leader_election.cc:341] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 223c92ae8f63402f8643e03d914e8e7e
I20240502 08:16:37.799639 24343 consensus_queue.cc:237] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.799767 24329 leader_election.cc:288] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:37.801730 24175 tablet_bootstrap.cc:492] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:37.801874 24175 ts_tablet_manager.cc:1367] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240502 08:16:37.802561 24175 raft_consensus.cc:348] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.802778 24175 raft_consensus.cc:374] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.802842 24175 raft_consensus.cc:729] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.803053 24175 consensus_queue.cc:260] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.805126 24173 tablet_bootstrap.cc:492] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.805392 24173 ts_tablet_manager.cc:1367] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.004s
I20240502 08:16:37.806015 24175 ts_tablet_manager.cc:1398] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:37.806309 24175 tablet_bootstrap.cc:492] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:37.806571 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 9700ea9dc58442e3ab1b7835a8c1e22c with delete_type TABLET_DATA_DELETED (replaced by tablet 5d04312bd81f46ed8c49c42b1af1aec9) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:37.808017 24030 tablet_service.cc:1467] Processing CreateTablet for tablet aacdf506221044ffada0fec0c7d42d0b (DEFAULT_TABLE table=default.Table_102eeba031aa4389820d28985935dd4c [id=9480d59a76fc401e959a1e41caafc845]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.808244 24173 raft_consensus.cc:348] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:37.808506 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aacdf506221044ffada0fec0c7d42d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.808562 24173 raft_consensus.cc:374] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.808707 24173 raft_consensus.cc:729] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.808993 24173 consensus_queue.cc:260] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:37.809890 24173 ts_tablet_manager.cc:1398] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:37.810201 24173 tablet_bootstrap.cc:492] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:37.812325 24173 tablet_bootstrap.cc:654] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.812747 24174 tablet_bootstrap.cc:492] T 9b1eb762780c450bbaf380d17db564b6 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
W20240502 08:16:37.820154 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 77c49da1e0f0435ba241a6a66526e790 consensus state has no leader
I20240502 08:16:37.820194 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_02ea392fafea47a28cf6565e8e60dc5c" } new_table_name: "default.Table_44a0f173f72b49eea2cc2243bc11c430" modify_external_catalogs: true
I20240502 08:16:37.820479 24173 tablet_bootstrap.cc:492] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.820505 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet 5160fda2643e4b80bc401bc48f97eff9 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:37.820631 24173 ts_tablet_manager.cc:1367] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20240502 08:16:37.820894 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_480b6f612ca64d2caaa95f87d6ce8464"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:37.821138 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2e0d27237b2f4eb89acb7eb245578f1f" } new_table_name: "default.Table_f816cb42e01045ae9df44baeb445f439" modify_external_catalogs: true
I20240502 08:16:37.821290 24173 raft_consensus.cc:348] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
W20240502 08:16:37.821444 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_480b6f612ca64d2caaa95f87d6ce8464 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:37.821470 24173 raft_consensus.cc:374] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.821528 24173 raft_consensus.cc:729] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.821795 24173 consensus_queue.cc:260] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:37.822340 24173 ts_tablet_manager.cc:1398] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:37.822623 23603 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_241ebfbdbdc94f27a8ff9035c037c15d [id=eca4080e8ebf46e9b58c85ee1d8675ca] alter complete (version 3)
I20240502 08:16:37.822892 24173 tablet_bootstrap.cc:492] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:37.823302 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_241ebfbdbdc94f27a8ff9035c037c15d" } new_table_name: "default.Table_de4adf089f7347969051330f6f43e162" modify_external_catalogs: true
I20240502 08:16:37.823243 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_307041e730f248a5bd4d265f08aad5e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:37.823823 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_307041e730f248a5bd4d265f08aad5e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:37.824728 24173 tablet_bootstrap.cc:654] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.825543 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c613372da52548cc93464839137beade"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:37.825671 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7f44f7f1aed6460cb1e73392da0987b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:37.826128 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c613372da52548cc93464839137beade in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:37.826251 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f44f7f1aed6460cb1e73392da0987b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:37.827840 23608 master_service.cc:946] ReplaceTablet: received request to replace tablet 4f1736b496354e2dadd844242d3867f8 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:37.833339 24457 raft_consensus.cc:482] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.833554 24457 raft_consensus.cc:504] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.834460 24457 leader_election.cc:288] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:37.836009 24173 tablet_bootstrap.cc:492] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:37.836172 24173 ts_tablet_manager.cc:1367] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20240502 08:16:37.837179 24355 raft_consensus.cc:482] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.837407 24355 raft_consensus.cc:504] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.838114 24355 leader_election.cc:288] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:37.838172 23723 leader_election.cc:341] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 6f7518b1e36147a9b9f0c58bca17061a
I20240502 08:16:37.839556 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df565c1c6564efd88fa557c50522d7e" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:37.840194 23993 leader_election.cc:341] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:37.837071 24173 raft_consensus.cc:348] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:37.840664 24173 raft_consensus.cc:374] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.840832 24173 raft_consensus.cc:729] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.841105 24173 consensus_queue.cc:260] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:37.841889 24173 ts_tablet_manager.cc:1398] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:16:37.842178 24173 tablet_bootstrap.cc:492] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:37.844049 24173 tablet_bootstrap.cc:654] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:37.846727 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 9700ea9dc58442e3ab1b7835a8c1e22c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9700ea9dc58442e3ab1b7835a8c1e22c already in progress: creating tablet
I20240502 08:16:37.810417 24175 tablet_bootstrap.cc:654] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.848237 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 96fcb895c8f449deb0190de75af18c3c (DEFAULT_TABLE table=default.Table_22e9411445aa459286b4f9038d84651b [id=e6b3d41989d84333b86973785d6fc2a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.849404 24408 tablet.cc:1722] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:37.835587 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f7518b1e36147a9b9f0c58bca17061a" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:37.849718 23924 raft_consensus.cc:2454] T 6f7518b1e36147a9b9f0c58bca17061a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:37.838845 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df565c1c6564efd88fa557c50522d7e" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:37.850133 23723 leader_election.cc:302] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
W20240502 08:16:37.850306 23996 leader_election.cc:341] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:37.850406 23996 leader_election.cc:302] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:37.850574 24457 raft_consensus.cc:2781] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:37.816267 24183 raft_consensus.cc:482] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.850675 24457 raft_consensus.cc:482] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:37.850759 24457 raft_consensus.cc:3037] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.850760 24183 raft_consensus.cc:504] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.851377 24183 leader_election.cc:288] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:37.852288 23861 leader_election.cc:341] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: a15ff3120b1943a69ddb609d0575ae1a
I20240502 08:16:37.853215 24457 raft_consensus.cc:504] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.853641 24355 raft_consensus.cc:2726] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:37.854116 24417 tablet.cc:1722] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:37.854406 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a15ff3120b1943a69ddb609d0575ae1a" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:37.854633 23787 raft_consensus.cc:2454] T a15ff3120b1943a69ddb609d0575ae1a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:37.854882 24457 leader_election.cc:288] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:37.855513 23858 leader_election.cc:302] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:37.855857 24183 raft_consensus.cc:2781] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:37.855978 24183 raft_consensus.cc:482] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:37.856086 24183 raft_consensus.cc:3037] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.858760 24183 raft_consensus.cc:504] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.859711 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a15ff3120b1943a69ddb609d0575ae1a" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
W20240502 08:16:37.859889 23723 leader_election.cc:341] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 6f7518b1e36147a9b9f0c58bca17061a
I20240502 08:16:37.859913 23787 raft_consensus.cc:3037] T a15ff3120b1943a69ddb609d0575ae1a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.860081 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fcb895c8f449deb0190de75af18c3c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:37.860249 23861 leader_election.cc:341] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: a15ff3120b1943a69ddb609d0575ae1a
I20240502 08:16:37.862100 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 194) took 1222 ms. Trace:
I20240502 08:16:37.862196 24180 rpcz_store.cc:276] 0502 08:16:36.650920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:36.650998 (+ 78us) service_pool.cc:225] Handling call
0502 08:16:37.861930 (+1210932us) ts_tablet_manager.cc:1120] Deleting tablet 10adabdd31f848548ef6f2408ccc2050
0502 08:16:37.861941 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:37.862048 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1115478}
I20240502 08:16:37.862368 23787 raft_consensus.cc:2454] T a15ff3120b1943a69ddb609d0575ae1a P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:37.862581 24355 raft_consensus.cc:482] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:16:37.862694 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 10adabdd31f848548ef6f2408ccc2050 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10adabdd31f848548ef6f2408ccc2050 already in progress: creating tablet
I20240502 08:16:37.862725 23858 leader_election.cc:302] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:37.862728 24355 raft_consensus.cc:504] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:37.863358 24355 leader_election.cc:288] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:37.864449 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 197) took 1178 ms. Trace:
I20240502 08:16:37.864856 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3d4b4584dd45b58ff0daa4f1c09a0a" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:37.865660 23996 leader_election.cc:341] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:37.866026 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f7518b1e36147a9b9f0c58bca17061a" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:37.866272 23923 raft_consensus.cc:3037] T 6f7518b1e36147a9b9f0c58bca17061a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.869172 23923 raft_consensus.cc:2454] T 6f7518b1e36147a9b9f0c58bca17061a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:37.869386 24183 leader_election.cc:288] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:37.869550 24183 raft_consensus.cc:2781] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:37.869684 24183 raft_consensus.cc:686] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:16:37.869629 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 96fcb895c8f449deb0190de75af18c3c (DEFAULT_TABLE table=default.Table_22e9411445aa459286b4f9038d84651b [id=e6b3d41989d84333b86973785d6fc2a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.869999 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fcb895c8f449deb0190de75af18c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.870069 24183 consensus_queue.cc:237] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.874720 24180 rpcz_store.cc:276] 0502 08:16:36.685708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:36.685958 (+ 250us) service_pool.cc:225] Handling call
0502 08:16:37.863819 (+1177861us) ts_tablet_manager.cc:1120] Deleting tablet f36ceb0222ef473a828ca795f393174d
0502 08:16:37.863830 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:37.864398 (+ 568us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1177660}
W20240502 08:16:37.875950 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet f36ceb0222ef473a828ca795f393174d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f36ceb0222ef473a828ca795f393174d already in progress: creating tablet
I20240502 08:16:37.865381 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3d4b4584dd45b58ff0daa4f1c09a0a" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:37.876377 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 196) took 1205 ms. Trace:
I20240502 08:16:37.876521 24180 rpcz_store.cc:276] 0502 08:16:36.670906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:36.674934 (+ 4028us) service_pool.cc:225] Handling call
0502 08:16:37.876243 (+1201309us) ts_tablet_manager.cc:1120] Deleting tablet d8f7b5f018d84680ac5533bf99102d31
0502 08:16:37.876253 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:37.876323 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1134059}
W20240502 08:16:37.876742 23993 leader_election.cc:341] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:37.876878 23993 leader_election.cc:302] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:37.877264 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet d8f7b5f018d84680ac5533bf99102d31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8f7b5f018d84680ac5533bf99102d31 already in progress: creating tablet
I20240502 08:16:37.877266 24355 raft_consensus.cc:2726] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:37.877982 23767 tablet_service.cc:1467] Processing CreateTablet for tablet c1e746013fde4de4862f213792e235ed (DEFAULT_TABLE table=default.Table_725d048135a04e32bd016e60ac974715 [id=ecfe360ec5554bdcac562a1c614e1672]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.878378 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e746013fde4de4862f213792e235ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.878590 24035 tablet_service.cc:1467] Processing CreateTablet for tablet c1e746013fde4de4862f213792e235ed (DEFAULT_TABLE table=default.Table_725d048135a04e32bd016e60ac974715 [id=ecfe360ec5554bdcac562a1c614e1672]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.878933 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e746013fde4de4862f213792e235ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.879113 23761 tablet_service.cc:1467] Processing CreateTablet for tablet bd4a31e5aa834adeb9f59267247c6980 (DEFAULT_TABLE table=default.Table_4e40fa1cf0ab4dedb2c6aadb2418901b [id=fd781805eb4c45f09fe6460144472816]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.879494 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4a31e5aa834adeb9f59267247c6980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.880991 24033 tablet_service.cc:1467] Processing CreateTablet for tablet bd4a31e5aa834adeb9f59267247c6980 (DEFAULT_TABLE table=default.Table_4e40fa1cf0ab4dedb2c6aadb2418901b [id=fd781805eb4c45f09fe6460144472816]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.881364 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4a31e5aa834adeb9f59267247c6980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.882400 24355 raft_consensus.cc:482] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.882544 24355 raft_consensus.cc:504] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:37.883148 24355 leader_election.cc:288] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:37.883486 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5738cd77bbe147b49215ba9639d7f547" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:37.883682 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5738cd77bbe147b49215ba9639d7f547" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:37.883904 23996 leader_election.cc:341] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:37.884110 23993 leader_election.cc:341] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:37.884277 23993 leader_election.cc:302] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:37.886054 24355 raft_consensus.cc:2726] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:37.886689 23434 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 96830.4us
I20240502 08:16:37.866727 24174 tablet_bootstrap.cc:654] T 9b1eb762780c450bbaf380d17db564b6 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.868168 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 96fcb895c8f449deb0190de75af18c3c (DEFAULT_TABLE table=default.Table_22e9411445aa459286b4f9038d84651b [id=e6b3d41989d84333b86973785d6fc2a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.897298 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fcb895c8f449deb0190de75af18c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.898540 24183 raft_consensus.cc:482] T c7aa03e73acd4a9da5574b9ba93415ae P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.898722 24183 raft_consensus.cc:504] T c7aa03e73acd4a9da5574b9ba93415ae P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.898905 23605 master_service.cc:946] ReplaceTablet: received request to replace tablet da4be1d381a64b7baeb1c60cb17d924a from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:37.900840 24183 leader_election.cc:288] T c7aa03e73acd4a9da5574b9ba93415ae P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:37.901940 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 200) took 1050 ms. Trace:
I20240502 08:16:37.902062 24180 rpcz_store.cc:276] 0502 08:16:36.851611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:36.851894 (+ 283us) service_pool.cc:225] Handling call
0502 08:16:37.901710 (+1049816us) ts_tablet_manager.cc:1120] Deleting tablet 01675f637a5a4645949abfd87b3d09a2
0502 08:16:37.901724 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:37.901873 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1049595}
I20240502 08:16:37.902670 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 201) took 1050 ms. Trace:
I20240502 08:16:37.902767 24180 rpcz_store.cc:276] 0502 08:16:36.851745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:36.852137 (+ 392us) service_pool.cc:225] Handling call
0502 08:16:37.902504 (+1050367us) ts_tablet_manager.cc:1120] Deleting tablet 2bfb532c9d4c41afb7acaa4831841306
0502 08:16:37.902515 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:37.902603 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1050202}
I20240502 08:16:37.903290 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 202) took 1051 ms. Trace:
I20240502 08:16:37.903393 24180 rpcz_store.cc:276] 0502 08:16:36.851851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:36.852322 (+ 471us) service_pool.cc:225] Handling call
0502 08:16:37.903104 (+1050782us) ts_tablet_manager.cc:1120] Deleting tablet 6bb2d0a4898544f182ab8070d29a7940
0502 08:16:37.903114 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:37.903206 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1050612}
I20240502 08:16:37.903923 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 199) took 1052 ms. Trace:
W20240502 08:16:37.904067 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 45f02efa659e457ab57bc32ed3f66bb9 has no consensus state
W20240502 08:16:37.905104 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 01675f637a5a4645949abfd87b3d09a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01675f637a5a4645949abfd87b3d09a2 already in progress: creating tablet
W20240502 08:16:37.905373 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 2bfb532c9d4c41afb7acaa4831841306 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2bfb532c9d4c41afb7acaa4831841306 already in progress: creating tablet
W20240502 08:16:37.905588 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 6bb2d0a4898544f182ab8070d29a7940 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bb2d0a4898544f182ab8070d29a7940 already in progress: creating tablet
I20240502 08:16:37.907722 24175 tablet_bootstrap.cc:492] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:37.907887 24175 ts_tablet_manager.cc:1367] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.102s user 0.008s sys 0.005s
I20240502 08:16:37.911491 24355 raft_consensus.cc:482] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.911670 24355 raft_consensus.cc:504] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:37.911897 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 128c88af54d24ffd895e9de19cdb4c8a has no consensus state
I20240502 08:16:37.912324 24355 leader_election.cc:288] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:37.914180 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 4f1736b496354e2dadd844242d3867f8
I20240502 08:16:37.914615 23608 catalog_manager.cc:6554] ReplaceTablet: tablet 4f1736b496354e2dadd844242d3867f8 deleted and replaced by tablet 6746c25111d04f20a6f30ce0838d71f5
I20240502 08:16:37.916105 23603 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 768167b977c14b53a7537879e349d354 (table default.Table_d10cdf44509d4b4aabd9fb5c4a035f4b [id=ab1669d2a12f4187809987ed41fa0379]). Expected version 2 got 1
I20240502 08:16:37.917044 24410 tablet.cc:1722] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:37.920229 23601 catalog_manager.cc:5484] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:37.920740 23601 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 1ce5ccfd29724dc49afd0892b99f3169 (table default.Table_bdbe807258a5454795eaff8c27bb17dd [id=3a9d7e8fe2c84cb4942fee9053a012b0]). Expected version 1 got 0
I20240502 08:16:37.911406 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet a15ff3120b1943a69ddb609d0575ae1a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:37.911599 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet d8324e9b796047428e2fe918f07f9b32 with delete_type TABLET_DATA_DELETED (replaced by tablet 4302fec5c51e4c6c841710bab50b2997) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:37.911746 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f7518b1e36147a9b9f0c58bca17061a with delete_type TABLET_DATA_DELETED (replaced by tablet d8f7b5f018d84680ac5533bf99102d31) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:37.923318 24180 rpcz_store.cc:276] 0502 08:16:36.851476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:36.852587 (+ 1111us) service_pool.cc:225] Handling call
0502 08:16:37.903758 (+1051171us) ts_tablet_manager.cc:1120] Deleting tablet 6788ffd2eb2248c3b84e474de56c37a9
0502 08:16:37.903768 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:37.903858 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1050936}
I20240502 08:16:37.915097 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1736b496354e2dadd844242d3867f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 6746c25111d04f20a6f30ce0838d71f5) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:37.923416 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7aa03e73acd4a9da5574b9ba93415ae" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:37.912921 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da4be1d381a64b7baeb1c60cb17d924a" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:37.924082 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da4be1d381a64b7baeb1c60cb17d924a" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:37.924278 23993 leader_election.cc:341] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:37.924635 23996 leader_election.cc:341] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:37.924782 23996 leader_election.cc:302] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:37.925850 24355 raft_consensus.cc:2726] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:37.926191 24175 raft_consensus.cc:348] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.926275 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8c4dfc5c68e84b6ab815e9f333143678" } modify_external_catalogs: true
I20240502 08:16:37.926474 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8c4dfc5c68e84b6ab815e9f333143678" } modify_external_catalogs: true
W20240502 08:16:37.927816 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 6788ffd2eb2248c3b84e474de56c37a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6788ffd2eb2248c3b84e474de56c37a9 already in progress: creating tablet
I20240502 08:16:37.928267 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_becb097785284ffba69cdbe40e6480ab" } modify_external_catalogs: true
I20240502 08:16:37.928447 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_becb097785284ffba69cdbe40e6480ab" } modify_external_catalogs: true
I20240502 08:16:37.929057 24180 tablet_replica.cc:330] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:37.929250 24180 raft_consensus.cc:2229] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.929425 24180 raft_consensus.cc:2258] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:37.929651 24175 raft_consensus.cc:374] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.929772 24175 raft_consensus.cc:729] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.930003 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4e40fa1cf0ab4dedb2c6aadb2418901b" } modify_external_catalogs: true
I20240502 08:16:37.930039 24175 consensus_queue.cc:260] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.930155 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4e40fa1cf0ab4dedb2c6aadb2418901b" } modify_external_catalogs: true
I20240502 08:16:37.930701 24175 ts_tablet_manager.cc:1398] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.023s user 0.003s sys 0.000s
I20240502 08:16:37.931764 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2849e243f1a94b139d3234f0c8e3c755" } modify_external_catalogs: true
I20240502 08:16:37.931960 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2849e243f1a94b139d3234f0c8e3c755" } modify_external_catalogs: true
I20240502 08:16:37.932243 24175 tablet_bootstrap.cc:492] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:37.933176 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6f4d25ef5b90455ca233af7b70b9073a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:37.934036 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f4d25ef5b90455ca233af7b70b9073a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:37.935339 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9507475756c3401fa75fc34d4a4b7814" } modify_external_catalogs: true
I20240502 08:16:37.935506 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9507475756c3401fa75fc34d4a4b7814" } modify_external_catalogs: true
I20240502 08:16:37.936146 24175 tablet_bootstrap.cc:654] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.936162 24180 ts_tablet_manager.cc:1875] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.938413 24174 tablet_bootstrap.cc:492] T 9b1eb762780c450bbaf380d17db564b6 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:37.939026 24174 ts_tablet_manager.cc:1367] T 9b1eb762780c450bbaf380d17db564b6 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.126s user 0.004s sys 0.000s
I20240502 08:16:37.939231 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet 405ea360814e4593a289d753821277ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.939569 24181 tablet_replica.cc:330] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:37.939754 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 with delete_type TABLET_DATA_DELETED (replaced by tablet b811b81ddf2b4dc9b5e5f80331bff294) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.939815 24181 raft_consensus.cc:2229] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.939843 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet a7c3800fb2784a1dba12d01c56c226e5 with delete_type TABLET_DATA_DELETED (replaced by tablet f4b291bdb81a4efbac7f6691e5577bbe) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.940008 24181 raft_consensus.cc:2258] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:37.940152 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df565c1c6564efd88fa557c50522d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.939538 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet c709efdd2ce74b5a8ab7278b402ebfe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.940505 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet d8324e9b796047428e2fe918f07f9b32 with delete_type TABLET_DATA_DELETED (replaced by tablet 4302fec5c51e4c6c841710bab50b2997) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.940732 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet 05cbbf9614094afaa6f7cee0af597242 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.940665 24174 raft_consensus.cc:348] T 9b1eb762780c450bbaf380d17db564b6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.940856 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 5160fda2643e4b80bc401bc48f97eff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.940912 24174 raft_consensus.cc:374] T 9b1eb762780c450bbaf380d17db564b6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.940980 24174 raft_consensus.cc:729] T 9b1eb762780c450bbaf380d17db564b6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.941162 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1736b496354e2dadd844242d3867f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 6746c25111d04f20a6f30ce0838d71f5) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.941254 24174 consensus_queue.cc:260] T 9b1eb762780c450bbaf380d17db564b6 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.941632 24181 ts_tablet_manager.cc:1875] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.942395 24408 tablet.cc:1722] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240502 08:16:37.950713 24343 raft_consensus.cc:482] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.951020 24343 raft_consensus.cc:504] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.952052 24343 leader_election.cc:288] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:37.952680 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8324e9b796047428e2fe918f07f9b32" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:37.952913 24060 raft_consensus.cc:2454] T d8324e9b796047428e2fe918f07f9b32 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:37.952962 24174 ts_tablet_manager.cc:1398] T 9b1eb762780c450bbaf380d17db564b6 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240502 08:16:37.953282 23723 leader_election.cc:302] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:37.954130 24410 tablet.cc:1722] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:37.955585 24457 raft_consensus.cc:2781] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:37.955678 24457 raft_consensus.cc:482] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:37.955780 24457 raft_consensus.cc:3037] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.957257 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet c316e1aeab224634b760641a4ce188c8 with delete_type TABLET_DATA_DELETED (replaced by tablet d5bab1b7c48248a7973c27d0a865b34b) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.957862 24181 ts_tablet_manager.cc:1888] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:37.957996 24181 log.cc:1196] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/405ea360814e4593a289d753821277ff
I20240502 08:16:37.958338 24181 ts_tablet_manager.cc:1909] T 405ea360814e4593a289d753821277ff P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:37.958338 24457 raft_consensus.cc:504] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.958971 24457 leader_election.cc:288] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:37.959226 24181 tablet_replica.cc:330] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:37.959432 24181 raft_consensus.cc:2229] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.959605 24181 raft_consensus.cc:2258] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:37.959681 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet da4be1d381a64b7baeb1c60cb17d924a
I20240502 08:16:37.960072 23605 catalog_manager.cc:6554] ReplaceTablet: tablet da4be1d381a64b7baeb1c60cb17d924a deleted and replaced by tablet fbef606ce7ac499d9bd84b49d804621c
I20240502 08:16:37.960163 24181 ts_tablet_manager.cc:1875] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.961344 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet da4be1d381a64b7baeb1c60cb17d924a with delete_type TABLET_DATA_DELETED (replaced by tablet fbef606ce7ac499d9bd84b49d804621c) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:37.961714 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 4a57ed188315493198e95cefb6907ab6
I20240502 08:16:37.962044 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 319fabf91458438ba5e40ab68b323055
I20240502 08:16:37.962494 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8324e9b796047428e2fe918f07f9b32" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:37.962762 24059 raft_consensus.cc:3037] T d8324e9b796047428e2fe918f07f9b32 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.962986 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 319fabf91458438ba5e40ab68b323055 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:37.963577 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 405ea360814e4593a289d753821277ff (table default.Table_3602f20179c64041a63ab185d9359de6 [id=6d4996e259e34d51b7ee13c8efa7e2bd]) successfully deleted
I20240502 08:16:37.964581 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a57ed188315493198e95cefb6907ab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:37.966254 24059 raft_consensus.cc:2454] T d8324e9b796047428e2fe918f07f9b32 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:37.966728 23607 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 768167b977c14b53a7537879e349d354 (table default.Table_d10cdf44509d4b4aabd9fb5c4a035f4b [id=ab1669d2a12f4187809987ed41fa0379]). Expected version 2 got 1
I20240502 08:16:37.969983 24175 tablet_bootstrap.cc:492] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:37.970598 24181 ts_tablet_manager.cc:1888] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:37.970641 24492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.126:43265 every 8 connection(s)
I20240502 08:16:37.971019 24181 log.cc:1196] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/4ea2dee2d5914cf0bf4b37d16865e9b0
I20240502 08:16:37.971230 24175 ts_tablet_manager.cc:1367] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20240502 08:16:37.971525 23723 leader_election.cc:302] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:37.972062 24181 ts_tablet_manager.cc:1909] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:37.972082 24175 raft_consensus.cc:348] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.972316 24175 raft_consensus.cc:374] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.972410 24175 raft_consensus.cc:729] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.972625 24175 consensus_queue.cc:260] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.970978 24241 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.126:43265
I20240502 08:16:37.973121 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 319fabf91458438ba5e40ab68b323055 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.973372 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a57ed188315493198e95cefb6907ab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.973632 24343 raft_consensus.cc:2781] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:37.973815 24343 raft_consensus.cc:686] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:37.974123 24343 consensus_queue.cc:237] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:37.974951 24241 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data/info.pb
I20240502 08:16:37.976915 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet da4be1d381a64b7baeb1c60cb17d924a with delete_type TABLET_DATA_DELETED (replaced by tablet fbef606ce7ac499d9bd84b49d804621c) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.977452 24174 tablet_bootstrap.cc:492] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:37.980587 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet bd4a31e5aa834adeb9f59267247c6980
I20240502 08:16:37.981966 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4a31e5aa834adeb9f59267247c6980 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:37.982137 23429 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2vVFYb/build/asan/bin/kudu as pid 24241
I20240502 08:16:37.982523 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 98a96292b28a4b0aad88fe0a96730a68
I20240502 08:16:37.983409 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 98a96292b28a4b0aad88fe0a96730a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:37.987265 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet 98a96292b28a4b0aad88fe0a96730a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.987682 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4a31e5aa834adeb9f59267247c6980 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:37.989501 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 (table default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418]) successfully deleted
I20240502 08:16:37.991258 24181 tablet_replica.cc:330] T a7c3800fb2784a1dba12d01c56c226e5 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:37.991729 24181 raft_consensus.cc:2229] T a7c3800fb2784a1dba12d01c56c226e5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:37.992242 24181 raft_consensus.cc:2258] T a7c3800fb2784a1dba12d01c56c226e5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:37.993432 24181 ts_tablet_manager.cc:1875] T a7c3800fb2784a1dba12d01c56c226e5 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:37.999703 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet b742c51f6204449a804889815604d5cb
I20240502 08:16:38.000654 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet b742c51f6204449a804889815604d5cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.005434 24410 tablet.cc:1722] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240502 08:16:38.007217 24174 tablet_bootstrap.cc:654] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.008414 23601 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_d10cdf44509d4b4aabd9fb5c4a035f4b [id=ab1669d2a12f4187809987ed41fa0379] alter complete (version 2)
I20240502 08:16:38.009724 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet b742c51f6204449a804889815604d5cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:38.010980 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 485c772245fe48a39fc6ceb6377ac0b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.011458 24175 ts_tablet_manager.cc:1398] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20240502 08:16:38.011824 24175 tablet_bootstrap.cc:492] T c709efdd2ce74b5a8ab7278b402ebfe9 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:38.012130 24181 ts_tablet_manager.cc:1888] T a7c3800fb2784a1dba12d01c56c226e5 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.012257 24181 log.cc:1196] T a7c3800fb2784a1dba12d01c56c226e5 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/a7c3800fb2784a1dba12d01c56c226e5
I20240502 08:16:38.012712 24181 ts_tablet_manager.cc:1909] T a7c3800fb2784a1dba12d01c56c226e5 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:38.013298 24175 tablet_bootstrap.cc:654] T c709efdd2ce74b5a8ab7278b402ebfe9 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.013922 24181 tablet_replica.cc:330] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:38.014338 24181 raft_consensus.cc:2229] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.014595 24181 raft_consensus.cc:2258] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.014694 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet a7c3800fb2784a1dba12d01c56c226e5 (table default.Table_295f41167e6e481ea2443d5f8d02a998 [id=73559755c4594bcfa591d054964154ae]) successfully deleted
I20240502 08:16:38.015537 24181 ts_tablet_manager.cc:1875] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.023057 24181 ts_tablet_manager.cc:1888] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.023375 24181 log.cc:1196] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/1df565c1c6564efd88fa557c50522d7e
I20240502 08:16:38.023794 24181 ts_tablet_manager.cc:1909] T 1df565c1c6564efd88fa557c50522d7e P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:38.023907 24173 tablet_bootstrap.cc:492] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:38.024081 24173 ts_tablet_manager.cc:1367] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.182s user 0.038s sys 0.017s
I20240502 08:16:38.024617 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 5738cd77bbe147b49215ba9639d7f547 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:38.025049 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 203ed05d0bdc4b1f99207423b7fc848b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:38.025224 24173 raft_consensus.cc:348] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.025502 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet f36ceb0222ef473a828ca795f393174d with delete_type TABLET_DATA_DELETED (replaced by tablet df9402cf8abc4b4d893d42f9a3a99016) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:38.025648 24173 raft_consensus.cc:374] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.025748 24173 raft_consensus.cc:729] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.025887 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet b63fe5b6d4db4750800ecd429fa97ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:38.026064 24173 consensus_queue.cc:260] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.026330 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet ba3d4b4584dd45b58ff0daa4f1c09a0a with delete_type TABLET_DATA_DELETED (replaced by tablet bd4a31e5aa834adeb9f59267247c6980) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:38.027084 24173 ts_tablet_manager.cc:1398] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20240502 08:16:38.028246 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 223c92ae8f63402f8643e03d914e8e7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.031437 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 7a97427e819b4826b1ed89e6d021c8d4 (DEFAULT_TABLE table=default.Table_e66f5d1a72b347f7a7b39bddbf74b263 [id=9b0006a9ac9b437fa608c587d73b09f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.030824 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 7a97427e819b4826b1ed89e6d021c8d4 (DEFAULT_TABLE table=default.Table_e66f5d1a72b347f7a7b39bddbf74b263 [id=9b0006a9ac9b437fa608c587d73b09f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.034591 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a97427e819b4826b1ed89e6d021c8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.033459 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2f863e7f34fb4bb7a9770adbca87da84" } new_table_name: "default.Table_81d96be059774b5bbaf8c8d21b2935ce" modify_external_catalogs: true
I20240502 08:16:38.028000 24173 tablet_bootstrap.cc:492] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:38.038708 23604 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_de4adf089f7347969051330f6f43e162 [id=eca4080e8ebf46e9b58c85ee1d8675ca] alter complete (version 4)
I20240502 08:16:38.038942 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_95df2e76edc746a08164b4ff25809909"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.039460 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95df2e76edc746a08164b4ff25809909 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.039651 24173 tablet_bootstrap.cc:654] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.040794 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9a479fbf2ac24ac1838fc0599feb088a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.041523 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a479fbf2ac24ac1838fc0599feb088a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.042369 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9837c3c4dad548c18b554594be454c46" } new_table_name: "default.Table_35b808b0487f463dbcfa4a773a2a20f0" modify_external_catalogs: true
I20240502 08:16:38.044514 24181 tablet_replica.cc:330] T c709efdd2ce74b5a8ab7278b402ebfe9 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:38.045367 24181 raft_consensus.cc:2229] T c709efdd2ce74b5a8ab7278b402ebfe9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.046084 24181 raft_consensus.cc:2258] T c709efdd2ce74b5a8ab7278b402ebfe9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.047277 24181 ts_tablet_manager.cc:1875] T c709efdd2ce74b5a8ab7278b402ebfe9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.047288 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b1eb762780c450bbaf380d17db564b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.047734 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet c08228fcd1304528b9446003851f6c70 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:38.048816 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_59f82dbde55b406ba5271520c5bab104"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.049463 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59f82dbde55b406ba5271520c5bab104 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.050673 24180 ts_tablet_manager.cc:1888] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.050628 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_cda7da2a134f43a8a3c79a6609c3e52e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.051280 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cda7da2a134f43a8a3c79a6609c3e52e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.052480 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5dcb3ffd44ea42de93aad8dcced74ed4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.053002 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5dcb3ffd44ea42de93aad8dcced74ed4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.053158 24180 log.cc:1196] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/ab70513985b54416856a29b32cecae4d
I20240502 08:16:38.053751 24180 ts_tablet_manager.cc:1909] T ab70513985b54416856a29b32cecae4d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:38.053802 24181 ts_tablet_manager.cc:1888] T c709efdd2ce74b5a8ab7278b402ebfe9 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.054157 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 1df565c1c6564efd88fa557c50522d7e (table default.Table_2107771cb030402280a54f03d8b2bce4 [id=0a35939b821f468c9c7c86425fd688a5]) successfully deleted
I20240502 08:16:38.054795 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet e9592a041a9543e8b41f043927807131 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:38.055842 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 203) took 1182 ms. Trace:
I20240502 08:16:38.055765 24173 tablet_bootstrap.cc:492] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:38.055975 24180 rpcz_store.cc:276] 0502 08:16:36.873363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:36.873412 (+ 49us) service_pool.cc:225] Handling call
0502 08:16:37.928959 (+1055547us) ts_tablet_manager.cc:1120] Deleting tablet ab70513985b54416856a29b32cecae4d
0502 08:16:37.928969 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:37.979952 (+ 50983us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:38.050532 (+ 70580us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:38.055805 (+ 5273us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1055301}
I20240502 08:16:38.056124 24173 ts_tablet_manager.cc:1367] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.005s
I20240502 08:16:38.056455 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet ab70513985b54416856a29b32cecae4d (table default.Table_d9b6d863d79341ebb994770c34ae2174 [id=259cb14e2ef6427f8dfba46f188050dd]) successfully deleted
I20240502 08:16:38.056921 24173 raft_consensus.cc:348] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.057142 24173 raft_consensus.cc:374] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.057240 24173 raft_consensus.cc:729] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.057452 24173 consensus_queue.cc:260] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.058041 24173 ts_tablet_manager.cc:1398] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:38.058400 24173 tablet_bootstrap.cc:492] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:38.060042 24173 tablet_bootstrap.cc:654] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.062610 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet d5bab1b7c48248a7973c27d0a865b34b with delete_type TABLET_DATA_DELETED (replaced by tablet 7c374f42f353497fa1d745164e7fbf61) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:38.062912 24181 log.cc:1196] T c709efdd2ce74b5a8ab7278b402ebfe9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/c709efdd2ce74b5a8ab7278b402ebfe9
I20240502 08:16:38.063521 24181 ts_tablet_manager.cc:1909] T c709efdd2ce74b5a8ab7278b402ebfe9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:38.064772 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 with delete_type TABLET_DATA_DELETED (replaced by tablet b811b81ddf2b4dc9b5e5f80331bff294) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:38.065979 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5d28098dcbc4761a34ee65769d6fd7c consensus state has no leader
I20240502 08:16:38.032821 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a97427e819b4826b1ed89e6d021c8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.070011 24181 tablet_replica.cc:330] T d8324e9b796047428e2fe918f07f9b32 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:38.070189 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet c709efdd2ce74b5a8ab7278b402ebfe9 (table default.Table_34b1b262bb424bde8bd994ada1f03434 [id=322a04a326c24fc4ad2c3f1384db0fd3]) successfully deleted
I20240502 08:16:38.070472 24181 raft_consensus.cc:2229] T d8324e9b796047428e2fe918f07f9b32 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.070657 24181 raft_consensus.cc:2258] T d8324e9b796047428e2fe918f07f9b32 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.071254 24181 ts_tablet_manager.cc:1875] T d8324e9b796047428e2fe918f07f9b32 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.071451 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1e208733844829b13222e00a432439 (DEFAULT_TABLE table=default.Table_7c04460336ea48e79db571d95cdb1bcc [id=8ea9f7fff43846dfa8310549e8e57f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.071624 24106 heartbeater.cc:347] Connected to a master server at 127.22.225.126:43265
I20240502 08:16:38.071861 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a1e208733844829b13222e00a432439. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.076326 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1e208733844829b13222e00a432439 (DEFAULT_TABLE table=default.Table_7c04460336ea48e79db571d95cdb1bcc [id=8ea9f7fff43846dfa8310549e8e57f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.077031 24173 tablet_bootstrap.cc:492] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:38.077075 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a1e208733844829b13222e00a432439. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.077055 24493 sys_catalog.cc:263] Verifying existing consensus state
I20240502 08:16:38.077268 24173 ts_tablet_manager.cc:1367] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.002s
I20240502 08:16:38.078527 24173 raft_consensus.cc:348] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.078838 24173 raft_consensus.cc:374] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.079092 24173 raft_consensus.cc:729] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.079388 24173 consensus_queue.cc:260] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.079459 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd8338500f54604abcd6d42702c2eca (DEFAULT_TABLE table=default.Table_482a1b0b7a064e02805ac3b86fc62f99 [id=3c1d614616374739aeb928e797479df4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.079910 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cd8338500f54604abcd6d42702c2eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.080323 24173 ts_tablet_manager.cc:1398] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:16:38.080701 24173 tablet_bootstrap.cc:492] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:38.080863 23834 heartbeater.cc:347] Connected to a master server at 127.22.225.126:43265
I20240502 08:16:38.082518 24173 tablet_bootstrap.cc:654] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.085770 24493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077: Bootstrap starting.
I20240502 08:16:38.087507 24181 ts_tablet_manager.cc:1888] T d8324e9b796047428e2fe918f07f9b32 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.088543 24181 log.cc:1196] T d8324e9b796047428e2fe918f07f9b32 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/d8324e9b796047428e2fe918f07f9b32
I20240502 08:16:38.089126 24181 ts_tablet_manager.cc:1909] T d8324e9b796047428e2fe918f07f9b32 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:38.089510 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 8633864e248f416cb651da3b0c2aa936 with delete_type TABLET_DATA_DELETED (replaced by tablet 9761df0bbdee44af90135bffb8b5828f) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:38.096760 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f816cb42e01045ae9df44baeb445f439" } new_table_name: "default.Table_d67be3894f494223a45d8c5a2c3e6113" modify_external_catalogs: true
I20240502 08:16:38.097615 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 208) took 1168 ms. Trace:
I20240502 08:16:38.098778 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd8338500f54604abcd6d42702c2eca (DEFAULT_TABLE table=default.Table_482a1b0b7a064e02805ac3b86fc62f99 [id=3c1d614616374739aeb928e797479df4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.099275 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cd8338500f54604abcd6d42702c2eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.101619 24180 rpcz_store.cc:276] 0502 08:16:36.929376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:36.929435 (+ 59us) service_pool.cc:225] Handling call
0502 08:16:38.097384 (+1167949us) ts_tablet_manager.cc:1120] Deleting tablet c470d420c45843a69a63d9800b7cfc97
0502 08:16:38.097396 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:38.097535 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1167688}
W20240502 08:16:38.102830 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet c470d420c45843a69a63d9800b7cfc97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c470d420c45843a69a63d9800b7cfc97 already in progress: creating tablet
I20240502 08:16:38.103456 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 210) took 1166 ms. Trace:
I20240502 08:16:38.104033 24180 rpcz_store.cc:276] 0502 08:16:36.936644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:36.937226 (+ 582us) service_pool.cc:225] Handling call
0502 08:16:38.103282 (+1166056us) ts_tablet_manager.cc:1120] Deleting tablet e0af5f4daf3149738647a08219efb245
0502 08:16:38.103295 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:38.103404 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1163979}
W20240502 08:16:38.105535 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e0af5f4daf3149738647a08219efb245 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0af5f4daf3149738647a08219efb245 already in progress: creating tablet
I20240502 08:16:38.106093 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 209) took 1169 ms. Trace:
I20240502 08:16:38.106652 24493 log.cc:824] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:38.106660 24180 rpcz_store.cc:276] 0502 08:16:36.936311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:36.936522 (+ 211us) service_pool.cc:225] Handling call
0502 08:16:38.105945 (+1169423us) ts_tablet_manager.cc:1120] Deleting tablet ba3d4b4584dd45b58ff0daa4f1c09a0a
0502 08:16:38.105955 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:38.106053 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1165539}
I20240502 08:16:38.108832 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 211) took 1167 ms. Trace:
I20240502 08:16:38.109371 24180 rpcz_store.cc:276] 0502 08:16:36.941387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:36.941858 (+ 471us) service_pool.cc:225] Handling call
0502 08:16:38.108674 (+1166816us) ts_tablet_manager.cc:1120] Deleting tablet 171406810543435e832a47ad5b3c80a4
0502 08:16:38.108686 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:38.108797 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1166551}
W20240502 08:16:38.110476 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 171406810543435e832a47ad5b3c80a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 171406810543435e832a47ad5b3c80a4 already in progress: creating tablet
I20240502 08:16:38.109411 23761 tablet_service.cc:1467] Processing CreateTablet for tablet e549cad1b3be4b28abb0bf5af720eb52 (DEFAULT_TABLE table=default.Table_b461b560f3094407a1766253cfc2ece7 [id=4b9348e7a75242788274eee8f4c94838]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.111099 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e549cad1b3be4b28abb0bf5af720eb52. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:38.112167 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e686db13bb7541c69ae88f0fb7e27061 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e686db13bb7541c69ae88f0fb7e27061 already in progress: creating tablet
I20240502 08:16:38.112426 24355 raft_consensus.cc:482] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.112617 24355 raft_consensus.cc:504] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:38.113314 24355 leader_election.cc:288] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:38.113708 24483 raft_consensus.cc:482] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.113853 24483 raft_consensus.cc:504] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.114003 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08183523bf4a4d1c9020eb693aa07878" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:38.114645 24180 tablet_replica.cc:330] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:38.114687 24483 leader_election.cc:288] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:38.114794 23993 leader_election.cc:341] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.114957 24180 raft_consensus.cc:2229] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.114964 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08183523bf4a4d1c9020eb693aa07878" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:38.115218 24180 raft_consensus.cc:2258] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:38.115713 23996 leader_election.cc:341] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.115830 23996 leader_election.cc:302] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:38.115955 24180 ts_tablet_manager.cc:1875] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.116094 24355 raft_consensus.cc:2726] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:38.116843 23861 leader_election.cc:341] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 405ea360814e4593a289d753821277ff
W20240502 08:16:38.119525 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6c52f1cae3a4ad38a9f95bfaf055e76 consensus state has no leader
I20240502 08:16:38.120887 24035 tablet_service.cc:1467] Processing CreateTablet for tablet e549cad1b3be4b28abb0bf5af720eb52 (DEFAULT_TABLE table=default.Table_b461b560f3094407a1766253cfc2ece7 [id=4b9348e7a75242788274eee8f4c94838]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.121382 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e549cad1b3be4b28abb0bf5af720eb52. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:38.130510 23858 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 267772us
I20240502 08:16:38.131381 23723 leader_election.cc:302] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:38.131704 24329 raft_consensus.cc:2781] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:38.131862 24329 raft_consensus.cc:686] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:38.132160 24329 consensus_queue.cc:237] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.132548 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7aa03e73acd4a9da5574b9ba93415ae" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:38.132776 23787 raft_consensus.cc:2379] T c7aa03e73acd4a9da5574b9ba93415ae P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in this term.
I20240502 08:16:38.134335 24174 tablet_bootstrap.cc:492] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:38.134626 24174 ts_tablet_manager.cc:1367] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.157s user 0.002s sys 0.006s
I20240502 08:16:38.134889 23971 heartbeater.cc:347] Connected to a master server at 127.22.225.126:43265
I20240502 08:16:38.141777 23834 heartbeater.cc:347] Connected to a master server at 127.22.225.126:43265
I20240502 08:16:38.142124 24481 raft_consensus.cc:482] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.142292 24481 raft_consensus.cc:504] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.142203 24174 raft_consensus.cc:348] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.142566 24174 raft_consensus.cc:374] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.142661 24174 raft_consensus.cc:729] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.142911 24174 consensus_queue.cc:260] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.143023 24481 leader_election.cc:288] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:38.143458 24174 ts_tablet_manager.cc:1398] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 08:16:38.143759 24174 tablet_bootstrap.cc:492] T 1df565c1c6564efd88fa557c50522d7e P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
W20240502 08:16:38.143862 23861 leader_election.cc:341] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: d8324e9b796047428e2fe918f07f9b32
I20240502 08:16:38.145427 24174 tablet_bootstrap.cc:654] T 1df565c1c6564efd88fa557c50522d7e P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.146837 24175 tablet_bootstrap.cc:492] T c709efdd2ce74b5a8ab7278b402ebfe9 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:38.147019 24175 ts_tablet_manager.cc:1367] T c709efdd2ce74b5a8ab7278b402ebfe9 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.135s user 0.005s sys 0.005s
I20240502 08:16:38.147460 24180 ts_tablet_manager.cc:1888] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:16:38.147982 24175 raft_consensus.cc:348] T c709efdd2ce74b5a8ab7278b402ebfe9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.148288 24175 raft_consensus.cc:374] T c709efdd2ce74b5a8ab7278b402ebfe9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.148553 24175 raft_consensus.cc:729] T c709efdd2ce74b5a8ab7278b402ebfe9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.148871 24175 consensus_queue.cc:260] T c709efdd2ce74b5a8ab7278b402ebfe9 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.149732 23899 tablet_service.cc:1467] Processing CreateTablet for tablet bd4a31e5aa834adeb9f59267247c6980 (DEFAULT_TABLE table=default.Table_4e40fa1cf0ab4dedb2c6aadb2418901b [id=fd781805eb4c45f09fe6460144472816]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.150441 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4a31e5aa834adeb9f59267247c6980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.150501 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8324e9b796047428e2fe918f07f9b32" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:38.150580 24180 log.cc:1196] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/ec4028a8e66442bfabac9e587aed00bb
I20240502 08:16:38.150863 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 7a97427e819b4826b1ed89e6d021c8d4 (DEFAULT_TABLE table=default.Table_e66f5d1a72b347f7a7b39bddbf74b263 [id=9b0006a9ac9b437fa608c587d73b09f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.151072 24180 ts_tablet_manager.cc:1909] T ec4028a8e66442bfabac9e587aed00bb P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:38.151674 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet ec4028a8e66442bfabac9e587aed00bb (table default.Table_62b86da21eb84bc387b64725dabda733 [id=1e590d80e3e34e1aac8a0c7908a9d813]) successfully deleted
I20240502 08:16:38.152083 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fec81a33cad74ba3bfbc7787a7bd5829"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:38.152442 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8324e9b796047428e2fe918f07f9b32" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:38.152630 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fec81a33cad74ba3bfbc7787a7bd5829 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.152740 24483 raft_consensus.cc:482] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.153192 24483 raft_consensus.cc:504] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.152937 23925 raft_consensus.cc:2454] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:38.153894 24483 leader_election.cc:288] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:38.153988 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 8633864e248f416cb651da3b0c2aa936 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8633864e248f416cb651da3b0c2aa936 already in progress: creating tablet
W20240502 08:16:38.154712 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 8810b6f61db14fed8b12d36fed413dc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8810b6f61db14fed8b12d36fed413dc8 already in progress: creating tablet
I20240502 08:16:38.154994 23905 tablet_service.cc:1467] Processing CreateTablet for tablet c1e746013fde4de4862f213792e235ed (DEFAULT_TABLE table=default.Table_725d048135a04e32bd016e60ac974715 [id=ecfe360ec5554bdcac562a1c614e1672]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.155428 24180 tablet_replica.cc:330] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:38.155401 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet d8324e9b796047428e2fe918f07f9b32 (table default.Table_006d60eaae72439f829df411c32c1514 [id=e3f148f833bc4d99a3ab2ec8d5258507]) successfully deleted
I20240502 08:16:38.155649 24180 raft_consensus.cc:2229] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
W20240502 08:16:38.155673 23861 leader_election.cc:341] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: ab70513985b54416856a29b32cecae4d
I20240502 08:16:38.155804 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a97427e819b4826b1ed89e6d021c8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.155900 24180 pending_rounds.cc:62] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:16:38.155983 24493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=5 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:16:38.155953 24180 pending_rounds.cc:68] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156419709091840 op_type: NO_OP noop_request { }
I20240502 08:16:38.156078 24180 raft_consensus.cc:2866] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:38.156172 24180 raft_consensus.cc:2258] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.156612 24493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077: Bootstrap complete.
I20240502 08:16:38.156862 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1736b496354e2dadd844242d3867f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 6746c25111d04f20a6f30ce0838d71f5) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.157434 24180 ts_tablet_manager.cc:1875] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.161914 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0f919156226c440294ddd9a6e379cbff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.162485 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f919156226c440294ddd9a6e379cbff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.151497 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_28b3620523074c23866e772b117bf0ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.163357 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28b3620523074c23866e772b117bf0ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.164515 24180 ts_tablet_manager.cc:1888] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:38.164753 24180 log.cc:1196] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/5160fda2643e4b80bc401bc48f97eff9
I20240502 08:16:38.165297 24180 ts_tablet_manager.cc:1909] T 5160fda2643e4b80bc401bc48f97eff9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:38.166006 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 5160fda2643e4b80bc401bc48f97eff9 (table default.Table_7e2b03f5ed46441d9c62c30319eab83b [id=8db1f2345d324647881132f139888dee]) successfully deleted
I20240502 08:16:38.166448 24180 tablet_replica.cc:330] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:38.166707 24180 raft_consensus.cc:2229] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.166951 24180 raft_consensus.cc:2258] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.167598 24180 ts_tablet_manager.cc:1875] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.170657 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_70034c04442f49f6a42cb66dccfd1cc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.171180 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70034c04442f49f6a42cb66dccfd1cc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.170578 24493 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:38.171720 24493 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc8a323e6b9429eaaf4e0c1698ad077, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.172223 24329 raft_consensus.cc:482] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.172379 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cdfdcb2e5f8b4b7bbd2d75d2ea00936e" } new_table_name: "default.Table_0a4ff982972e4fdaae1583619c9d6b04" modify_external_catalogs: true
I20240502 08:16:38.172362 24329 raft_consensus.cc:504] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.172603 24493 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:38.172986 24329 leader_election.cc:288] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:38.173739 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_dae0bce7072a4d50aa63b702ee775fcf" } new_table_name: "default.Table_5e74a1a8d1ce46018625fb11e1dac26e" modify_external_catalogs: true
W20240502 08:16:38.174110 23723 leader_election.cc:341] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 4ea2dee2d5914cf0bf4b37d16865e9b0
I20240502 08:16:38.174504 24175 ts_tablet_manager.cc:1398] T c709efdd2ce74b5a8ab7278b402ebfe9 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240502 08:16:38.174834 24175 tablet_bootstrap.cc:492] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:38.174940 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e746013fde4de4862f213792e235ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.176676 24175 tablet_bootstrap.cc:654] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.184295 24174 tablet_bootstrap.cc:492] T 1df565c1c6564efd88fa557c50522d7e P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:38.185145 24174 ts_tablet_manager.cc:1367] T 1df565c1c6564efd88fa557c50522d7e P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20240502 08:16:38.186184 24174 raft_consensus.cc:348] T 1df565c1c6564efd88fa557c50522d7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.186501 24174 raft_consensus.cc:374] T 1df565c1c6564efd88fa557c50522d7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.186640 24174 raft_consensus.cc:729] T 1df565c1c6564efd88fa557c50522d7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.187834 24355 raft_consensus.cc:482] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.187453 24174 consensus_queue.cc:260] T 1df565c1c6564efd88fa557c50522d7e P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.188520 24174 ts_tablet_manager.cc:1398] T 1df565c1c6564efd88fa557c50522d7e P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:16:38.191685 24180 ts_tablet_manager.cc:1888] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.193990 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 5160fda2643e4b80bc401bc48f97eff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.194391 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f7518b1e36147a9b9f0c58bca17061a with delete_type TABLET_DATA_DELETED (replaced by tablet d8f7b5f018d84680ac5533bf99102d31) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.195585 24175 tablet_bootstrap.cc:492] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:38.195763 24175 ts_tablet_manager.cc:1367] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20240502 08:16:38.188051 23607 master_service.cc:946] ReplaceTablet: received request to replace tablet b1fac7df16fc41d7a3297055a7258270 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:38.190665 23603 master_service.cc:946] ReplaceTablet: received request to replace tablet 71aa916247c449fea718b35869a6df20 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:38.196568 24175 raft_consensus.cc:348] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.196795 24175 raft_consensus.cc:374] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.196873 24175 raft_consensus.cc:729] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.197129 24175 consensus_queue.cc:260] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.197809 24175 ts_tablet_manager.cc:1398] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:38.197973 24180 log.cc:1196] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/c316e1aeab224634b760641a4ce188c8
I20240502 08:16:38.198357 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 with delete_type TABLET_DATA_DELETED (replaced by tablet b811b81ddf2b4dc9b5e5f80331bff294) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.198410 24180 ts_tablet_manager.cc:1909] T c316e1aeab224634b760641a4ce188c8 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:38.199484 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet c316e1aeab224634b760641a4ce188c8 (table default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]) successfully deleted
I20240502 08:16:38.192119 24174 tablet_bootstrap.cc:492] T da4be1d381a64b7baeb1c60cb17d924a P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:38.200673 24509 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } } }
I20240502 08:16:38.201040 24509 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:38.202869 24493 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:38.204912 24174 tablet_bootstrap.cc:654] T da4be1d381a64b7baeb1c60cb17d924a P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.205330 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet d8324e9b796047428e2fe918f07f9b32 with delete_type TABLET_DATA_DELETED (replaced by tablet 4302fec5c51e4c6c841710bab50b2997) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.205868 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet a15ff3120b1943a69ddb609d0575ae1a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.206352 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 32150c7e130f4069a2a3a02e625bfb93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.206790 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 223c92ae8f63402f8643e03d914e8e7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:34 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.207049 24355 raft_consensus.cc:504] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:38.192896 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a286e67e441e43a1b4738cd219dbea98" } modify_external_catalogs: true
I20240502 08:16:38.207733 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a286e67e441e43a1b4738cd219dbea98" } modify_external_catalogs: true
I20240502 08:16:38.207854 24355 leader_election.cc:288] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
W20240502 08:16:38.195688 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 128c88af54d24ffd895e9de19cdb4c8a has no consensus state
I20240502 08:16:38.209856 24181 tablet_replica.cc:330] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:38.210199 24181 raft_consensus.cc:2229] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.210494 24181 raft_consensus.cc:2258] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.210614 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7bae786c8c2e44bd9e893ff90696d216" } modify_external_catalogs: true
I20240502 08:16:38.210804 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7bae786c8c2e44bd9e893ff90696d216" } modify_external_catalogs: true
I20240502 08:16:38.211238 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36ceb0222ef473a828ca795f393174d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:38.212541 23993 leader_election.cc:341] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.212764 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet c709efdd2ce74b5a8ab7278b402ebfe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.213008 24355 consensus_queue.cc:1035] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:38.214138 24181 ts_tablet_manager.cc:1875] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:38.214586 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 4302fec5c51e4c6c841710bab50b2997 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4302fec5c51e4c6c841710bab50b2997 already in progress: creating tablet
I20240502 08:16:38.215888 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 405ea360814e4593a289d753821277ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:38.215911 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet ce758c4a9258427987e8e83f579a2aa2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce758c4a9258427987e8e83f579a2aa2 already in progress: creating tablet
I20240502 08:16:38.216490 24355 raft_consensus.cc:482] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.216634 24355 raft_consensus.cc:504] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:38.216849 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 0dff0d73a933427fb1f2686bac33eba1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0dff0d73a933427fb1f2686bac33eba1 already in progress: creating tablet
I20240502 08:16:38.216929 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df565c1c6564efd88fa557c50522d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.217391 24355 leader_election.cc:288] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
W20240502 08:16:38.217875 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet c08228fcd1304528b9446003851f6c70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c08228fcd1304528b9446003851f6c70 already in progress: creating tablet
I20240502 08:16:38.218606 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8633864e248f416cb651da3b0c2aa936" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:38.218927 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 9700ea9dc58442e3ab1b7835a8c1e22c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9700ea9dc58442e3ab1b7835a8c1e22c already in progress: creating tablet
W20240502 08:16:38.219117 23993 leader_election.cc:341] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:38.220048 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e549cad1b3be4b28abb0bf5af720eb52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e549cad1b3be4b28abb0bf5af720eb52 already in progress: creating tablet
I20240502 08:16:38.220929 24180 tablet_replica.cc:330] T a15ff3120b1943a69ddb609d0575ae1a P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:38.221312 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet c316e1aeab224634b760641a4ce188c8 with delete_type TABLET_DATA_DELETED (replaced by tablet d5bab1b7c48248a7973c27d0a865b34b) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.221652 24180 raft_consensus.cc:2229] T a15ff3120b1943a69ddb609d0575ae1a P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.221985 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet ab70513985b54416856a29b32cecae4d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.222525 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet da4be1d381a64b7baeb1c60cb17d924a with delete_type TABLET_DATA_DELETED (replaced by tablet fbef606ce7ac499d9bd84b49d804621c) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.222594 24180 raft_consensus.cc:2258] T a15ff3120b1943a69ddb609d0575ae1a P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.223353 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 319fabf91458438ba5e40ab68b323055 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.223579 24180 ts_tablet_manager.cc:1875] T a15ff3120b1943a69ddb609d0575ae1a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.223731 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a57ed188315493198e95cefb6907ab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.224277 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4a31e5aa834adeb9f59267247c6980 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.224781 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 98a96292b28a4b0aad88fe0a96730a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.228140 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8633864e248f416cb651da3b0c2aa936" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:38.228765 23996 leader_election.cc:341] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.228781 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36ceb0222ef473a828ca795f393174d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:38.228891 23996 leader_election.cc:302] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:38.229322 23996 leader_election.cc:341] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.229425 23996 leader_election.cc:302] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:38.230379 24355 raft_consensus.cc:2726] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:38.230625 24355 raft_consensus.cc:2726] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:38.230963 24179 tablet_replica.cc:330] T 5160fda2643e4b80bc401bc48f97eff9 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:38.231220 24179 raft_consensus.cc:2229] T 5160fda2643e4b80bc401bc48f97eff9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.231442 24179 raft_consensus.cc:2258] T 5160fda2643e4b80bc401bc48f97eff9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.232111 24179 ts_tablet_manager.cc:1875] T 5160fda2643e4b80bc401bc48f97eff9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.238855 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_dd3c838f4a014e178351d135f5d4567d" } modify_external_catalogs: true
I20240502 08:16:38.239069 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_dd3c838f4a014e178351d135f5d4567d" } modify_external_catalogs: true
I20240502 08:16:38.240175 24179 ts_tablet_manager.cc:1888] T 5160fda2643e4b80bc401bc48f97eff9 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.240284 24179 log.cc:1196] T 5160fda2643e4b80bc401bc48f97eff9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/5160fda2643e4b80bc401bc48f97eff9
I20240502 08:16:38.240844 24175 tablet_bootstrap.cc:492] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:38.242208 23971 heartbeater.cc:347] Connected to a master server at 127.22.225.126:43265
I20240502 08:16:38.242815 24180 ts_tablet_manager.cc:1888] T a15ff3120b1943a69ddb609d0575ae1a P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.242947 24180 log.cc:1196] T a15ff3120b1943a69ddb609d0575ae1a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/a15ff3120b1943a69ddb609d0575ae1a
I20240502 08:16:38.243269 24180 ts_tablet_manager.cc:1909] T a15ff3120b1943a69ddb609d0575ae1a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:38.243788 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet a15ff3120b1943a69ddb609d0575ae1a (table default.Table_bc6b3019e44547cda285ac0021565619 [id=c08bf22592064123bb9cab28935d81a5]) successfully deleted
I20240502 08:16:38.244853 24175 tablet_bootstrap.cc:654] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.246614 24179 ts_tablet_manager.cc:1909] T 5160fda2643e4b80bc401bc48f97eff9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:38.249346 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bcd1014009b046a7ba72d1d757e69e26" } modify_external_catalogs: true
I20240502 08:16:38.249512 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bcd1014009b046a7ba72d1d757e69e26" } modify_external_catalogs: true
I20240502 08:16:38.252051 24180 tablet_replica.cc:330] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:38.252732 24180 raft_consensus.cc:2229] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:38.253500 23925 raft_consensus.cc:3037] T c7aa03e73acd4a9da5574b9ba93415ae P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:38.254559 24179 tablet_replica.cc:330] T 6f7518b1e36147a9b9f0c58bca17061a P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:38.255162 24179 raft_consensus.cc:2229] T 6f7518b1e36147a9b9f0c58bca17061a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.255328 24179 raft_consensus.cc:2258] T 6f7518b1e36147a9b9f0c58bca17061a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.255815 24179 ts_tablet_manager.cc:1875] T 6f7518b1e36147a9b9f0c58bca17061a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.257050 24355 consensus_queue.cc:1035] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:38.257961 24180 pending_rounds.cc:62] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:16:38.258487 24173 tablet_bootstrap.cc:492] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:38.258667 24173 ts_tablet_manager.cc:1367] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.178s user 0.000s sys 0.007s
I20240502 08:16:38.258633 24180 pending_rounds.cc:68] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156420507877376 op_type: NO_OP noop_request { }
I20240502 08:16:38.259128 24180 raft_consensus.cc:2866] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:38.259666 24180 raft_consensus.cc:2258] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.261078 24180 ts_tablet_manager.cc:1875] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.263986 24173 raft_consensus.cc:348] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.264330 24173 raft_consensus.cc:374] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.264461 24173 raft_consensus.cc:729] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.264784 24173 consensus_queue.cc:260] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.269971 24173 ts_tablet_manager.cc:1398] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 08:16:38.270350 24173 tablet_bootstrap.cc:492] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:38.271855 24173 tablet_bootstrap.cc:654] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.269119 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7e64e6023d104b96b6f3ae5c7963e32e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:38.272960 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet b742c51f6204449a804889815604d5cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:38.273083 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e64e6023d104b96b6f3ae5c7963e32e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.273458 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b1eb762780c450bbaf380d17db564b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.279327 24483 raft_consensus.cc:482] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.279513 24179 ts_tablet_manager.cc:1888] T 6f7518b1e36147a9b9f0c58bca17061a P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.279489 24483 raft_consensus.cc:504] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.279685 24179 log.cc:1196] T 6f7518b1e36147a9b9f0c58bca17061a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/6f7518b1e36147a9b9f0c58bca17061a
I20240502 08:16:38.280059 24179 ts_tablet_manager.cc:1909] T 6f7518b1e36147a9b9f0c58bca17061a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:38.280246 24483 leader_election.cc:288] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:38.280577 24179 tablet_replica.cc:330] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:38.280766 24179 raft_consensus.cc:2229] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.280920 24179 raft_consensus.cc:2258] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.281409 24179 ts_tablet_manager.cc:1875] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.282549 24180 ts_tablet_manager.cc:1888] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:38.284598 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_81d96be059774b5bbaf8c8d21b2935ce" } modify_external_catalogs: true
I20240502 08:16:38.285262 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_81d96be059774b5bbaf8c8d21b2935ce" } modify_external_catalogs: true
W20240502 08:16:38.286799 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 4f1736b496354e2dadd844242d3867f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f1736b496354e2dadd844242d3867f8 already in progress: creating tablet
I20240502 08:16:38.287796 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 5160fda2643e4b80bc401bc48f97eff9 (table default.Table_7e2b03f5ed46441d9c62c30319eab83b [id=8db1f2345d324647881132f139888dee]) successfully deleted
I20240502 08:16:38.288076 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 6f7518b1e36147a9b9f0c58bca17061a (table default.Table_498b372c15a04b70b87bd67b543501cf [id=38e955d6bd644d24a7c45172ab60282b]) successfully deleted
I20240502 08:16:38.284358 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8324e9b796047428e2fe918f07f9b32" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:38.287559 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea2dee2d5914cf0bf4b37d16865e9b0" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:38.288658 24180 log.cc:1196] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/d8324e9b796047428e2fe918f07f9b32
I20240502 08:16:38.288712 23787 raft_consensus.cc:2454] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:38.289093 24180 ts_tablet_manager.cc:1909] T d8324e9b796047428e2fe918f07f9b32 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:38.291415 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet d8324e9b796047428e2fe918f07f9b32 (table default.Table_006d60eaae72439f829df411c32c1514 [id=e3f148f833bc4d99a3ab2ec8d5258507]) successfully deleted
W20240502 08:16:38.293685 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b844458ae73f413cbf2113027a5af359 consensus state has no leader
I20240502 08:16:38.293797 24180 tablet_replica.cc:330] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:38.294484 24355 raft_consensus.cc:482] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.294667 24355 raft_consensus.cc:504] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:38.294896 24180 raft_consensus.cc:2229] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:38.295140 24180 pending_rounds.cc:62] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:16:38.295202 24180 pending_rounds.cc:68] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156421150576640 op_type: NO_OP noop_request { }
I20240502 08:16:38.295428 24355 leader_election.cc:288] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:38.295883 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642744194e22461f8316a78689fd99ff" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:38.296435 23993 leader_election.cc:341] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.295336 24180 raft_consensus.cc:2866] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:38.297089 24180 raft_consensus.cc:2258] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.297910 23608 master_service.cc:946] ReplaceTablet: received request to replace tablet 0882b7509a74475780eb700818aab398 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:38.298682 24180 ts_tablet_manager.cc:1875] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:38.304241 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dead13bba95f43cb809e17f6d7fba028 consensus state has no leader
I20240502 08:16:38.304519 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet ce758c4a9258427987e8e83f579a2aa2 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:38.311153 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c613372da52548cc93464839137beade" } new_table_name: "default.Table_8fe8a3cf1fbd4fe8825f078e4640f619" modify_external_catalogs: true
I20240502 08:16:38.315479 24180 ts_tablet_manager.cc:1888] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:38.318374 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cc30014a1ac45de937751eed36877a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:38.319126 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet b1fac7df16fc41d7a3297055a7258270
I20240502 08:16:38.319680 23607 catalog_manager.cc:6554] ReplaceTablet: tablet b1fac7df16fc41d7a3297055a7258270 deleted and replaced by tablet d6ce0b4b01fb49cdb51796a35cea5ced
I20240502 08:16:38.322212 24173 tablet_bootstrap.cc:492] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:38.322520 24173 ts_tablet_manager.cc:1367] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.003s
I20240502 08:16:38.323037 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fac7df16fc41d7a3297055a7258270 with delete_type TABLET_DATA_DELETED (replaced by tablet d6ce0b4b01fb49cdb51796a35cea5ced) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:38.323333 24180 log.cc:1196] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/6f7518b1e36147a9b9f0c58bca17061a
I20240502 08:16:38.321954 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fac7df16fc41d7a3297055a7258270 with delete_type TABLET_DATA_DELETED (replaced by tablet d6ce0b4b01fb49cdb51796a35cea5ced) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.316596 24174 tablet_bootstrap.cc:492] T da4be1d381a64b7baeb1c60cb17d924a P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:38.323480 24173 raft_consensus.cc:348] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.323699 24174 ts_tablet_manager.cc:1367] T da4be1d381a64b7baeb1c60cb17d924a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.132s user 0.010s sys 0.019s
I20240502 08:16:38.323731 24173 raft_consensus.cc:374] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.323853 24173 raft_consensus.cc:729] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.324290 24173 consensus_queue.cc:260] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.325404 24173 ts_tablet_manager.cc:1398] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:16:38.325795 24173 tablet_bootstrap.cc:492] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:38.326035 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d6f5ca82488a40a69121f1b779d8f874"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.329871 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6f5ca82488a40a69121f1b779d8f874 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.335438 24181 ts_tablet_manager.cc:1888] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.336293 24181 log.cc:1196] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/05cbbf9614094afaa6f7cee0af597242
I20240502 08:16:38.336735 24181 ts_tablet_manager.cc:1909] T 05cbbf9614094afaa6f7cee0af597242 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:38.337229 24181 tablet_replica.cc:330] T 5160fda2643e4b80bc401bc48f97eff9 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
W20240502 08:16:38.281639 23861 leader_election.cc:341] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 4ea2dee2d5914cf0bf4b37d16865e9b0
W20240502 08:16:38.337409 23861 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 55832.3us
I20240502 08:16:38.282007 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1e208733844829b13222e00a432439 (DEFAULT_TABLE table=default.Table_7c04460336ea48e79db571d95cdb1bcc [id=8ea9f7fff43846dfa8310549e8e57f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.337929 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 05cbbf9614094afaa6f7cee0af597242 (table default.Table_6c583799e352452a87de6faa19d88490 [id=80701c77b9b04253bba1d4438e6f2a78]) successfully deleted
I20240502 08:16:38.337976 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642744194e22461f8316a78689fd99ff" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:38.338223 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a1e208733844829b13222e00a432439. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.324559 24174 raft_consensus.cc:348] T da4be1d381a64b7baeb1c60cb17d924a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.338311 24174 raft_consensus.cc:374] T da4be1d381a64b7baeb1c60cb17d924a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.338397 24174 raft_consensus.cc:729] T da4be1d381a64b7baeb1c60cb17d924a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.338611 24174 consensus_queue.cc:260] T da4be1d381a64b7baeb1c60cb17d924a P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.339097 24174 ts_tablet_manager.cc:1398] T da4be1d381a64b7baeb1c60cb17d924a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.015s user 0.000s sys 0.000s
I20240502 08:16:38.339313 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet dcdd765895254a39b86b692b5d812c83
I20240502 08:16:38.339396 24174 tablet_bootstrap.cc:492] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:38.340768 24174 tablet_bootstrap.cc:654] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.341540 24180 ts_tablet_manager.cc:1909] T 6f7518b1e36147a9b9f0c58bca17061a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:38.282481 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 485c772245fe48a39fc6ceb6377ac0b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.342767 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 6f7518b1e36147a9b9f0c58bca17061a (table default.Table_498b372c15a04b70b87bd67b543501cf [id=38e955d6bd644d24a7c45172ab60282b]) successfully deleted
I20240502 08:16:38.282940 23893 tablet_service.cc:1467] Processing CreateTablet for tablet e549cad1b3be4b28abb0bf5af720eb52 (DEFAULT_TABLE table=default.Table_b461b560f3094407a1766253cfc2ece7 [id=4b9348e7a75242788274eee8f4c94838]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.342975 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet dcdd765895254a39b86b692b5d812c83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.343255 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e549cad1b3be4b28abb0bf5af720eb52. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:38.344668 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 4f1736b496354e2dadd844242d3867f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f1736b496354e2dadd844242d3867f8 already in progress: creating tablet
I20240502 08:16:38.345494 24180 tablet_replica.cc:330] T da4be1d381a64b7baeb1c60cb17d924a P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:38.345687 24180 raft_consensus.cc:2229] T da4be1d381a64b7baeb1c60cb17d924a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.285072 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet c709efdd2ce74b5a8ab7278b402ebfe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.345863 24180 raft_consensus.cc:2258] T da4be1d381a64b7baeb1c60cb17d924a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.288007 23858 leader_election.cc:302] T c7aa03e73acd4a9da5574b9ba93415ae P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:38.288467 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd8338500f54604abcd6d42702c2eca (DEFAULT_TABLE table=default.Table_482a1b0b7a064e02805ac3b86fc62f99 [id=3c1d614616374739aeb928e797479df4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.352808 24481 raft_consensus.cc:2726] T c7aa03e73acd4a9da5574b9ba93415ae P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:38.352895 23858 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 64907.2us
I20240502 08:16:38.352975 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cd8338500f54604abcd6d42702c2eca. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:38.354116 23858 leader_election.cc:341] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 405ea360814e4593a289d753821277ff
I20240502 08:16:38.354333 23858 leader_election.cc:302] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:38.354638 24481 raft_consensus.cc:2726] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:38.354800 23858 leader_election.cc:341] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: ab70513985b54416856a29b32cecae4d
I20240502 08:16:38.354969 23858 leader_election.cc:302] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:38.355218 24481 raft_consensus.cc:2726] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:38.355376 23858 leader_election.cc:341] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: Tablet not RUNNING: STOPPED
I20240502 08:16:38.355536 23858 leader_election.cc:302] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:38.355811 24481 raft_consensus.cc:2726] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:38.355947 23858 leader_election.cc:302] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
W20240502 08:16:38.356182 23858 raft_consensus.cc:2670] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:16:38.356755 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea2dee2d5914cf0bf4b37d16865e9b0" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:38.310941 24483 raft_consensus.cc:482] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.357239 24483 raft_consensus.cc:504] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.357884 24483 leader_election.cc:288] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:38.358062 24417 tablet.cc:1722] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240502 08:16:38.314683 24175 tablet_bootstrap.cc:492] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:38.358495 24175 ts_tablet_manager.cc:1367] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.118s user 0.004s sys 0.000s
I20240502 08:16:38.359176 24175 raft_consensus.cc:348] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.359330 24175 raft_consensus.cc:374] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.359412 24175 raft_consensus.cc:729] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.359611 24175 consensus_queue.cc:260] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.360097 24175 ts_tablet_manager.cc:1398] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:16:38.360385 24175 tablet_bootstrap.cc:492] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:38.362248 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fac7df16fc41d7a3297055a7258270 with delete_type TABLET_DATA_DELETED (replaced by tablet d6ce0b4b01fb49cdb51796a35cea5ced) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.363771 24181 raft_consensus.cc:2229] T 5160fda2643e4b80bc401bc48f97eff9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.364012 24181 raft_consensus.cc:2258] T 5160fda2643e4b80bc401bc48f97eff9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.366086 24180 ts_tablet_manager.cc:1875] T da4be1d381a64b7baeb1c60cb17d924a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:38.366966 23723 leader_election.cc:341] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.367126 23723 leader_election.cc:302] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:38.367908 24489 raft_consensus.cc:2726] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:38.369096 23861 leader_election.cc:341] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 32150c7e130f4069a2a3a02e625bfb93
I20240502 08:16:38.369684 24181 ts_tablet_manager.cc:1875] T 5160fda2643e4b80bc401bc48f97eff9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.377048 24175 tablet_bootstrap.cc:654] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.378123 24180 ts_tablet_manager.cc:1888] T da4be1d381a64b7baeb1c60cb17d924a P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.380739 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet dcdd765895254a39b86b692b5d812c83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.381294 24179 ts_tablet_manager.cc:1888] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.381425 24179 log.cc:1196] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/4ea2dee2d5914cf0bf4b37d16865e9b0
W20240502 08:16:38.381580 23858 leader_election.cc:341] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 32150c7e130f4069a2a3a02e625bfb93
I20240502 08:16:38.381731 23858 leader_election.cc:302] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:38.381824 24179 ts_tablet_manager.cc:1909] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:38.382534 24483 raft_consensus.cc:2726] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:38.380939 24180 log.cc:1196] T da4be1d381a64b7baeb1c60cb17d924a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/da4be1d381a64b7baeb1c60cb17d924a
I20240502 08:16:38.383311 24180 ts_tablet_manager.cc:1909] T da4be1d381a64b7baeb1c60cb17d924a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:38.384377 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet da4be1d381a64b7baeb1c60cb17d924a (table default.Table_1e7f2516f34b483d913cf5080a8febec [id=d5c976b2debc472fa1d7c28ae73ca32a]) successfully deleted
I20240502 08:16:38.336185 24173 tablet_bootstrap.cc:654] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:38.341895 23996 leader_election.cc:341] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.346266 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet dcdd765895254a39b86b692b5d812c83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:38.384938 23996 leader_election.cc:302] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:38.385437 24511 raft_consensus.cc:2726] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:38.385712 24181 ts_tablet_manager.cc:1888] T 5160fda2643e4b80bc401bc48f97eff9 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.385864 24181 log.cc:1196] T 5160fda2643e4b80bc401bc48f97eff9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/5160fda2643e4b80bc401bc48f97eff9
I20240502 08:16:38.386225 24181 ts_tablet_manager.cc:1909] T 5160fda2643e4b80bc401bc48f97eff9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:38.390486 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 5160fda2643e4b80bc401bc48f97eff9 (table default.Table_7e2b03f5ed46441d9c62c30319eab83b [id=8db1f2345d324647881132f139888dee]) successfully deleted
W20240502 08:16:38.391000 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 319fabf91458438ba5e40ab68b323055 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 319fabf91458438ba5e40ab68b323055 already in progress: creating tablet
W20240502 08:16:38.391343 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 4f1736b496354e2dadd844242d3867f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f1736b496354e2dadd844242d3867f8 already in progress: creating tablet
I20240502 08:16:38.391762 24181 tablet_replica.cc:330] T c316e1aeab224634b760641a4ce188c8 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:38.392450 24181 raft_consensus.cc:2229] T c316e1aeab224634b760641a4ce188c8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.392624 24528 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20240502 08:16:38.393492 24181 raft_consensus.cc:2258] T c316e1aeab224634b760641a4ce188c8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.396471 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet da4be1d381a64b7baeb1c60cb17d924a with delete_type TABLET_DATA_DELETED (replaced by tablet fbef606ce7ac499d9bd84b49d804621c) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.398074 24181 ts_tablet_manager.cc:1875] T c316e1aeab224634b760641a4ce188c8 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.404445 24175 tablet_bootstrap.cc:492] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:38.404575 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet c5d28098dcbc4761a34ee65769d6fd7c
I20240502 08:16:38.404572 24179 tablet_replica.cc:330] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:38.405202 24179 raft_consensus.cc:2229] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.405634 24175 ts_tablet_manager.cc:1367] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.007s
I20240502 08:16:38.406435 24179 raft_consensus.cc:2258] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.407011 24179 ts_tablet_manager.cc:1875] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.407985 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e1af8994d95141788ddb43bac3cc5a90" } new_table_name: "default.Table_1704983c4496461d9fa8ea54781f5cb1" modify_external_catalogs: true
W20240502 08:16:38.409171 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e47459b5f5bb4ecb96f76240fefa1f5d has no consensus state
I20240502 08:16:38.409505 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet 4a57ed188315493198e95cefb6907ab6 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:38.410135 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7af51f5f2feb42ca8dbe067300bbbfd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.410763 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7af51f5f2feb42ca8dbe067300bbbfd9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.411077 24175 raft_consensus.cc:348] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:38.411327 24175 raft_consensus.cc:374] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.411398 24175 raft_consensus.cc:729] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.411612 24175 consensus_queue.cc:260] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:38.414080 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d28098dcbc4761a34ee65769d6fd7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:38.414444 24174 tablet_bootstrap.cc:492] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:38.414678 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d28098dcbc4761a34ee65769d6fd7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.418061 24181 ts_tablet_manager.cc:1888] T c316e1aeab224634b760641a4ce188c8 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.418251 24181 log.cc:1196] T c316e1aeab224634b760641a4ce188c8 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/c316e1aeab224634b760641a4ce188c8
I20240502 08:16:38.418689 24181 ts_tablet_manager.cc:1909] T c316e1aeab224634b760641a4ce188c8 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:38.420246 23605 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet d6ce0b4b01fb49cdb51796a35cea5ced
I20240502 08:16:38.420661 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_94efd28ada5b4024ab65e4d5feda0b38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.421228 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94efd28ada5b4024ab65e4d5feda0b38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.421432 24409 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" instance_seqno: 1714637792902555) as {username='slave'} at 127.22.225.66:53815; Asking this server to re-register.
I20240502 08:16:38.422539 23971 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:38.393483 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_30df43df108e43d4b44b20c60eeb8287"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.425467 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30df43df108e43d4b44b20c60eeb8287 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.427582 24409 ts_manager.cc:194] Registered new tserver with Master: 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:38.427639 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 (table default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418]) successfully deleted
I20240502 08:16:38.429265 24175 ts_tablet_manager.cc:1398] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240502 08:16:38.429337 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e9f712275e34498c9f78493dc08b37f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.429827 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9f712275e34498c9f78493dc08b37f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.414675 24174 ts_tablet_manager.cc:1367] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.075s user 0.010s sys 0.001s
I20240502 08:16:38.433538 24175 tablet_bootstrap.cc:492] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:38.439742 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df565c1c6564efd88fa557c50522d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.439960 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet ba3d4b4584dd45b58ff0daa4f1c09a0a with delete_type TABLET_DATA_DELETED (replaced by tablet bd4a31e5aa834adeb9f59267247c6980) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.440208 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d28098dcbc4761a34ee65769d6fd7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.441268 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet c316e1aeab224634b760641a4ce188c8 (table default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]) successfully deleted
I20240502 08:16:38.444303 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_50e65b10059b451aa2a131c796cb3844" } new_table_name: "default.Table_8ba240c391da43978f93a27dce48e557" modify_external_catalogs: true
W20240502 08:16:38.445430 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 319fabf91458438ba5e40ab68b323055 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 319fabf91458438ba5e40ab68b323055 already in progress: creating tablet
W20240502 08:16:38.447443 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 4a57ed188315493198e95cefb6907ab6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a57ed188315493198e95cefb6907ab6 already in progress: creating tablet
W20240502 08:16:38.448083 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 4a57ed188315493198e95cefb6907ab6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a57ed188315493198e95cefb6907ab6 already in progress: creating tablet
I20240502 08:16:38.448212 24175 tablet_bootstrap.cc:654] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.448448 24181 tablet_replica.cc:330] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:38.448782 24181 raft_consensus.cc:2229] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.448989 24181 raft_consensus.cc:2258] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.449625 24174 raft_consensus.cc:348] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:38.449898 24174 raft_consensus.cc:374] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.450001 24174 raft_consensus.cc:729] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.450421 24181 ts_tablet_manager.cc:1875] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.452183 24174 consensus_queue.cc:260] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:38.454991 24174 ts_tablet_manager.cc:1398] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240502 08:16:38.457007 24409 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" instance_seqno: 1714637792402425) as {username='slave'} at 127.22.225.65:59489; Asking this server to re-register.
W20240502 08:16:38.458166 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet bd4a31e5aa834adeb9f59267247c6980 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd4a31e5aa834adeb9f59267247c6980 already in progress: creating tablet
W20240502 08:16:38.458712 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 98a96292b28a4b0aad88fe0a96730a68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98a96292b28a4b0aad88fe0a96730a68 already in progress: creating tablet
W20240502 08:16:38.459049 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet b742c51f6204449a804889815604d5cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b742c51f6204449a804889815604d5cb already in progress: creating tablet
I20240502 08:16:38.458135 23834 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:38.463300 24174 tablet_bootstrap.cc:492] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:38.465790 24175 tablet_bootstrap.cc:492] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:38.466490 24175 ts_tablet_manager.cc:1367] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.003s
I20240502 08:16:38.467375 24174 tablet_bootstrap.cc:654] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.467989 24511 raft_consensus.cc:482] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.468291 24175 raft_consensus.cc:348] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.468256 24511 raft_consensus.cc:504] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.469069 24511 leader_election.cc:288] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:38.470150 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bab1b7c48248a7973c27d0a865b34b" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:38.470894 23993 leader_election.cc:341] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.471336 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_af7c06dd8b3f4085b62eaef9ff9b57cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.471931 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af7c06dd8b3f4085b62eaef9ff9b57cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.469753 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bab1b7c48248a7973c27d0a865b34b" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:38.472537 23996 leader_election.cc:341] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.472658 23996 leader_election.cc:302] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:38.472942 24511 raft_consensus.cc:2726] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:38.473094 24175 raft_consensus.cc:374] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.473389 24175 raft_consensus.cc:729] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.474009 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet ba3d4b4584dd45b58ff0daa4f1c09a0a with delete_type TABLET_DATA_DELETED (replaced by tablet bd4a31e5aa834adeb9f59267247c6980) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.471936 24179 ts_tablet_manager.cc:1888] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.475023 24179 log.cc:1196] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/d8324e9b796047428e2fe918f07f9b32
I20240502 08:16:38.475821 24175 consensus_queue.cc:260] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:38.475968 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ba1baa2279045b3adc58d187a12cbf3 has no consensus state
I20240502 08:16:38.477459 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_60a54110edb74227a1ae3a379d85e68d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.478042 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60a54110edb74227a1ae3a379d85e68d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.483166 24175 ts_tablet_manager.cc:1398] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 08:16:38.483611 24181 ts_tablet_manager.cc:1888] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.475545 24179 ts_tablet_manager.cc:1909] T d8324e9b796047428e2fe918f07f9b32 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:38.488066 24181 log.cc:1196] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/da4be1d381a64b7baeb1c60cb17d924a
I20240502 08:16:38.488526 24181 ts_tablet_manager.cc:1909] T da4be1d381a64b7baeb1c60cb17d924a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:38.488135 24175 tablet_bootstrap.cc:492] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:38.489658 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet da4be1d381a64b7baeb1c60cb17d924a (table default.Table_1e7f2516f34b483d913cf5080a8febec [id=d5c976b2debc472fa1d7c28ae73ca32a]) successfully deleted
I20240502 08:16:38.490036 24181 tablet_replica.cc:330] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:38.490262 24181 raft_consensus.cc:2229] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240502 08:16:38.490265 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 98a96292b28a4b0aad88fe0a96730a68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98a96292b28a4b0aad88fe0a96730a68 already in progress: creating tablet
I20240502 08:16:38.490469 24181 raft_consensus.cc:2258] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:38.490490 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet bd4a31e5aa834adeb9f59267247c6980 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd4a31e5aa834adeb9f59267247c6980 already in progress: creating tablet
W20240502 08:16:38.490717 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet b742c51f6204449a804889815604d5cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b742c51f6204449a804889815604d5cb already in progress: creating tablet
I20240502 08:16:38.491014 24181 ts_tablet_manager.cc:1875] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.491050 24179 tablet_replica.cc:330] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:38.492087 24179 raft_consensus.cc:2229] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:38.493546 24174 tablet_bootstrap.cc:492] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:38.494413 24179 pending_rounds.cc:62] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b: Trying to abort 1 pending ops.
I20240502 08:16:38.494868 24174 ts_tablet_manager.cc:1367] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.002s
I20240502 08:16:38.495153 24179 pending_rounds.cc:68] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156420079783936 op_type: NO_OP noop_request { }
I20240502 08:16:38.496439 24174 raft_consensus.cc:348] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:38.496978 24179 raft_consensus.cc:2866] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:38.497334 24179 raft_consensus.cc:2258] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.498096 24181 ts_tablet_manager.cc:1888] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.498255 24181 log.cc:1196] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/5738cd77bbe147b49215ba9639d7f547
I20240502 08:16:38.498605 24181 ts_tablet_manager.cc:1909] T 5738cd77bbe147b49215ba9639d7f547 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:38.497766 24175 tablet_bootstrap.cc:654] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.499564 24181 tablet_replica.cc:330] T 203ed05d0bdc4b1f99207423b7fc848b P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:38.499714 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 5738cd77bbe147b49215ba9639d7f547 (table default.Table_c84ff62b97d44dd2ad4a1d9927383457 [id=1bfba58d949847308b00af3275167658]) successfully deleted
I20240502 08:16:38.499804 24181 raft_consensus.cc:2229] T 203ed05d0bdc4b1f99207423b7fc848b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.499951 24181 raft_consensus.cc:2258] T 203ed05d0bdc4b1f99207423b7fc848b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.500039 24179 ts_tablet_manager.cc:1875] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.500456 24181 ts_tablet_manager.cc:1875] T 203ed05d0bdc4b1f99207423b7fc848b P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.502581 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet d8324e9b796047428e2fe918f07f9b32 (table default.Table_006d60eaae72439f829df411c32c1514 [id=e3f148f833bc4d99a3ab2ec8d5258507]) successfully deleted
I20240502 08:16:38.458300 24180 tablet_replica.cc:330] T 485c772245fe48a39fc6ceb6377ac0b6 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:38.505817 24180 raft_consensus.cc:2229] T 485c772245fe48a39fc6ceb6377ac0b6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.506350 24180 raft_consensus.cc:2258] T 485c772245fe48a39fc6ceb6377ac0b6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.506565 24409 ts_manager.cc:194] Registered new tserver with Master: a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:38.507339 24180 ts_tablet_manager.cc:1875] T 485c772245fe48a39fc6ceb6377ac0b6 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.511494 24511 raft_consensus.cc:482] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.511678 24511 raft_consensus.cc:504] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:38.512323 24511 leader_election.cc:288] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:38.512708 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b63fe5b6d4db4750800ecd429fa97ed1" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:38.513475 23996 leader_election.cc:341] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.514880 24173 tablet_bootstrap.cc:492] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:38.515040 24173 ts_tablet_manager.cc:1367] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.189s user 0.007s sys 0.007s
I20240502 08:16:38.516024 24173 raft_consensus.cc:348] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.516250 24173 raft_consensus.cc:374] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.516338 24173 raft_consensus.cc:729] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.516646 24173 consensus_queue.cc:260] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.517412 24173 ts_tablet_manager.cc:1398] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:38.517712 24173 tablet_bootstrap.cc:492] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:38.518128 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b63fe5b6d4db4750800ecd429fa97ed1" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:38.518237 24174 raft_consensus.cc:374] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.518354 24174 raft_consensus.cc:729] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.518726 24174 consensus_queue.cc:260] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
W20240502 08:16:38.518885 23993 leader_election.cc:341] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.519006 23993 leader_election.cc:302] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:38.519843 24511 raft_consensus.cc:2726] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:38.520346 24174 ts_tablet_manager.cc:1398] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.025s user 0.000s sys 0.002s
I20240502 08:16:38.520797 24409 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" instance_seqno: 1714637793363499) as {username='slave'} at 127.22.225.67:34497; Asking this server to re-register.
I20240502 08:16:38.521210 24174 tablet_bootstrap.cc:492] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:38.522330 24106 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:38.522541 24181 ts_tablet_manager.cc:1888] T 203ed05d0bdc4b1f99207423b7fc848b P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.522687 24181 log.cc:1196] T 203ed05d0bdc4b1f99207423b7fc848b P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/203ed05d0bdc4b1f99207423b7fc848b
I20240502 08:16:38.523102 24181 ts_tablet_manager.cc:1909] T 203ed05d0bdc4b1f99207423b7fc848b P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:38.523905 24181 tablet_replica.cc:330] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:38.524091 24181 raft_consensus.cc:2229] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.524292 24181 raft_consensus.cc:2258] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.524837 24181 ts_tablet_manager.cc:1875] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.526326 24173 tablet_bootstrap.cc:654] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.526700 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 203ed05d0bdc4b1f99207423b7fc848b (table default.Table_68bf91a14ffa42cdbb10a1d7ca56cb08 [id=59d945a73daa4918ba17063eae0b9b93]) successfully deleted
I20240502 08:16:38.527558 24174 tablet_bootstrap.cc:654] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.530529 24409 ts_manager.cc:194] Registered new tserver with Master: 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:38.533955 24180 ts_tablet_manager.cc:1888] T 485c772245fe48a39fc6ceb6377ac0b6 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.534122 24180 log.cc:1196] T 485c772245fe48a39fc6ceb6377ac0b6 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/485c772245fe48a39fc6ceb6377ac0b6
I20240502 08:16:38.534929 24181 ts_tablet_manager.cc:1888] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.535117 24181 log.cc:1196] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/f36ceb0222ef473a828ca795f393174d
I20240502 08:16:38.535567 24181 ts_tablet_manager.cc:1909] T f36ceb0222ef473a828ca795f393174d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:38.536198 24180 ts_tablet_manager.cc:1909] T 485c772245fe48a39fc6ceb6377ac0b6 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:38.537864 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet f36ceb0222ef473a828ca795f393174d (table default.Table_4b967ef2b2fc4e2caa4219afcae841a6 [id=c32fc96889554c32bdf1d4550633e91f]) successfully deleted
I20240502 08:16:38.538216 24181 tablet_replica.cc:330] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:38.538450 24181 raft_consensus.cc:2229] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.538667 24181 raft_consensus.cc:2258] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.539193 24181 ts_tablet_manager.cc:1875] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:38.545442 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9761df0bbdee44af90135bffb8b5828f has no consensus state
I20240502 08:16:38.549036 24175 tablet_bootstrap.cc:492] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:38.549261 24175 ts_tablet_manager.cc:1367] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.002s
I20240502 08:16:38.551275 24181 ts_tablet_manager.cc:1888] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.551316 24180 tablet_replica.cc:330] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:38.550244 24175 raft_consensus.cc:348] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:38.553195 24175 raft_consensus.cc:374] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.553308 24175 raft_consensus.cc:729] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.553599 24175 consensus_queue.cc:260] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:38.551553 24180 raft_consensus.cc:2229] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:38.554399 24180 pending_rounds.cc:62] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:16:38.554390 24509 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.554476 24180 pending_rounds.cc:68] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156419789033472 op_type: NO_OP noop_request { }
I20240502 08:16:38.554664 24180 raft_consensus.cc:2866] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:38.554638 24509 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:38.554781 24180 raft_consensus.cc:2258] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.555868 24180 ts_tablet_manager.cc:1875] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.556782 24509 leader_election.cc:288] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84f3c07a31414c4f8fb61c44dc9f16e1 (127.22.225.125:36189), f87df6a45ad94428a9aa6b2796a0fd25 (127.22.225.124:41745)
I20240502 08:16:38.557495 24179 ts_tablet_manager.cc:1888] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:38.557713 23549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" is_pre_election: true
I20240502 08:16:38.553320 24181 log.cc:1196] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/b63fe5b6d4db4750800ecd429fa97ed1
I20240502 08:16:38.558640 24511 raft_consensus.cc:482] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.558812 24511 raft_consensus.cc:504] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.559428 24511 leader_election.cc:288] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:38.558812 24181 ts_tablet_manager.cc:1909] T b63fe5b6d4db4750800ecd429fa97ed1 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:38.559795 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9592a041a9543e8b41f043927807131" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:38.560302 23993 leader_election.cc:341] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.560524 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet b63fe5b6d4db4750800ecd429fa97ed1 (table default.Table_07371eaf91cb4ab199af62382b640d64 [id=0026ee826a3e489087f873579507c110]) successfully deleted
I20240502 08:16:38.560873 24181 tablet_replica.cc:330] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:38.560974 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 485c772245fe48a39fc6ceb6377ac0b6 (table default.Table_0528f5d019fe489db8bed1f71adac3cb [id=7dd2b25a6676488d8f3a8e2e6b65046e]) successfully deleted
I20240502 08:16:38.561038 24181 raft_consensus.cc:2229] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.561203 24181 raft_consensus.cc:2258] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:38.561947 23597 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 1ba1baa2279045b3adc58d187a12cbf3 (table default.Table_1704983c4496461d9fa8ea54781f5cb1 [id=1dbff884223d4ad9843fe2512d1f295c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1ba1baa2279045b3adc58d187a12cbf3 has no consensus state
I20240502 08:16:38.563391 24181 ts_tablet_manager.cc:1875] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.561759 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9592a041a9543e8b41f043927807131" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:38.564543 23924 raft_consensus.cc:2454] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:38.562115 24483 raft_consensus.cc:482] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.564868 24483 raft_consensus.cc:504] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.566068 23996 leader_election.cc:302] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:38.566352 24511 raft_consensus.cc:2781] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:38.566438 24511 raft_consensus.cc:482] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:38.566524 24511 raft_consensus.cc:3037] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:38.566972 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c44e58448899483fb6f36f8ee844e61d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.567610 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c44e58448899483fb6f36f8ee844e61d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.565538 24483 leader_election.cc:288] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:38.567651 23861 leader_election.cc:341] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 1df565c1c6564efd88fa557c50522d7e
I20240502 08:16:38.562991 24175 ts_tablet_manager.cc:1398] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.014s user 0.002s sys 0.001s
I20240502 08:16:38.569216 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df565c1c6564efd88fa557c50522d7e" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:38.569289 24511 raft_consensus.cc:504] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.569432 23787 raft_consensus.cc:2454] T 1df565c1c6564efd88fa557c50522d7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:38.569912 23858 leader_election.cc:302] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:38.569938 24511 leader_election.cc:288] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:38.570293 24329 raft_consensus.cc:482] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.570436 24329 raft_consensus.cc:504] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.570950 24329 leader_election.cc:288] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:38.571345 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9592a041a9543e8b41f043927807131" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:38.571512 23924 raft_consensus.cc:3037] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:38.571487 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9592a041a9543e8b41f043927807131" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:38.571801 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bd9e5acccefd4757b59b162649c7747c" } new_table_name: "default.Table_827517f3c36f43e585faba823c6d068d" modify_external_catalogs: true
W20240502 08:16:38.571967 23993 leader_election.cc:341] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.572433 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet e9592a041a9543e8b41f043927807131 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.572871 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c709efdd2ce74b5a8ab7278b402ebfe9" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:38.573065 23925 raft_consensus.cc:2454] T c709efdd2ce74b5a8ab7278b402ebfe9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:38.573009 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b9be11b881924c8380270cc629f05765"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.573462 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9be11b881924c8380270cc629f05765 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.573454 23723 leader_election.cc:302] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:38.573812 24329 raft_consensus.cc:2781] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:38.573897 24329 raft_consensus.cc:482] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:38.573985 24329 raft_consensus.cc:3037] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:38.574874 24483 raft_consensus.cc:2781] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:38.574960 24483 raft_consensus.cc:482] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:38.575047 24483 raft_consensus.cc:3037] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:38.576637 24181 ts_tablet_manager.cc:1888] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.576758 24181 log.cc:1196] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/ba3d4b4584dd45b58ff0daa4f1c09a0a
I20240502 08:16:38.577131 24181 ts_tablet_manager.cc:1909] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:38.577332 24483 raft_consensus.cc:504] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.578091 24181 tablet_replica.cc:330] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:38.578272 24181 raft_consensus.cc:2229] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.578403 24181 raft_consensus.cc:2258] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.578613 23924 raft_consensus.cc:2454] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:38.578902 24181 ts_tablet_manager.cc:1875] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.580327 23996 leader_election.cc:302] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:38.580549 23996 raft_consensus.cc:2670] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240502 08:16:38.582002 23861 leader_election.cc:341] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 1df565c1c6564efd88fa557c50522d7e
I20240502 08:16:38.582186 24483 leader_election.cc:288] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:38.582615 23761 tablet_service.cc:1467] Processing CreateTablet for tablet f5ccd43e78574f3288b199fa14bcc0e8 (DEFAULT_TABLE table=default.Table_95df2e76edc746a08164b4ff25809909 [id=d5a04f8005714d15a3710236c85b1317]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.582984 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5ccd43e78574f3288b199fa14bcc0e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.583333 24175 tablet_bootstrap.cc:492] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:38.583366 23606 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 23ce83c8001d4b2988b88c0ed9bf948c (table default.Table_190eae2a8c64440f8d19842f8b6ac254 [id=54454efbe9394170822f237dc8e6115e]). Expected version 1 got 0
I20240502 08:16:38.584918 24175 tablet_bootstrap.cc:654] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.585361 24181 ts_tablet_manager.cc:1888] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.585500 24181 log.cc:1196] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/e9592a041a9543e8b41f043927807131
I20240502 08:16:38.585875 24181 ts_tablet_manager.cc:1909] T e9592a041a9543e8b41f043927807131 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:38.586782 24181 tablet_replica.cc:330] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:38.586825 24173 tablet_bootstrap.cc:492] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:38.586035 24179 log.cc:1196] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/a15ff3120b1943a69ddb609d0575ae1a
I20240502 08:16:38.586977 24181 raft_consensus.cc:2229] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.587168 24181 raft_consensus.cc:2258] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.587698 24181 ts_tablet_manager.cc:1875] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.587669 24032 tablet_service.cc:1467] Processing CreateTablet for tablet f797d3c6bdc4493398954fbbf9943971 (DEFAULT_TABLE table=default.Table_7f44f7f1aed6460cb1e73392da0987b8 [id=07d8592503e843e18faedfcce845a974]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.587743 24033 tablet_service.cc:1467] Processing CreateTablet for tablet f5ccd43e78574f3288b199fa14bcc0e8 (DEFAULT_TABLE table=default.Table_95df2e76edc746a08164b4ff25809909 [id=d5a04f8005714d15a3710236c85b1317]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.588027 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f797d3c6bdc4493398954fbbf9943971. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.588255 24030 tablet_service.cc:1467] Processing CreateTablet for tablet c5c377e037894728ac084b6500ad20de (DEFAULT_TABLE table=default.Table_9a479fbf2ac24ac1838fc0599feb088a [id=57a76411ce5b4a9c8b973ce5d2da53ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.588644 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c377e037894728ac084b6500ad20de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.589516 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 654ce33d1e3d472eb6ae6a726eb803bb (DEFAULT_TABLE table=default.Table_6f4d25ef5b90455ca233af7b70b9073a [id=2fe4c270673b4204aaccc289d3b9d929]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.589905 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654ce33d1e3d472eb6ae6a726eb803bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.589923 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 24a4104f4cb64889ad66810788ad6b29 (DEFAULT_TABLE table=default.Table_307041e730f248a5bd4d265f08aad5e4 [id=f2316137abe74f5c8e79faf0c4638540]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.590265 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24a4104f4cb64889ad66810788ad6b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.593694 24179 ts_tablet_manager.cc:1909] T a15ff3120b1943a69ddb609d0575ae1a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:38.593843 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 654ce33d1e3d472eb6ae6a726eb803bb (DEFAULT_TABLE table=default.Table_6f4d25ef5b90455ca233af7b70b9073a [id=2fe4c270673b4204aaccc289d3b9d929]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.594195 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654ce33d1e3d472eb6ae6a726eb803bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.594795 24179 tablet_replica.cc:330] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:38.594990 24179 raft_consensus.cc:2229] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.595160 24179 raft_consensus.cc:2258] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.595641 24179 ts_tablet_manager.cc:1875] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.586980 24173 ts_tablet_manager.cc:1367] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.003s
I20240502 08:16:38.596751 24173 raft_consensus.cc:348] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.596884 23761 tablet_service.cc:1467] Processing CreateTablet for tablet c5c377e037894728ac084b6500ad20de (DEFAULT_TABLE table=default.Table_9a479fbf2ac24ac1838fc0599feb088a [id=57a76411ce5b4a9c8b973ce5d2da53ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.596984 24173 raft_consensus.cc:374] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.597082 24173 raft_consensus.cc:729] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.597290 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c377e037894728ac084b6500ad20de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.597321 24173 consensus_queue.cc:260] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.597824 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 24a4104f4cb64889ad66810788ad6b29 (DEFAULT_TABLE table=default.Table_307041e730f248a5bd4d265f08aad5e4 [id=f2316137abe74f5c8e79faf0c4638540]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.598775 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet ba3d4b4584dd45b58ff0daa4f1c09a0a (table default.Table_4e40fa1cf0ab4dedb2c6aadb2418901b [id=fd781805eb4c45f09fe6460144472816]) successfully deleted
I20240502 08:16:38.599017 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet e9592a041a9543e8b41f043927807131 (table default.Table_6c584db923e147d380281ecb1acd11a1 [id=467a7f6db73641ca85a1898b251938fb]) successfully deleted
W20240502 08:16:38.599318 23723 leader_election.cc:341] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: c709efdd2ce74b5a8ab7278b402ebfe9
I20240502 08:16:38.600215 24329 raft_consensus.cc:504] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.600489 24175 tablet_bootstrap.cc:492] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:38.600636 24175 ts_tablet_manager.cc:1367] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240502 08:16:38.601256 24329 leader_election.cc:288] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:38.601346 24175 raft_consensus.cc:348] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:38.603068 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df565c1c6564efd88fa557c50522d7e" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:38.603327 23787 raft_consensus.cc:3037] T 1df565c1c6564efd88fa557c50522d7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:38.601536 24175 raft_consensus.cc:374] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.605386 24175 raft_consensus.cc:729] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.605697 24175 consensus_queue.cc:260] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:38.606441 24174 tablet_bootstrap.cc:492] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:38.606639 24174 ts_tablet_manager.cc:1367] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.086s user 0.004s sys 0.004s
I20240502 08:16:38.606765 23787 raft_consensus.cc:2454] T 1df565c1c6564efd88fa557c50522d7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:38.607434 24174 raft_consensus.cc:348] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.607653 24174 raft_consensus.cc:374] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.607733 24174 raft_consensus.cc:729] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.607950 24174 consensus_queue.cc:260] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.608619 24174 ts_tablet_manager.cc:1398] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:38.609015 24174 tablet_bootstrap.cc:492] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:38.611040 24174 tablet_bootstrap.cc:654] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.611182 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet a15ff3120b1943a69ddb609d0575ae1a (table default.Table_bc6b3019e44547cda285ac0021565619 [id=c08bf22592064123bb9cab28935d81a5]) successfully deleted
I20240502 08:16:38.611402 24483 raft_consensus.cc:482] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.611636 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24a4104f4cb64889ad66810788ad6b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.611546 24483 raft_consensus.cc:504] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.612144 24483 leader_election.cc:288] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:38.612409 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5ccd43e78574f3288b199fa14bcc0e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.612408 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c709efdd2ce74b5a8ab7278b402ebfe9" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:38.612613 23924 raft_consensus.cc:3037] T c709efdd2ce74b5a8ab7278b402ebfe9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:38.612660 23767 tablet_service.cc:1467] Processing CreateTablet for tablet f797d3c6bdc4493398954fbbf9943971 (DEFAULT_TABLE table=default.Table_7f44f7f1aed6460cb1e73392da0987b8 [id=07d8592503e843e18faedfcce845a974]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.613039 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f797d3c6bdc4493398954fbbf9943971. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.613514 23905 tablet_service.cc:1467] Processing CreateTablet for tablet f5ccd43e78574f3288b199fa14bcc0e8 (DEFAULT_TABLE table=default.Table_95df2e76edc746a08164b4ff25809909 [id=d5a04f8005714d15a3710236c85b1317]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.613943 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5ccd43e78574f3288b199fa14bcc0e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.614576 23899 tablet_service.cc:1467] Processing CreateTablet for tablet c5c377e037894728ac084b6500ad20de (DEFAULT_TABLE table=default.Table_9a479fbf2ac24ac1838fc0599feb088a [id=57a76411ce5b4a9c8b973ce5d2da53ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.616097 24355 raft_consensus.cc:482] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.616261 24355 raft_consensus.cc:504] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.617066 24511 raft_consensus.cc:482] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.617688 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ce83c8001d4b2988b88c0ed9bf948c" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:38.618052 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c377e037894728ac084b6500ad20de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.619199 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 654ce33d1e3d472eb6ae6a726eb803bb (DEFAULT_TABLE table=default.Table_6f4d25ef5b90455ca233af7b70b9073a [id=2fe4c270673b4204aaccc289d3b9d929]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.620821 24173 ts_tablet_manager.cc:1398] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20240502 08:16:38.621454 23861 leader_election.cc:341] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 485c772245fe48a39fc6ceb6377ac0b6
I20240502 08:16:38.621199 24511 raft_consensus.cc:504] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.621956 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 1da01e2c2f054ccfb53383e70e16da84 (DEFAULT_TABLE table=default.Table_dcff63ecc4f740fd999047021472be23 [id=f596bcb8d6f04ac680be38e763a33750]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.622107 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba1baa2279045b3adc58d187a12cbf3 (DEFAULT_TABLE table=default.Table_1704983c4496461d9fa8ea54781f5cb1 [id=1dbff884223d4ad9843fe2512d1f295c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.622150 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ce83c8001d4b2988b88c0ed9bf948c" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:38.622440 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba1baa2279045b3adc58d187a12cbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.623076 24511 leader_election.cc:288] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:38.623109 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 7ed919909289410e867881baab841f53 (DEFAULT_TABLE table=default.Table_bd4df4dff42547d6aba1825676ef7879 [id=ee3912407a934444ad2c62f7b83d93ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.623423 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba1baa2279045b3adc58d187a12cbf3 (DEFAULT_TABLE table=default.Table_1704983c4496461d9fa8ea54781f5cb1 [id=1dbff884223d4ad9843fe2512d1f295c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.623602 24040 tablet_service.cc:1467] Processing CreateTablet for tablet cb7b37f776df47da9aa6e7fba82a8999 (DEFAULT_TABLE table=default.Table_3def35abf8ad4ff49c5ae77acb410e01 [id=5550f8374136404e84d93cc14b9cf834]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.624814 23606 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 08183523bf4a4d1c9020eb693aa07878 (table default.Table_fcd8e870383b4c8f9b40db6fb2a7149b [id=0a2f8dbb62a64b0a96931ab5f8c17540]). Expected version 1 got 0
W20240502 08:16:38.625610 23993 leader_election.cc:341] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.631778 23924 raft_consensus.cc:2454] T c709efdd2ce74b5a8ab7278b402ebfe9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:38.634835 24174 tablet_bootstrap.cc:492] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:38.636627 24174 ts_tablet_manager.cc:1367] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240502 08:16:38.637737 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_44a0f173f72b49eea2cc2243bc11c430" } new_table_name: "default.Table_54b9e5042d19438ab2690d06917ec10c" modify_external_catalogs: true
I20240502 08:16:38.638200 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ec5e04e87fba492ab9a96d4034db240a" } modify_external_catalogs: true
I20240502 08:16:38.638372 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ec5e04e87fba492ab9a96d4034db240a" } modify_external_catalogs: true
I20240502 08:16:38.638979 24329 raft_consensus.cc:482] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.639118 24329 raft_consensus.cc:504] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:38.639679 24329 leader_election.cc:288] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:38.642509 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 7ed919909289410e867881baab841f53 (DEFAULT_TABLE table=default.Table_bd4df4dff42547d6aba1825676ef7879 [id=ee3912407a934444ad2c62f7b83d93ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.644019 23764 tablet_service.cc:1467] Processing CreateTablet for tablet a0224317e65747e1add533d33472f7f3 (DEFAULT_TABLE table=default.Table_5782cc56973c43cfac93b2560c62c688 [id=9a3faa2c394841cfbfab3425282d84d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.644701 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 78ddfc7b74644a0bbf8222d5f90e6658 (DEFAULT_TABLE table=default.Table_0b7c860d457f421294ca1d37248e40b5 [id=c06501dd41a640279c769c9ab3926ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.644834 23620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" is_pre_election: true
I20240502 08:16:38.649080 24375 leader_election.cc:302] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcc8a323e6b9429eaaf4e0c1698ad077; no voters: 84f3c07a31414c4f8fb61c44dc9f16e1, f87df6a45ad94428a9aa6b2796a0fd25
I20240502 08:16:38.649933 24509 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:38.650539 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 6e7432f1df94431a83cf68356e3e78a3 (DEFAULT_TABLE table=default.Table_a89a2cdb74684df5bcbe15389816c50b [id=04819651040e46c580d553290e3fa250]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.651587 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 577b149f52e44d5d9a9f6b9d85a6864d (DEFAULT_TABLE table=default.Table_cb7682ca6f0f4d93922aa35e427c9cfa [id=89eea7c9830f47d2bd1c95044d5af4d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.652089 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 751808dfc1964b539f6f836d786156aa (DEFAULT_TABLE table=default.Table_97709af0716042cebdc01ed2d06dcf20 [id=911d4d5a86e54a5696d37a2bd9f69733]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.652767 23757 tablet_service.cc:1467] Processing CreateTablet for tablet dd3078cc1f6247609f8bd017ed076b45 (DEFAULT_TABLE table=default.Table_bac7f229ce9541b39c0a70951431b689 [id=f9f2bf985f124269982869050bbecd70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.653108 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 45f02efa659e457ab57bc32ed3f66bb9 (DEFAULT_TABLE table=default.Table_44a0f173f72b49eea2cc2243bc11c430 [id=27f4152aaee24a3bad52e0390b8fa649]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.653326 23765 tablet_service.cc:1467] Processing CreateTablet for tablet cb7b37f776df47da9aa6e7fba82a8999 (DEFAULT_TABLE table=default.Table_3def35abf8ad4ff49c5ae77acb410e01 [id=5550f8374136404e84d93cc14b9cf834]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.653528 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 1da01e2c2f054ccfb53383e70e16da84 (DEFAULT_TABLE table=default.Table_dcff63ecc4f740fd999047021472be23 [id=f596bcb8d6f04ac680be38e763a33750]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.654971 23762 tablet_service.cc:1467] Processing CreateTablet for tablet b59c2f93d6514c9cbf358ab96f1a06b5 (DEFAULT_TABLE table=default.Table_bbf9d7694fe942de8aa8bd4ed61336a5 [id=f1d6eb661b0143ee82e797cde61391c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.624166 24027 tablet_service.cc:1467] Processing CreateTablet for tablet a0224317e65747e1add533d33472f7f3 (DEFAULT_TABLE table=default.Table_5782cc56973c43cfac93b2560c62c688 [id=9a3faa2c394841cfbfab3425282d84d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.657717 24355 leader_election.cc:288] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:38.658773 24180 ts_tablet_manager.cc:1888] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:38.658557 24174 raft_consensus.cc:348] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:38.660440 24173 tablet_bootstrap.cc:492] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:38.660492 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1da01e2c2f054ccfb53383e70e16da84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.661604 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ed919909289410e867881baab841f53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.661831 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 577b149f52e44d5d9a9f6b9d85a6864d (DEFAULT_TABLE table=default.Table_cb7682ca6f0f4d93922aa35e427c9cfa [id=89eea7c9830f47d2bd1c95044d5af4d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.662147 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 751808dfc1964b539f6f836d786156aa (DEFAULT_TABLE table=default.Table_97709af0716042cebdc01ed2d06dcf20 [id=911d4d5a86e54a5696d37a2bd9f69733]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.662748 24180 log.cc:1196] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/223c92ae8f63402f8643e03d914e8e7e
I20240502 08:16:38.658594 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 001e3f4f72f64dc69dbd521724912f07 (DEFAULT_TABLE table=default.Table_8b1436ec53e349658e5e39d3e10da3e1 [id=5c21117955d946418c656330984ae747]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.662891 24174 raft_consensus.cc:374] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.657274 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1d946bd6de48f1a8d8fe5f179d22b6 (DEFAULT_TABLE table=default.Table_7e12ddab0f5241b4ac31de16f1467292 [id=6b44d73f1e0b4185a950f7bd4ed2f430]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.667464 24175 ts_tablet_manager.cc:1398] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240502 08:16:38.667569 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654ce33d1e3d472eb6ae6a726eb803bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.667645 23900 tablet_service.cc:1467] Processing CreateTablet for tablet f797d3c6bdc4493398954fbbf9943971 (DEFAULT_TABLE table=default.Table_7f44f7f1aed6460cb1e73392da0987b8 [id=07d8592503e843e18faedfcce845a974]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.667855 24175 tablet_bootstrap.cc:492] T 203ed05d0bdc4b1f99207423b7fc848b P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:38.668016 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cc30014a1ac45de937751eed36877a9" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:38.668054 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 45f02efa659e457ab57bc32ed3f66bb9 (DEFAULT_TABLE table=default.Table_44a0f173f72b49eea2cc2243bc11c430 [id=27f4152aaee24a3bad52e0390b8fa649]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.668476 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f797d3c6bdc4493398954fbbf9943971. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.668596 24034 tablet_service.cc:1467] Processing CreateTablet for tablet dd3078cc1f6247609f8bd017ed076b45 (DEFAULT_TABLE table=default.Table_bac7f229ce9541b39c0a70951431b689 [id=f9f2bf985f124269982869050bbecd70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.668529 24174 raft_consensus.cc:729] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.669075 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 955e7bd2216646c6aef5acce8d5b0d77 (DEFAULT_TABLE table=default.Table_b3278c5d6d0b4593ac33bc9977f5c3b1 [id=e86031550cec46a793785c0b3859fa92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.669294 24026 tablet_service.cc:1467] Processing CreateTablet for tablet b9f83bcf640f465bb95d87a991cc5ff6 (DEFAULT_TABLE table=default.Table_e00515baeaae4ce9ad93000143f2a53a [id=13293a9ab7004e8a965e004f2fb5a14a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.669457 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 33c8131846dc41678827b86e87be18d3 (DEFAULT_TABLE table=default.Table_df744f06850c416f98d94d2df132a03b [id=5849a3c8b28e47439b73e2137187fac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.669636 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1d946bd6de48f1a8d8fe5f179d22b6 (DEFAULT_TABLE table=default.Table_7e12ddab0f5241b4ac31de16f1467292 [id=6b44d73f1e0b4185a950f7bd4ed2f430]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.662858 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 33c8131846dc41678827b86e87be18d3 (DEFAULT_TABLE table=default.Table_df744f06850c416f98d94d2df132a03b [id=5849a3c8b28e47439b73e2137187fac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.671770 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 4b8d81f071d74b218fcfd4f4e8c20bab (DEFAULT_TABLE table=default.Table_dbe385cc3f6b48d39bee298ca9490248 [id=aa825e5ca73c4dafa9585126d5c48df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.672760 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba1baa2279045b3adc58d187a12cbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.673974 24483 raft_consensus.cc:482] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.674110 24483 raft_consensus.cc:504] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.674520 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba1baa2279045b3adc58d187a12cbf3 (DEFAULT_TABLE table=default.Table_1704983c4496461d9fa8ea54781f5cb1 [id=1dbff884223d4ad9843fe2512d1f295c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.674865 24483 leader_election.cc:288] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:38.674870 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba1baa2279045b3adc58d187a12cbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.673971 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 78ddfc7b74644a0bbf8222d5f90e6658 (DEFAULT_TABLE table=default.Table_0b7c860d457f421294ca1d37248e40b5 [id=c06501dd41a640279c769c9ab3926ad5]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:38.673888 23996 leader_election.cc:341] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.677677 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7b37f776df47da9aa6e7fba82a8999. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.677989 24022 tablet_service.cc:1467] Processing CreateTablet for tablet 001e3f4f72f64dc69dbd521724912f07 (DEFAULT_TABLE table=default.Table_8b1436ec53e349658e5e39d3e10da3e1 [id=5c21117955d946418c656330984ae747]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.677539 23996 leader_election.cc:302] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:38.678859 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3043c5ed98c401b82c6e967ca3f3613 consensus state has no leader
W20240502 08:16:38.678819 23996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 50891.5us
I20240502 08:16:38.679152 23607 master_service.cc:946] ReplaceTablet: received request to replace tablet d5bab1b7c48248a7973c27d0a865b34b from {username='slave'} at 127.0.0.1:36668
W20240502 08:16:38.679219 23996 leader_election.cc:341] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.680027 24355 raft_consensus.cc:2726] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:38.680739 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet 5a1e208733844829b13222e00a432439 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:38.682204 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0224317e65747e1add533d33472f7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.683320 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 751808dfc1964b539f6f836d786156aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.683391 24483 raft_consensus.cc:482] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.683563 24483 raft_consensus.cc:504] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.684209 24483 leader_election.cc:288] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:38.685635 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0a0743012671403884555c6274a7f3da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:38.685937 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45f02efa659e457ab57bc32ed3f66bb9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:38.686159 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a0743012671403884555c6274a7f3da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:38.687842 23861 leader_election.cc:341] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: c316e1aeab224634b760641a4ce188c8
I20240502 08:16:38.690052 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd3078cc1f6247609f8bd017ed076b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.690841 24173 tablet_bootstrap.cc:654] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.693579 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577b149f52e44d5d9a9f6b9d85a6864d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.694018 24021 tablet_service.cc:1467] Processing CreateTablet for tablet b59c2f93d6514c9cbf358ab96f1a06b5 (DEFAULT_TABLE table=default.Table_bbf9d7694fe942de8aa8bd4ed61336a5 [id=f1d6eb661b0143ee82e797cde61391c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.695379 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2047aed3b97c4f4283bd8f6e80aeb92e" } modify_external_catalogs: true
I20240502 08:16:38.695597 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2047aed3b97c4f4283bd8f6e80aeb92e" } modify_external_catalogs: true
W20240502 08:16:38.696960 23861 leader_election.cc:341] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 9b1eb762780c450bbaf380d17db564b6
I20240502 08:16:38.699571 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bd4df4dff42547d6aba1825676ef7879" } new_table_name: "default.Table_9d47c3a9169244efb08e234d37647e64" modify_external_catalogs: true
I20240502 08:16:38.709630 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 6e7432f1df94431a83cf68356e3e78a3 (DEFAULT_TABLE table=default.Table_a89a2cdb74684df5bcbe15389816c50b [id=04819651040e46c580d553290e3fa250]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.709762 23900 tablet_service.cc:1467] Processing CreateTablet for tablet cb7b37f776df47da9aa6e7fba82a8999 (DEFAULT_TABLE table=default.Table_3def35abf8ad4ff49c5ae77acb410e01 [id=5550f8374136404e84d93cc14b9cf834]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.710101 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e7432f1df94431a83cf68356e3e78a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.710623 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 5d04312bd81f46ed8c49c42b1af1aec9 (DEFAULT_TABLE table=default.Table_9741459a96e649fb8151cc28b642523e [id=2b9169f285754bfda0f4f8dc7f5cb653]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.711472 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 1da01e2c2f054ccfb53383e70e16da84 (DEFAULT_TABLE table=default.Table_dcff63ecc4f740fd999047021472be23 [id=f596bcb8d6f04ac680be38e763a33750]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.711902 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1da01e2c2f054ccfb53383e70e16da84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.712778 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7b37f776df47da9aa6e7fba82a8999. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.717155 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 1e5de793e74247dd9e73b7b1d58027d9
I20240502 08:16:38.717470 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 848a0865eb2b4828a2594a17d96869d0
I20240502 08:16:38.717178 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 0882b7509a74475780eb700818aab398
I20240502 08:16:38.718772 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 71aa916247c449fea718b35869a6df20
I20240502 08:16:38.718855 23608 catalog_manager.cc:6554] ReplaceTablet: tablet 0882b7509a74475780eb700818aab398 deleted and replaced by tablet eb96ad2289754e428a1c109b2d59c53b
I20240502 08:16:38.719213 23603 catalog_manager.cc:6554] ReplaceTablet: tablet 71aa916247c449fea718b35869a6df20 deleted and replaced by tablet 0501575a84ec454780f48574bf3155fe
I20240502 08:16:38.721730 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet b11f504b93574e42a9bf30fde6d08207
I20240502 08:16:38.721554 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6bdd34511d19462d91feac0f19089b68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.721925 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 45f02efa659e457ab57bc32ed3f66bb9 consensus state has no leader
W20240502 08:16:38.722177 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bdd34511d19462d91feac0f19089b68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.722350 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fba18fd93e6c4364bead807a40bd1689"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:38.722455 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_47643e78190448bf93ad852cabe9b8ff" } new_table_name: "default.Table_9d372763eb03435f8b7074a802d86ffe" modify_external_catalogs: true
I20240502 08:16:38.723099 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 6e7432f1df94431a83cf68356e3e78a3 (DEFAULT_TABLE table=default.Table_a89a2cdb74684df5bcbe15389816c50b [id=04819651040e46c580d553290e3fa250]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:38.723555 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fba18fd93e6c4364bead807a40bd1689 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.725252 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bac7f229ce9541b39c0a70951431b689" } modify_external_catalogs: true
I20240502 08:16:38.725821 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bac7f229ce9541b39c0a70951431b689" } modify_external_catalogs: true
I20240502 08:16:38.727586 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5b51e3b192594b119d6719a685fe3513"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.732299 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b51e3b192594b119d6719a685fe3513 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.734180 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet b6417aafbdcf416ea7bff56bce5e6ddb from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:38.734553 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 7c374f42f353497fa1d745164e7fbf61
W20240502 08:16:38.736330 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ed919909289410e867881baab841f53 consensus state has no leader
I20240502 08:16:38.682654 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cc30014a1ac45de937751eed36877a9" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:38.738137 24180 ts_tablet_manager.cc:1909] T 223c92ae8f63402f8643e03d914e8e7e P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:38.738181 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 5a1e208733844829b13222e00a432439
I20240502 08:16:38.738649 23602 catalog_manager.cc:6554] ReplaceTablet: tablet 5a1e208733844829b13222e00a432439 deleted and replaced by tablet 8910bc67c6234a538c8160e7c020653c
I20240502 08:16:38.740548 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 751808dfc1964b539f6f836d786156aa (DEFAULT_TABLE table=default.Table_97709af0716042cebdc01ed2d06dcf20 [id=911d4d5a86e54a5696d37a2bd9f69733]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.741153 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 7ed919909289410e867881baab841f53 (DEFAULT_TABLE table=default.Table_bd4df4dff42547d6aba1825676ef7879 [id=ee3912407a934444ad2c62f7b83d93ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.741426 23767 tablet_service.cc:1467] Processing CreateTablet for tablet b9f83bcf640f465bb95d87a991cc5ff6 (DEFAULT_TABLE table=default.Table_e00515baeaae4ce9ad93000143f2a53a [id=13293a9ab7004e8a965e004f2fb5a14a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.741643 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 955e7bd2216646c6aef5acce8d5b0d77 (DEFAULT_TABLE table=default.Table_b3278c5d6d0b4593ac33bc9977f5c3b1 [id=e86031550cec46a793785c0b3859fa92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.741772 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 24a4104f4cb64889ad66810788ad6b29 (DEFAULT_TABLE table=default.Table_307041e730f248a5bd4d265f08aad5e4 [id=f2316137abe74f5c8e79faf0c4638540]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.742331 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b59c2f93d6514c9cbf358ab96f1a06b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.742502 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ed919909289410e867881baab841f53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.742740 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 128c88af54d24ffd895e9de19cdb4c8a (DEFAULT_TABLE table=default.Table_d67be3894f494223a45d8c5a2c3e6113 [id=f7982b1d892942648cf3fca657c2668c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.746930 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 6746c25111d04f20a6f30ce0838d71f5 (DEFAULT_TABLE table=default.Table_02f448a8a9844def8ecda8716fdc352e [id=79a03d22b7e34f76bb3242c640129a13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.747121 23894 tablet_service.cc:1467] Processing CreateTablet for tablet a0224317e65747e1add533d33472f7f3 (DEFAULT_TABLE table=default.Table_5782cc56973c43cfac93b2560c62c688 [id=9a3faa2c394841cfbfab3425282d84d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.749698 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 955e7bd2216646c6aef5acce8d5b0d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.750522 24329 raft_consensus.cc:482] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.750703 24329 raft_consensus.cc:504] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.751703 24329 leader_election.cc:288] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:38.743173 24174 consensus_queue.cc:260] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:38.752872 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b141c2b67ffe43ff97d32c9fff1d1fe0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:38.753404 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33c8131846dc41678827b86e87be18d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.753641 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 751808dfc1964b539f6f836d786156aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.752074 23752 tablet_service.cc:1467] Processing CreateTablet for tablet 5d04312bd81f46ed8c49c42b1af1aec9 (DEFAULT_TABLE table=default.Table_9741459a96e649fb8151cc28b642523e [id=2b9169f285754bfda0f4f8dc7f5cb653]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.754240 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6746c25111d04f20a6f30ce0838d71f5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:38.754442 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b141c2b67ffe43ff97d32c9fff1d1fe0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.754925 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_03aea495ba404f0e9997271e8221d441"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:38.755400 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128c88af54d24ffd895e9de19cdb4c8a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:38.755528 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03aea495ba404f0e9997271e8221d441 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.757164 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24a4104f4cb64889ad66810788ad6b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.757256 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 45f02efa659e457ab57bc32ed3f66bb9 (DEFAULT_TABLE table=default.Table_44a0f173f72b49eea2cc2243bc11c430 [id=27f4152aaee24a3bad52e0390b8fa649]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.757308 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 577b149f52e44d5d9a9f6b9d85a6864d (DEFAULT_TABLE table=default.Table_cb7682ca6f0f4d93922aa35e427c9cfa [id=89eea7c9830f47d2bd1c95044d5af4d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.757653 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45f02efa659e457ab57bc32ed3f66bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.757853 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ed919909289410e867881baab841f53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.757191 23896 tablet_service.cc:1467] Processing CreateTablet for tablet dd3078cc1f6247609f8bd017ed076b45 (DEFAULT_TABLE table=default.Table_bac7f229ce9541b39c0a70951431b689 [id=f9f2bf985f124269982869050bbecd70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.758277 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd3078cc1f6247609f8bd017ed076b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.759742 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577b149f52e44d5d9a9f6b9d85a6864d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:38.760828 23993 leader_election.cc:341] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.760984 23993 leader_election.cc:302] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:38.761417 24355 raft_consensus.cc:2726] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:38.763686 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8fe8a3cf1fbd4fe8825f078e4640f619" } new_table_name: "default.Table_922974ba60ab43e5af71756d9209fd0d" modify_external_catalogs: true
I20240502 08:16:38.766701 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 78ddfc7b74644a0bbf8222d5f90e6658 (DEFAULT_TABLE table=default.Table_0b7c860d457f421294ca1d37248e40b5 [id=c06501dd41a640279c769c9ab3926ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.766788 23892 tablet_service.cc:1467] Processing CreateTablet for tablet b9f83bcf640f465bb95d87a991cc5ff6 (DEFAULT_TABLE table=default.Table_e00515baeaae4ce9ad93000143f2a53a [id=13293a9ab7004e8a965e004f2fb5a14a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.767156 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 751808dfc1964b539f6f836d786156aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.769171 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0224317e65747e1add533d33472f7f3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:38.771327 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6417aafbdcf416ea7bff56bce5e6ddb consensus state has no leader
I20240502 08:16:38.772320 24180 tablet_replica.cc:330] T 9b1eb762780c450bbaf380d17db564b6 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:38.772984 24180 raft_consensus.cc:2229] T 9b1eb762780c450bbaf380d17db564b6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.774255 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet b6417aafbdcf416ea7bff56bce5e6ddb
I20240502 08:16:38.776068 23601 catalog_manager.cc:6554] ReplaceTablet: tablet b6417aafbdcf416ea7bff56bce5e6ddb deleted and replaced by tablet 96b8f21c64e343d88d84e9acd0bc69b4
I20240502 08:16:38.777155 24181 ts_tablet_manager.cc:1888] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.778565 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd3078cc1f6247609f8bd017ed076b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.781601 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 223c92ae8f63402f8643e03d914e8e7e (table default.Table_00eb74cd24954c52b5484751f035f4c1 [id=b42e15a2350f4c9a8097ccb0cf1aaddc]) successfully deleted
I20240502 08:16:38.781765 24175 tablet_bootstrap.cc:654] T 203ed05d0bdc4b1f99207423b7fc848b P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.782397 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ddfc7b74644a0bbf8222d5f90e6658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.782810 24179 ts_tablet_manager.cc:1888] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.783918 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0224317e65747e1add533d33472f7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.785403 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet dd3078cc1f6247609f8bd017ed076b45
I20240502 08:16:38.786666 24179 log.cc:1196] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/32150c7e130f4069a2a3a02e625bfb93
I20240502 08:16:38.787787 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f83bcf640f465bb95d87a991cc5ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.788511 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 33c8131846dc41678827b86e87be18d3 (DEFAULT_TABLE table=default.Table_df744f06850c416f98d94d2df132a03b [id=5849a3c8b28e47439b73e2137187fac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.790172 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 955e7bd2216646c6aef5acce8d5b0d77 (DEFAULT_TABLE table=default.Table_b3278c5d6d0b4593ac33bc9977f5c3b1 [id=e86031550cec46a793785c0b3859fa92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.791008 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33c8131846dc41678827b86e87be18d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.793464 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 955e7bd2216646c6aef5acce8d5b0d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.796502 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 4b8d81f071d74b218fcfd4f4e8c20bab (DEFAULT_TABLE table=default.Table_dbe385cc3f6b48d39bee298ca9490248 [id=aa825e5ca73c4dafa9585126d5c48df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.798723 24181 log.cc:1196] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/d5bab1b7c48248a7973c27d0a865b34b
I20240502 08:16:38.802732 23897 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1d946bd6de48f1a8d8fe5f179d22b6 (DEFAULT_TABLE table=default.Table_7e12ddab0f5241b4ac31de16f1467292 [id=6b44d73f1e0b4185a950f7bd4ed2f430]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.809079 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1d946bd6de48f1a8d8fe5f179d22b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.815122 24174 ts_tablet_manager.cc:1398] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.136s user 0.004s sys 0.001s
I20240502 08:16:38.817008 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1da01e2c2f054ccfb53383e70e16da84. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:38.817059 23723 leader_election.cc:341] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: ba3d4b4584dd45b58ff0daa4f1c09a0a
I20240502 08:16:38.817617 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1d946bd6de48f1a8d8fe5f179d22b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.821756 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b8d81f071d74b218fcfd4f4e8c20bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.824784 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7b37f776df47da9aa6e7fba82a8999. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.827912 23759 tablet_service.cc:1467] Processing CreateTablet for tablet a2da8b9d3f374d8a812336057c0ce7d2 (DEFAULT_TABLE table=default.Table_e87bd15719d642bfbc0c9ab785ff62d1 [id=76b58ce209a24b9a95b405c1c50de3ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.828339 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2da8b9d3f374d8a812336057c0ce7d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.829067 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b59c2f93d6514c9cbf358ab96f1a06b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.830809 23765 tablet_service.cc:1467] Processing CreateTablet for tablet fbef606ce7ac499d9bd84b49d804621c (DEFAULT_TABLE table=default.Table_1e7f2516f34b483d913cf5080a8febec [id=d5c976b2debc472fa1d7c28ae73ca32a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.831311 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbef606ce7ac499d9bd84b49d804621c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.832125 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet 5738cd77bbe147b49215ba9639d7f547 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.832482 23766 tablet_service.cc:1467] Processing CreateTablet for tablet c00f29fb76b44d6ea4f76963d2d0751c (DEFAULT_TABLE table=default.Table_480b6f612ca64d2caaa95f87d6ce8464 [id=a1004106109d47e7a1f9c6337480d3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.832826 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00f29fb76b44d6ea4f76963d2d0751c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.833184 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 9761df0bbdee44af90135bffb8b5828f (DEFAULT_TABLE table=default.Table_8ba240c391da43978f93a27dce48e557 [id=1b5536950db244d1978b154da0be33e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.834053 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ddfc7b74644a0bbf8222d5f90e6658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.834884 24180 raft_consensus.cc:2258] T 9b1eb762780c450bbaf380d17db564b6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.835553 24180 ts_tablet_manager.cc:1875] T 9b1eb762780c450bbaf380d17db564b6 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.836020 23761 tablet_service.cc:1467] Processing CreateTablet for tablet e47459b5f5bb4ecb96f76240fefa1f5d (DEFAULT_TABLE table=default.Table_8fe8a3cf1fbd4fe8825f078e4640f619 [id=51cd52edc7ae46738d9ffa2bd8083fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.836527 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47459b5f5bb4ecb96f76240fefa1f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.837754 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e7432f1df94431a83cf68356e3e78a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.838133 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e5de793e74247dd9e73b7b1d58027d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.838399 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet 848a0865eb2b4828a2594a17d96869d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.838768 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet 0882b7509a74475780eb700818aab398 with delete_type TABLET_DATA_DELETED (replaced by tablet eb96ad2289754e428a1c109b2d59c53b) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.839010 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet 71aa916247c449fea718b35869a6df20 with delete_type TABLET_DATA_DELETED (replaced by tablet 0501575a84ec454780f48574bf3155fe) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.839239 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet b11f504b93574e42a9bf30fde6d08207 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.839473 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c374f42f353497fa1d745164e7fbf61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.839702 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a1e208733844829b13222e00a432439 with delete_type TABLET_DATA_DELETED (replaced by tablet 8910bc67c6234a538c8160e7c020653c) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.839983 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet b6417aafbdcf416ea7bff56bce5e6ddb with delete_type TABLET_DATA_DELETED (replaced by tablet 96b8f21c64e343d88d84e9acd0bc69b4) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.840240 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet dd3078cc1f6247609f8bd017ed076b45 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.841747 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f83bcf640f465bb95d87a991cc5ff6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:38.842234 23723 leader_election.cc:341] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: c709efdd2ce74b5a8ab7278b402ebfe9
I20240502 08:16:38.842499 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577b149f52e44d5d9a9f6b9d85a6864d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.842336 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c1886293aee349b2b9e7177c1cfd6d26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.842922 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1886293aee349b2b9e7177c1cfd6d26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.844192 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 001e3f4f72f64dc69dbd521724912f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.847962 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45f02efa659e457ab57bc32ed3f66bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.850522 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d04312bd81f46ed8c49c42b1af1aec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.856020 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b8d81f071d74b218fcfd4f4e8c20bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.856309 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d04312bd81f46ed8c49c42b1af1aec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.857139 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 128c88af54d24ffd895e9de19cdb4c8a (DEFAULT_TABLE table=default.Table_d67be3894f494223a45d8c5a2c3e6113 [id=f7982b1d892942648cf3fca657c2668c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.858464 24355 raft_consensus.cc:482] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:16:38.860569 23723 leader_election.cc:341] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: e9592a041a9543e8b41f043927807131
I20240502 08:16:38.860886 24329 consensus_queue.cc:1035] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:38.865428 24174 tablet_bootstrap.cc:492] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:38.866966 24174 tablet_bootstrap.cc:654] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:38.867426 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1d946bd6de48f1a8d8fe5f179d22b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.868422 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33c8131846dc41678827b86e87be18d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.868638 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 955e7bd2216646c6aef5acce8d5b0d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.867779 24355 raft_consensus.cc:504] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.873482 24181 ts_tablet_manager.cc:1909] T d5bab1b7c48248a7973c27d0a865b34b P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:38.875684 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9761df0bbdee44af90135bffb8b5828f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.877238 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 6746c25111d04f20a6f30ce0838d71f5 (DEFAULT_TABLE table=default.Table_02f448a8a9844def8ecda8716fdc352e [id=79a03d22b7e34f76bb3242c640129a13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.879341 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b742c51f6204449a804889815604d5cb" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:38.879827 24483 raft_consensus.cc:482] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.879891 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 001e3f4f72f64dc69dbd521724912f07 (DEFAULT_TABLE table=default.Table_8b1436ec53e349658e5e39d3e10da3e1 [id=5c21117955d946418c656330984ae747]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.880023 24483 raft_consensus.cc:504] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.879837 24179 ts_tablet_manager.cc:1909] T 32150c7e130f4069a2a3a02e625bfb93 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:38.882874 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ddfc7b74644a0bbf8222d5f90e6658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.882822 24029 tablet_service.cc:1467] Processing CreateTablet for tablet a2da8b9d3f374d8a812336057c0ce7d2 (DEFAULT_TABLE table=default.Table_e87bd15719d642bfbc0c9ab785ff62d1 [id=76b58ce209a24b9a95b405c1c50de3ca]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:38.884301 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e47459b5f5bb4ecb96f76240fefa1f5d consensus state has no leader
I20240502 08:16:38.887853 24173 tablet_bootstrap.cc:492] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:38.887802 24180 ts_tablet_manager.cc:1888] T 9b1eb762780c450bbaf380d17db564b6 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:38.888100 24180 log.cc:1196] T 9b1eb762780c450bbaf380d17db564b6 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/9b1eb762780c450bbaf380d17db564b6
I20240502 08:16:38.889187 24180 ts_tablet_manager.cc:1909] T 9b1eb762780c450bbaf380d17db564b6 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:38.889290 24355 leader_election.cc:288] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
W20240502 08:16:38.890380 23993 leader_election.cc:341] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.891119 24180 tablet_replica.cc:330] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:38.891165 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 9b1eb762780c450bbaf380d17db564b6 (table default.Table_4dd22967bcf147388ea4ceb9cd24b065 [id=ba56850714c2422eb5dbbc0f49df7ce7]) successfully deleted
I20240502 08:16:38.891567 24180 raft_consensus.cc:2229] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.891909 24180 raft_consensus.cc:2258] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.892740 24173 ts_tablet_manager.cc:1367] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.267s user 0.011s sys 0.027s
I20240502 08:16:38.893273 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128c88af54d24ffd895e9de19cdb4c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.893818 23608 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 08183523bf4a4d1c9020eb693aa07878 (table default.Table_fcd8e870383b4c8f9b40db6fb2a7149b [id=0a2f8dbb62a64b0a96931ab5f8c17540]). Expected version 1 got 0
I20240502 08:16:38.893779 24034 tablet_service.cc:1467] Processing CreateTablet for tablet fbef606ce7ac499d9bd84b49d804621c (DEFAULT_TABLE table=default.Table_1e7f2516f34b483d913cf5080a8febec [id=d5c976b2debc472fa1d7c28ae73ca32a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.895061 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f83bcf640f465bb95d87a991cc5ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.895367 24181 tablet_replica.cc:330] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:38.896582 24181 raft_consensus.cc:2229] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.896762 24181 raft_consensus.cc:2258] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.897325 24181 ts_tablet_manager.cc:1875] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.897637 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e7432f1df94431a83cf68356e3e78a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.897853 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbef606ce7ac499d9bd84b49d804621c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.898970 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2da8b9d3f374d8a812336057c0ce7d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.900449 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 001e3f4f72f64dc69dbd521724912f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.900357 23751 tablet_service.cc:1467] Processing CreateTablet for tablet af6efeebaebc495aba017a922e243c39 (DEFAULT_TABLE table=default.Table_0f919156226c440294ddd9a6e379cbff [id=f293b22818344aea9eb6232141432183]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.901156 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af6efeebaebc495aba017a922e243c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.904670 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet e9592a041a9543e8b41f043927807131 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:38.905551 23752 tablet_service.cc:1467] Processing CreateTablet for tablet edd7244b9f5741a8aa1c6835fe544676 (DEFAULT_TABLE table=default.Table_fec81a33cad74ba3bfbc7787a7bd5829 [id=9775ed3c642a4d7e81d5abc48acca163]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.906345 23753 tablet_service.cc:1467] Processing CreateTablet for tablet ee022e8e86f8477a8fa1e58d52644b03 (DEFAULT_TABLE table=default.Table_d6f5ca82488a40a69121f1b779d8f874 [id=0422356f66a74ad29c2e13e1ad36b654]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.906579 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd7244b9f5741a8aa1c6835fe544676. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.906818 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee022e8e86f8477a8fa1e58d52644b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.907127 24021 tablet_service.cc:1467] Processing CreateTablet for tablet c00f29fb76b44d6ea4f76963d2d0751c (DEFAULT_TABLE table=default.Table_480b6f612ca64d2caaa95f87d6ce8464 [id=a1004106109d47e7a1f9c6337480d3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.908286 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6746c25111d04f20a6f30ce0838d71f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.911561 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00f29fb76b44d6ea4f76963d2d0751c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.910871 24173 raft_consensus.cc:348] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.914472 23755 tablet_service.cc:1467] Processing CreateTablet for tablet e0b0d4871dd24607854100a4ef762e25 (DEFAULT_TABLE table=default.Table_5dcb3ffd44ea42de93aad8dcced74ed4 [id=910c7cae36264e39bbfca0be919538e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.914875 23760 tablet_service.cc:1467] Processing CreateTablet for tablet df08f0d3af4c4c7e8abce656714b1b7f (DEFAULT_TABLE table=default.Table_59f82dbde55b406ba5271520c5bab104 [id=8e9acf4c0dbf4590b82aaee07243ed0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.915248 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df08f0d3af4c4c7e8abce656714b1b7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.917241 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 89f57e3a39d54b7ab372d4b145b7a696 (DEFAULT_TABLE table=default.Table_28b3620523074c23866e772b117bf0ee [id=b09ad60d787b4740a493cb73c22b94a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.917896 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f57e3a39d54b7ab372d4b145b7a696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.917855 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_da5548ba323e495dafdd33eb59a9c6b7" } new_table_name: "default.Table_8a9cdf1f53b0494694035cd549988bf0" modify_external_catalogs: true
I20240502 08:16:38.918201 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b0d4871dd24607854100a4ef762e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.905673 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 25c089f0c1b14e419f250d03d4eaad12 (DEFAULT_TABLE table=default.Table_94efd28ada5b4024ab65e4d5feda0b38 [id=15b42ca7619645c8ac3cff50044f069b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.905951 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 07344b78894c42809391d0fe2a9cd8f7 (DEFAULT_TABLE table=default.Table_7e64e6023d104b96b6f3ae5c7963e32e [id=6b6ffe47b9534a618f0912d78e7bb92a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.906219 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 18d284f71937459db0c0676ce4568175 (DEFAULT_TABLE table=default.Table_7af51f5f2feb42ca8dbe067300bbbfd9 [id=7eff9f0e4b304a01b8d8c521bfdd1563]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.919829 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fb39fa7949f3412c83974132184b5d9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:38.920212 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_dc7d5494bb5d436486375f565bc4353b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.920393 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb39fa7949f3412c83974132184b5d9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:38.920702 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc7d5494bb5d436486375f565bc4353b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.921962 24483 leader_election.cc:288] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:38.922550 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b742c51f6204449a804889815604d5cb" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:38.923547 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_03aea495ba404f0e9997271e8221d441" } modify_external_catalogs: true
I20240502 08:16:38.923713 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_03aea495ba404f0e9997271e8221d441" } modify_external_catalogs: true
I20240502 08:16:38.924794 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7af51f5f2feb42ca8dbe067300bbbfd9" } modify_external_catalogs: true
I20240502 08:16:38.924968 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7af51f5f2feb42ca8dbe067300bbbfd9" } modify_external_catalogs: true
I20240502 08:16:38.925396 23601 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 08183523bf4a4d1c9020eb693aa07878 (table default.Table_fcd8e870383b4c8f9b40db6fb2a7149b [id=0a2f8dbb62a64b0a96931ab5f8c17540]). Expected version 1 got 0
I20240502 08:16:38.925870 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_97709af0716042cebdc01ed2d06dcf20" } new_table_name: "default.Table_ad81523678c3450c880efd1ecfc5f95e" modify_external_catalogs: true
I20240502 08:16:38.926512 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_f96549ae28214964b5530d879c8f04b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:38.926688 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9663c413d5be4a049eb7a02ececc1276" } modify_external_catalogs: true
I20240502 08:16:38.926901 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9663c413d5be4a049eb7a02ececc1276" } modify_external_catalogs: true
W20240502 08:16:38.927079 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f96549ae28214964b5530d879c8f04b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.927614 24173 raft_consensus.cc:374] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:38.928292 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07344b78894c42809391d0fe2a9cd8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.928591 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 5d04312bd81f46ed8c49c42b1af1aec9 (DEFAULT_TABLE table=default.Table_9741459a96e649fb8151cc28b642523e [id=2b9169f285754bfda0f4f8dc7f5cb653]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.928937 24179 tablet_replica.cc:330] T 223c92ae8f63402f8643e03d914e8e7e P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:38.929839 24173 raft_consensus.cc:729] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:38.931392 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18d284f71937459db0c0676ce4568175. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.936816 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 128c88af54d24ffd895e9de19cdb4c8a (DEFAULT_TABLE table=default.Table_d67be3894f494223a45d8c5a2c3e6113 [id=f7982b1d892942648cf3fca657c2668c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.936859 23896 tablet_service.cc:1467] Processing CreateTablet for tablet b59c2f93d6514c9cbf358ab96f1a06b5 (DEFAULT_TABLE table=default.Table_bbf9d7694fe942de8aa8bd4ed61336a5 [id=f1d6eb661b0143ee82e797cde61391c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.937532 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0a4a6e3b1dd044f1a0e88c6c03f8c8b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.938150 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a4a6e3b1dd044f1a0e88c6c03f8c8b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.938127 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a92f0fc8bc664448a6acb94dec9daa14" } modify_external_catalogs: true
I20240502 08:16:38.938289 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a92f0fc8bc664448a6acb94dec9daa14" } modify_external_catalogs: true
I20240502 08:16:38.930150 24173 consensus_queue.cc:260] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:38.941251 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 090bec30f5664421b9bfe3f2810afd14 (DEFAULT_TABLE table=default.Table_70034c04442f49f6a42cb66dccfd1cc1 [id=74be9e9508024ebeb4623c488e21faf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.943150 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 592ab8523b8d479d9caead321d5888c3 (DEFAULT_TABLE table=default.Table_af7c06dd8b3f4085b62eaef9ff9b57cb [id=11e47f3f89ff4bbbbb32ab105efe686c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.945909 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25c089f0c1b14e419f250d03d4eaad12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.946669 24329 raft_consensus.cc:482] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.946831 24329 raft_consensus.cc:504] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:38.948864 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 592eeecbf5514215b6464bd030081e4b (DEFAULT_TABLE table=default.Table_cda7da2a134f43a8a3c79a6609c3e52e [id=07050bd496e84a97a86fdc4f0b7811ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.949584 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 9761df0bbdee44af90135bffb8b5828f (DEFAULT_TABLE table=default.Table_8ba240c391da43978f93a27dce48e557 [id=1b5536950db244d1978b154da0be33e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.949971 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9761df0bbdee44af90135bffb8b5828f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.958643 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 9761df0bbdee44af90135bffb8b5828f (DEFAULT_TABLE table=default.Table_8ba240c391da43978f93a27dce48e557 [id=1b5536950db244d1978b154da0be33e0]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:38.959517 23861 leader_election.cc:341] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: da4be1d381a64b7baeb1c60cb17d924a
I20240502 08:16:38.959812 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 001e3f4f72f64dc69dbd521724912f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.959877 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 32150c7e130f4069a2a3a02e625bfb93 (table default.Table_90cefac5d31b4c449140db4b7772a0d2 [id=a895b3f375374ad6bb44a6f87fc7b98d]) successfully deleted
I20240502 08:16:38.966809 24173 ts_tablet_manager.cc:1398] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.055s user 0.001s sys 0.001s
I20240502 08:16:38.969872 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 592ab8523b8d479d9caead321d5888c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.971133 24173 tablet_bootstrap.cc:492] T 43e870c3a0e340d1bad1c1fa0380a762 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:38.974462 24483 raft_consensus.cc:482] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:16:38.974630 23861 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 68731.2us
W20240502 08:16:38.976078 23996 leader_election.cc:341] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:38.976213 23996 leader_election.cc:302] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:38.976462 24355 raft_consensus.cc:2726] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:38.978047 23607 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 1ce5ccfd29724dc49afd0892b99f3169 (table default.Table_bdbe807258a5454795eaff8c27bb17dd [id=3a9d7e8fe2c84cb4942fee9053a012b0]). Expected version 1 got 0
I20240502 08:16:38.978749 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fba18fd93e6c4364bead807a40bd1689" } new_table_name: "default.Table_22b226147b894b50a865f33b9f41491c" modify_external_catalogs: true
W20240502 08:16:38.978865 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 77c49da1e0f0435ba241a6a66526e790 consensus state has no leader
I20240502 08:16:38.979283 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_3e9140f5996646cbb7cad1837c94d43e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.979816 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e9140f5996646cbb7cad1837c94d43e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.979997 23606 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 53171eaf7ae94cf4941dbaf26712a084
I20240502 08:16:38.980546 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 18d284f71937459db0c0676ce4568175
I20240502 08:16:38.981415 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_39406aba7a1b46719724dece66b69fdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.981911 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39406aba7a1b46719724dece66b69fdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.982822 24179 raft_consensus.cc:2229] T 223c92ae8f63402f8643e03d914e8e7e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:38.982862 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 4b8d81f071d74b218fcfd4f4e8c20bab (DEFAULT_TABLE table=default.Table_dbe385cc3f6b48d39bee298ca9490248 [id=aa825e5ca73c4dafa9585126d5c48df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.983081 24179 raft_consensus.cc:2258] T 223c92ae8f63402f8643e03d914e8e7e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:38.984050 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0d21bbe9f22f46dabc7701e182ea31b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:38.984380 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ae8d973c3d53448d9068ba92779c32df" } new_table_name: "default.Table_299ae144ccc6402abe33d07ef693e5b4" modify_external_catalogs: true
W20240502 08:16:38.984535 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d21bbe9f22f46dabc7701e182ea31b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:38.985734 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 751808dfc1964b539f6f836d786156aa consensus state has no leader
I20240502 08:16:38.986338 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 9c03a4353c4e46e3bcdf3a36aac04bf5
I20240502 08:16:38.987049 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_3c57dba4df6d4a139583cb8ccb6e6ee9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.987481 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c57dba4df6d4a139583cb8ccb6e6ee9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:38.975210 24058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08183523bf4a4d1c9020eb693aa07878" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:38.989661 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 090bec30f5664421b9bfe3f2810afd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:38.993795 24058 raft_consensus.cc:2454] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:38.995127 24175 tablet_bootstrap.cc:492] T 203ed05d0bdc4b1f99207423b7fc848b P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:38.995414 24483 raft_consensus.cc:504] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:38.995620 24175 ts_tablet_manager.cc:1367] T 203ed05d0bdc4b1f99207423b7fc848b P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.328s user 0.011s sys 0.004s
I20240502 08:16:38.995707 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 6746c25111d04f20a6f30ce0838d71f5 (DEFAULT_TABLE table=default.Table_02f448a8a9844def8ecda8716fdc352e [id=79a03d22b7e34f76bb3242c640129a13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.995208 23888 tablet_service.cc:1467] Processing CreateTablet for tablet a2da8b9d3f374d8a812336057c0ce7d2 (DEFAULT_TABLE table=default.Table_e87bd15719d642bfbc0c9ab785ff62d1 [id=76b58ce209a24b9a95b405c1c50de3ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.995256 23886 tablet_service.cc:1514] Processing DeleteTablet for tablet b63fe5b6d4db4750800ecd429fa97ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:38.996604 23886 tablet_service.cc:1467] Processing CreateTablet for tablet e47459b5f5bb4ecb96f76240fefa1f5d (DEFAULT_TABLE table=default.Table_8fe8a3cf1fbd4fe8825f078e4640f619 [id=51cd52edc7ae46738d9ffa2bd8083fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.995301 23887 tablet_service.cc:1467] Processing CreateTablet for tablet fbef606ce7ac499d9bd84b49d804621c (DEFAULT_TABLE table=default.Table_1e7f2516f34b483d913cf5080a8febec [id=d5c976b2debc472fa1d7c28ae73ca32a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.995345 23895 tablet_service.cc:1467] Processing CreateTablet for tablet c00f29fb76b44d6ea4f76963d2d0751c (DEFAULT_TABLE table=default.Table_480b6f612ca64d2caaa95f87d6ce8464 [id=a1004106109d47e7a1f9c6337480d3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:38.995157 24179 ts_tablet_manager.cc:1875] T 223c92ae8f63402f8643e03d914e8e7e P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:38.997124 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0a0743012671403884555c6274a7f3da" } modify_external_catalogs: true
I20240502 08:16:38.997314 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0a0743012671403884555c6274a7f3da" } modify_external_catalogs: true
I20240502 08:16:38.997838 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_f0695a0f3e5c48c7aa4f998a18be36e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:38.998308 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0695a0f3e5c48c7aa4f998a18be36e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.002939 23723 leader_election.cc:302] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:39.004022 24489 raft_consensus.cc:2781] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:39.006980 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e5de793e74247dd9e73b7b1d58027d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.008049 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 848a0865eb2b4828a2594a17d96869d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.008282 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 0882b7509a74475780eb700818aab398 with delete_type TABLET_DATA_DELETED (replaced by tablet eb96ad2289754e428a1c109b2d59c53b) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.008472 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 71aa916247c449fea718b35869a6df20 with delete_type TABLET_DATA_DELETED (replaced by tablet 0501575a84ec454780f48574bf3155fe) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.008643 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet b11f504b93574e42a9bf30fde6d08207 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:39.011760 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43e870c3a0e340d1bad1c1fa0380a762 consensus state has no leader
I20240502 08:16:39.012497 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 5453cc0b93ac4cbb90a0588bb2100d60
I20240502 08:16:39.012773 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet f51b26043ef44b30846498cd1497607c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.012953 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c374f42f353497fa1d745164e7fbf61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.013144 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a1e208733844829b13222e00a432439 with delete_type TABLET_DATA_DELETED (replaced by tablet 8910bc67c6234a538c8160e7c020653c) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.013305 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet b6417aafbdcf416ea7bff56bce5e6ddb with delete_type TABLET_DATA_DELETED (replaced by tablet 96b8f21c64e343d88d84e9acd0bc69b4) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:39.013922 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a198c33a3e314b5b81a08244862b281a has no consensus state
I20240502 08:16:39.019872 24483 leader_election.cc:288] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:39.020771 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d04312bd81f46ed8c49c42b1af1aec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.013509 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet dd3078cc1f6247609f8bd017ed076b45 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.016659 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 592ab8523b8d479d9caead321d5888c3 (DEFAULT_TABLE table=default.Table_af7c06dd8b3f4085b62eaef9ff9b57cb [id=11e47f3f89ff4bbbbb32ab105efe686c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.022076 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 592eeecbf5514215b6464bd030081e4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.022682 24483 raft_consensus.cc:482] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:39.022825 24483 raft_consensus.cc:504] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.025607 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 848a0865eb2b4828a2594a17d96869d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.026098 24483 leader_election.cc:288] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:39.030282 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 71aa916247c449fea718b35869a6df20 with delete_type TABLET_DATA_DELETED (replaced by tablet 0501575a84ec454780f48574bf3155fe) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.030503 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet b11f504b93574e42a9bf30fde6d08207 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.030586 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.030691 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c374f42f353497fa1d745164e7fbf61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.030733 23752 tablet_service.cc:1467] Processing CreateTablet for tablet ba3aa2621fdb43fd8299f81af486ef1e (DEFAULT_TABLE table=default.Table_30df43df108e43d4b44b20c60eeb8287 [id=f8b12b3bb4904c18be60b567cb05b739]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.030844 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a1e208733844829b13222e00a432439 with delete_type TABLET_DATA_DELETED (replaced by tablet 8910bc67c6234a538c8160e7c020653c) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.031035 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet b6417aafbdcf416ea7bff56bce5e6ddb with delete_type TABLET_DATA_DELETED (replaced by tablet 96b8f21c64e343d88d84e9acd0bc69b4) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.031186 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet dd3078cc1f6247609f8bd017ed076b45 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.031515 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 592ab8523b8d479d9caead321d5888c3 (DEFAULT_TABLE table=default.Table_af7c06dd8b3f4085b62eaef9ff9b57cb [id=11e47f3f89ff4bbbbb32ab105efe686c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.034910 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b59c2f93d6514c9cbf358ab96f1a06b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.035987 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet 0882b7509a74475780eb700818aab398 with delete_type TABLET_DATA_DELETED (replaced by tablet eb96ad2289754e428a1c109b2d59c53b) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.036348 23892 tablet_service.cc:1467] Processing CreateTablet for tablet af6efeebaebc495aba017a922e243c39 (DEFAULT_TABLE table=default.Table_0f919156226c440294ddd9a6e379cbff [id=f293b22818344aea9eb6232141432183]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.038180 23900 tablet_service.cc:1467] Processing CreateTablet for tablet edd7244b9f5741a8aa1c6835fe544676 (DEFAULT_TABLE table=default.Table_fec81a33cad74ba3bfbc7787a7bd5829 [id=9775ed3c642a4d7e81d5abc48acca163]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.039510 24329 leader_election.cc:288] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:39.040963 23602 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 55d221e4faf04b4d8ad217e833d8bff6
I20240502 08:16:39.041014 23723 leader_election.cc:302] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:39.041522 24329 raft_consensus.cc:2781] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:39.041647 24329 raft_consensus.cc:482] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:39.041738 24329 raft_consensus.cc:3037] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.042212 23761 tablet_service.cc:1467] Processing CreateTablet for tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 (DEFAULT_TABLE table=default.Table_e9f712275e34498c9f78493dc08b37f5 [id=b2cb0ed2926d4be7ade0790203b57e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.042692 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6a19bb8705c45c0bfc537d3a6bf8ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.042088 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 5453cc0b93ac4cbb90a0588bb2100d60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.042963 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 25c089f0c1b14e419f250d03d4eaad12 (DEFAULT_TABLE table=default.Table_94efd28ada5b4024ab65e4d5feda0b38 [id=15b42ca7619645c8ac3cff50044f069b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.043726 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet 18d284f71937459db0c0676ce4568175 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.044589 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00f29fb76b44d6ea4f76963d2d0751c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.044791 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3aa2621fdb43fd8299f81af486ef1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.045744 24180 ts_tablet_manager.cc:1875] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:39.049037 24329 raft_consensus.cc:504] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.050133 24329 leader_election.cc:288] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:39.050647 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b8d81f071d74b218fcfd4f4e8c20bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.052690 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128c88af54d24ffd895e9de19cdb4c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.055605 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 18d284f71937459db0c0676ce4568175 (DEFAULT_TABLE table=default.Table_7af51f5f2feb42ca8dbe067300bbbfd9 [id=7eff9f0e4b304a01b8d8c521bfdd1563]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.059716 23897 tablet_service.cc:1467] Processing CreateTablet for tablet 07344b78894c42809391d0fe2a9cd8f7 (DEFAULT_TABLE table=default.Table_7e64e6023d104b96b6f3ae5c7963e32e [id=6b6ffe47b9534a618f0912d78e7bb92a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.061983 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18d284f71937459db0c0676ce4568175. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.062608 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbef606ce7ac499d9bd84b49d804621c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.063086 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47459b5f5bb4ecb96f76240fefa1f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.065025 24174 tablet_bootstrap.cc:492] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:39.065209 24174 ts_tablet_manager.cc:1367] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.200s user 0.005s sys 0.011s
I20240502 08:16:39.065766 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2da8b9d3f374d8a812336057c0ce7d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.066013 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25c089f0c1b14e419f250d03d4eaad12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.066072 24174 raft_consensus.cc:348] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.066263 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6746c25111d04f20a6f30ce0838d71f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.066305 24174 raft_consensus.cc:374] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.066382 24174 raft_consensus.cc:729] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.066658 24174 consensus_queue.cc:260] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.067157 24174 ts_tablet_manager.cc:1398] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:39.067523 24174 tablet_bootstrap.cc:492] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:39.069248 24174 tablet_bootstrap.cc:654] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.071627 23898 tablet_service.cc:1467] Processing CreateTablet for tablet ee022e8e86f8477a8fa1e58d52644b03 (DEFAULT_TABLE table=default.Table_d6f5ca82488a40a69121f1b779d8f874 [id=0422356f66a74ad29c2e13e1ad36b654]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.072542 23891 tablet_service.cc:1467] Processing CreateTablet for tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 (DEFAULT_TABLE table=default.Table_e9f712275e34498c9f78493dc08b37f5 [id=b2cb0ed2926d4be7ade0790203b57e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.075271 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 592eeecbf5514215b6464bd030081e4b (DEFAULT_TABLE table=default.Table_cda7da2a134f43a8a3c79a6609c3e52e [id=07050bd496e84a97a86fdc4f0b7811ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.077740 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd7244b9f5741a8aa1c6835fe544676. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.078445 23904 tablet_service.cc:1467] Processing CreateTablet for tablet ba3aa2621fdb43fd8299f81af486ef1e (DEFAULT_TABLE table=default.Table_30df43df108e43d4b44b20c60eeb8287 [id=f8b12b3bb4904c18be60b567cb05b739]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.081233 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07344b78894c42809391d0fe2a9cd8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.083058 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 592ab8523b8d479d9caead321d5888c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.085009 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af6efeebaebc495aba017a922e243c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.086071 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_df744f06850c416f98d94d2df132a03b" } new_table_name: "default.Table_bd9708a3da8240b4aa9444c8e9ed30d2" modify_external_catalogs: true
I20240502 08:16:39.089403 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9761df0bbdee44af90135bffb8b5828f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.090198 23899 tablet_service.cc:1467] Processing CreateTablet for tablet df08f0d3af4c4c7e8abce656714b1b7f (DEFAULT_TABLE table=default.Table_59f82dbde55b406ba5271520c5bab104 [id=8e9acf4c0dbf4590b82aaee07243ed0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.091270 24175 raft_consensus.cc:348] T 203ed05d0bdc4b1f99207423b7fc848b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.091840 24175 raft_consensus.cc:374] T 203ed05d0bdc4b1f99207423b7fc848b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.091991 24175 raft_consensus.cc:729] T 203ed05d0bdc4b1f99207423b7fc848b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.092429 24175 consensus_queue.cc:260] T 203ed05d0bdc4b1f99207423b7fc848b P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.093288 24175 ts_tablet_manager.cc:1398] T 203ed05d0bdc4b1f99207423b7fc848b P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.098s user 0.002s sys 0.000s
I20240502 08:16:39.093711 24175 tablet_bootstrap.cc:492] T 5738cd77bbe147b49215ba9639d7f547 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:39.095147 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e5de793e74247dd9e73b7b1d58027d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.095630 23889 tablet_service.cc:1467] Processing CreateTablet for tablet e0b0d4871dd24607854100a4ef762e25 (DEFAULT_TABLE table=default.Table_5dcb3ffd44ea42de93aad8dcced74ed4 [id=910c7cae36264e39bbfca0be919538e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.095790 24174 tablet_bootstrap.cc:492] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:39.095930 24174 ts_tablet_manager.cc:1367] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.008s
I20240502 08:16:39.096729 24174 raft_consensus.cc:348] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.096953 24174 raft_consensus.cc:374] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.097064 24174 raft_consensus.cc:729] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.097326 24174 consensus_queue.cc:260] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.099308 23858 leader_election.cc:302] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
W20240502 08:16:39.100338 23858 leader_election.cc:341] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 485c772245fe48a39fc6ceb6377ac0b6
I20240502 08:16:39.100419 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_da999521d34e478fab2c024a89fa0025"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:39.100747 23858 leader_election.cc:302] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:39.101080 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da999521d34e478fab2c024a89fa0025 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.101275 24483 raft_consensus.cc:2781] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:39.101912 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 592ab8523b8d479d9caead321d5888c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.102967 24180 ts_tablet_manager.cc:1888] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:39.103433 24180 log.cc:1196] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/4ea2dee2d5914cf0bf4b37d16865e9b0
I20240502 08:16:39.103953 24180 ts_tablet_manager.cc:1909] T 4ea2dee2d5914cf0bf4b37d16865e9b0 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:39.103220 24489 raft_consensus.cc:686] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:39.105283 24489 consensus_queue.cc:237] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.106958 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6a19bb8705c45c0bfc537d3a6bf8ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.111546 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 264) took 1101 ms. Trace:
I20240502 08:16:39.111704 24180 rpcz_store.cc:276] 0502 08:16:38.009891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.061650 (+ 51759us) service_pool.cc:225] Handling call
0502 08:16:38.890987 (+829337us) ts_tablet_manager.cc:1120] Deleting tablet 4ea2dee2d5914cf0bf4b37d16865e9b0
0502 08:16:38.890997 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:39.063040 (+172043us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:39.102820 (+ 39780us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:39.111262 (+ 8442us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":825983}
I20240502 08:16:39.113659 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 89f57e3a39d54b7ab372d4b145b7a696 (DEFAULT_TABLE table=default.Table_28b3620523074c23866e772b117bf0ee [id=b09ad60d787b4740a493cb73c22b94a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.114112 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f57e3a39d54b7ab372d4b145b7a696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.114477 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08183523bf4a4d1c9020eb693aa07878" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:39.114717 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet b742c51f6204449a804889815604d5cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.115136 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 4ea2dee2d5914cf0bf4b37d16865e9b0 (table default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418]) successfully deleted
I20240502 08:16:39.112021 24174 ts_tablet_manager.cc:1398] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.015s user 0.002s sys 0.002s
I20240502 08:16:39.115665 24174 tablet_bootstrap.cc:492] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:39.116459 24180 tablet_replica.cc:330] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:39.116709 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 25c089f0c1b14e419f250d03d4eaad12 (DEFAULT_TABLE table=default.Table_94efd28ada5b4024ab65e4d5feda0b38 [id=15b42ca7619645c8ac3cff50044f069b]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:39.117744 23861 leader_election.cc:341] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: ba3d4b4584dd45b58ff0daa4f1c09a0a
I20240502 08:16:39.118116 24174 tablet_bootstrap.cc:654] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.119923 24038 tablet_service.cc:1467] Processing CreateTablet for tablet af6efeebaebc495aba017a922e243c39 (DEFAULT_TABLE table=default.Table_0f919156226c440294ddd9a6e379cbff [id=f293b22818344aea9eb6232141432183]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.117715 24040 tablet_service.cc:1467] Processing CreateTablet for tablet e47459b5f5bb4ecb96f76240fefa1f5d (DEFAULT_TABLE table=default.Table_8fe8a3cf1fbd4fe8825f078e4640f619 [id=51cd52edc7ae46738d9ffa2bd8083fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.123740 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee022e8e86f8477a8fa1e58d52644b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.124367 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3aa2621fdb43fd8299f81af486ef1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.125339 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 592eeecbf5514215b6464bd030081e4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.126348 24557 raft_consensus.cc:2726] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:39.126950 24180 raft_consensus.cc:2229] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:39.128032 23610 master_service.cc:946] ReplaceTablet: received request to replace tablet b6417aafbdcf416ea7bff56bce5e6ddb from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:39.128350 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df08f0d3af4c4c7e8abce656714b1b7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.129055 24180 pending_rounds.cc:62] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:16:39.129137 24180 pending_rounds.cc:68] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156425139658752 op_type: NO_OP noop_request { }
I20240502 08:16:39.129305 24180 raft_consensus.cc:2866] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:39.129238 24483 raft_consensus.cc:686] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:16:39.129415 24180 raft_consensus.cc:2258] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:39.129755 24483 consensus_queue.cc:237] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.130347 24180 ts_tablet_manager.cc:1875] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:39.136008 24058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08183523bf4a4d1c9020eb693aa07878" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:39.137674 24180 ts_tablet_manager.cc:1888] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:39.137812 24180 log.cc:1196] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/c709efdd2ce74b5a8ab7278b402ebfe9
I20240502 08:16:39.138218 24180 ts_tablet_manager.cc:1909] T c709efdd2ce74b5a8ab7278b402ebfe9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:39.138978 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 07344b78894c42809391d0fe2a9cd8f7 (DEFAULT_TABLE table=default.Table_7e64e6023d104b96b6f3ae5c7963e32e [id=6b6ffe47b9534a618f0912d78e7bb92a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.139258 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet c709efdd2ce74b5a8ab7278b402ebfe9 (table default.Table_34b1b262bb424bde8bd994ada1f03434 [id=322a04a326c24fc4ad2c3f1384db0fd3]) successfully deleted
I20240502 08:16:39.140225 24174 tablet_bootstrap.cc:492] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:39.140385 24174 ts_tablet_manager.cc:1367] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.001s
I20240502 08:16:39.141144 24174 raft_consensus.cc:348] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.141351 24174 raft_consensus.cc:374] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.141423 24174 raft_consensus.cc:729] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.141710 24175 tablet_bootstrap.cc:654] T 5738cd77bbe147b49215ba9639d7f547 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.141683 24174 consensus_queue.cc:260] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.142309 24174 ts_tablet_manager.cc:1398] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:39.142587 24174 tablet_bootstrap.cc:492] T f36ceb0222ef473a828ca795f393174d P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:39.142661 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b0d4871dd24607854100a4ef762e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.144068 24174 tablet_bootstrap.cc:654] T f36ceb0222ef473a828ca795f393174d P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.147138 24179 ts_tablet_manager.cc:1888] T 223c92ae8f63402f8643e03d914e8e7e P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:39.148893 24180 tablet_replica.cc:330] T 1df565c1c6564efd88fa557c50522d7e P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:39.149142 24180 raft_consensus.cc:2229] T 1df565c1c6564efd88fa557c50522d7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:39.149323 24180 raft_consensus.cc:2258] T 1df565c1c6564efd88fa557c50522d7e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:39.149260 23603 master_service.cc:946] ReplaceTablet: received request to replace tablet dd3078cc1f6247609f8bd017ed076b45 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:39.150192 24180 ts_tablet_manager.cc:1875] T 1df565c1c6564efd88fa557c50522d7e P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:39.152779 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 33c8131846dc41678827b86e87be18d3 consensus state has no leader
I20240502 08:16:39.154580 23605 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 1ce5ccfd29724dc49afd0892b99f3169 (table default.Table_bdbe807258a5454795eaff8c27bb17dd [id=3a9d7e8fe2c84cb4942fee9053a012b0]). Expected version 1 got 0
I20240502 08:16:39.157743 23752 tablet_service.cc:1467] Processing CreateTablet for tablet 4c395a29c7a1485eab39ca107dd91fca (DEFAULT_TABLE table=default.Table_3c57dba4df6d4a139583cb8ccb6e6ee9 [id=272acbb9a6ca4f2dbcbfea6aacbe9cfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.158116 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c395a29c7a1485eab39ca107dd91fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.161715 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet a7c3800fb2784a1dba12d01c56c226e5 with delete_type TABLET_DATA_DELETED (replaced by tablet f4b291bdb81a4efbac7f6691e5577bbe) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.161620 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 090bec30f5664421b9bfe3f2810afd14 (DEFAULT_TABLE table=default.Table_70034c04442f49f6a42cb66dccfd1cc1 [id=74be9e9508024ebeb4623c488e21faf9]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:39.163645 23858 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 48568.6us
I20240502 08:16:39.163985 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 156c7356ece342aea0d8db667ebad89c (DEFAULT_TABLE table=default.Table_39406aba7a1b46719724dece66b69fdd [id=15b52f0688c24df2b6f3fda3732ceeb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.164429 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156c7356ece342aea0d8db667ebad89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.164484 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 18d284f71937459db0c0676ce4568175 (DEFAULT_TABLE table=default.Table_7af51f5f2feb42ca8dbe067300bbbfd9 [id=7eff9f0e4b304a01b8d8c521bfdd1563]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.166524 24031 tablet_service.cc:1467] Processing CreateTablet for tablet edd7244b9f5741a8aa1c6835fe544676 (DEFAULT_TABLE table=default.Table_fec81a33cad74ba3bfbc7787a7bd5829 [id=9775ed3c642a4d7e81d5abc48acca163]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.169920 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 63afd6e4fee24edeaba1209105f1c3e6 (DEFAULT_TABLE table=default.Table_3e9140f5996646cbb7cad1837c94d43e [id=84e36915119b4125a1b0d64b2d1647fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.170315 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63afd6e4fee24edeaba1209105f1c3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.172590 24179 log.cc:1196] T 223c92ae8f63402f8643e03d914e8e7e P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/223c92ae8f63402f8643e03d914e8e7e
I20240502 08:16:39.173880 23761 tablet_service.cc:1467] Processing CreateTablet for tablet ec10ff7251a5442a8ecc1881b34db74d (DEFAULT_TABLE table=default.Table_dc7d5494bb5d436486375f565bc4353b [id=1d7a43116fe74c35b505eccd165f99cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.174280 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec10ff7251a5442a8ecc1881b34db74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.176352 23603 master_service.cc:946] ReplaceTablet: received request to replace tablet 98a96292b28a4b0aad88fe0a96730a68 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:39.177489 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 592eeecbf5514215b6464bd030081e4b (DEFAULT_TABLE table=default.Table_cda7da2a134f43a8a3c79a6609c3e52e [id=07050bd496e84a97a86fdc4f0b7811ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.178216 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 55f3f09fc1f94c00a47b9087b98e221d (DEFAULT_TABLE table=default.Table_0d21bbe9f22f46dabc7701e182ea31b8 [id=106dd88de5eb49e89f0042beb7920736]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.178620 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55f3f09fc1f94c00a47b9087b98e221d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.183202 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 74b4c563d6bd4a44bb3bab58a3fcdc07 (DEFAULT_TABLE table=default.Table_f96549ae28214964b5530d879c8f04b6 [id=a9c009feee0e486794a8500121fe5487]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.183624 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b4c563d6bd4a44bb3bab58a3fcdc07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.185890 23750 tablet_service.cc:1467] Processing CreateTablet for tablet a4d151770ee44130adfd822518b9b017 (DEFAULT_TABLE table=default.Table_fb39fa7949f3412c83974132184b5d9f [id=8b62b02f324d4b6cbc7095de109955b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.186226 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4d151770ee44130adfd822518b9b017. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.186605 24179 ts_tablet_manager.cc:1909] T 223c92ae8f63402f8643e03d914e8e7e P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
W20240502 08:16:39.188436 23858 leader_election.cc:341] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 9b1eb762780c450bbaf380d17db564b6
I20240502 08:16:39.188573 23858 leader_election.cc:302] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:39.189256 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 9467974d67c64c7eb5e488ac51f1240d (DEFAULT_TABLE table=default.Table_c1886293aee349b2b9e7177c1cfd6d26 [id=0853ffc4d48549bc93e251a5dbc55901]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.189649 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9467974d67c64c7eb5e488ac51f1240d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.190330 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fcd8e870383b4c8f9b40db6fb2a7149b" } new_table_name: "default.Table_c72b83e8da844cb6a570c14f50293f34" modify_external_catalogs: true
I20240502 08:16:39.192286 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 090bec30f5664421b9bfe3f2810afd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.192915 24179 tablet_replica.cc:330] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:39.193181 24179 raft_consensus.cc:2229] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:39.193073 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c538b531b58147b89d720628ee7556dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:39.193300 23756 tablet_service.cc:1467] Processing CreateTablet for tablet b0e8982fe35f47cda376d9646b52e961 (DEFAULT_TABLE table=default.Table_b9be11b881924c8380270cc629f05765 [id=2459b7ca8e62437889f0c01eb20262ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.193804 24179 raft_consensus.cc:2258] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:39.193804 23768 tablet_service.cc:1467] Processing CreateTablet for tablet e5f5597402374527bf0e5e50178df713 (DEFAULT_TABLE table=default.Table_c44e58448899483fb6f36f8ee844e61d [id=ca51b24a648a4f2ea766c671f9058a1c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:39.193878 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c538b531b58147b89d720628ee7556dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.194103 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5f5597402374527bf0e5e50178df713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.194461 24179 ts_tablet_manager.cc:1875] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:39.197839 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_3256fb22eb624bc9bf0343e5e8477544"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:39.198416 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3256fb22eb624bc9bf0343e5e8477544 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.199726 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d67be3894f494223a45d8c5a2c3e6113" } modify_external_catalogs: true
I20240502 08:16:39.199898 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d67be3894f494223a45d8c5a2c3e6113" } modify_external_catalogs: true
I20240502 08:16:39.201182 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_dc7d5494bb5d436486375f565bc4353b" } new_table_name: "default.Table_810beecf9ff143d38f72f76e5548a1db" modify_external_catalogs: true
I20240502 08:16:39.202452 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_70034c04442f49f6a42cb66dccfd1cc1" } modify_external_catalogs: true
I20240502 08:16:39.202627 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_70034c04442f49f6a42cb66dccfd1cc1" } modify_external_catalogs: true
I20240502 08:16:39.204069 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e7935e6f9887457f9d33f01909abdcb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:39.204741 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7935e6f9887457f9d33f01909abdcb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.206192 24556 raft_consensus.cc:482] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:39.206426 24556 raft_consensus.cc:504] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.207504 24556 leader_election.cc:288] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:39.209182 23608 master_service.cc:946] ReplaceTablet: received request to replace tablet df9402cf8abc4b4d893d42f9a3a99016 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:39.210345 24060 raft_consensus.cc:2454] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:39.211016 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e8982fe35f47cda376d9646b52e961. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:39.223034 23858 leader_election.cc:341] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: c316e1aeab224634b760641a4ce188c8
I20240502 08:16:39.223278 23858 leader_election.cc:302] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:39.223610 24557 raft_consensus.cc:2726] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:39.223891 23858 leader_election.cc:341] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: da4be1d381a64b7baeb1c60cb17d924a
I20240502 08:16:39.224025 23858 leader_election.cc:302] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:39.224301 24557 raft_consensus.cc:2726] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:39.224933 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3d4b4584dd45b58ff0daa4f1c09a0a" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:39.225085 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 203ed05d0bdc4b1f99207423b7fc848b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.225211 23924 raft_consensus.cc:2454] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:39.225302 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 7fe583cbf4af493e82d3ec0508ea0f0b (DEFAULT_TABLE table=default.Table_0a4a6e3b1dd044f1a0e88c6c03f8c8b6 [id=8dc5018c97434885a14739188b199dc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.225874 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 92adebc0c69e4a52b975253656132495 (DEFAULT_TABLE table=default.Table_60a54110edb74227a1ae3a379d85e68d [id=07b4ec205e8a47d2a5d3b46ce99e8a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.225822 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 63e645bb6e2948e091e08433944a52f4 (DEFAULT_TABLE table=default.Table_5b51e3b192594b119d6719a685fe3513 [id=47b70f46b4d345618054c69267d33c95]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.226416 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 91f1ab141585441db46d3027833932b0 (DEFAULT_TABLE table=default.Table_b141c2b67ffe43ff97d32c9fff1d1fe0 [id=b2163faeeed146d9936a6e1a5cfbc46e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.226923 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 96b8f21c64e343d88d84e9acd0bc69b4 (DEFAULT_TABLE table=default.Table_9d372763eb03435f8b7074a802d86ffe [id=9ff8ea75a9bc4de194a70fe2088a94bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.227376 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 8910bc67c6234a538c8160e7c020653c (DEFAULT_TABLE table=default.Table_7c04460336ea48e79db571d95cdb1bcc [id=8ea9f7fff43846dfa8310549e8e57f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.227519 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9592a041a9543e8b41f043927807131" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:39.227855 23924 raft_consensus.cc:2379] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a297a6c7c2504ca68e87b24f6691ffb2 in current term 1: Already voted for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in this term.
I20240502 08:16:39.227803 23764 tablet_service.cc:1467] Processing CreateTablet for tablet a5a5a410b6314818b10927e35032f95d (DEFAULT_TABLE table=default.Table_6bdd34511d19462d91feac0f19089b68 [id=e5b9ab35a16943af94e5108dc0d29426]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.228240 23758 tablet_service.cc:1467] Processing CreateTablet for tablet a198c33a3e314b5b81a08244862b281a (DEFAULT_TABLE table=default.Table_22b226147b894b50a865f33b9f41491c [id=337cf81e0af74e8ca4333e4b1bfe4065]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.228670 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 0501575a84ec454780f48574bf3155fe (DEFAULT_TABLE table=default.Table_0dc68fd335024f2dbce6b1fcc698f1be [id=5cee878b3a31426c879a1bf95045bb09]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.225492 23760 tablet_service.cc:1467] Processing CreateTablet for tablet eb96ad2289754e428a1c109b2d59c53b (DEFAULT_TABLE table=default.Table_bc1e9d346aa846f8abca7504185f7cdb [id=8d27878c6c8f4234908a2c78cb8bbd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.229000 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08183523bf4a4d1c9020eb693aa07878" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:39.229208 23925 raft_consensus.cc:2454] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:39.225823 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe583cbf4af493e82d3ec0508ea0f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.231066 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08183523bf4a4d1c9020eb693aa07878" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:39.231293 23924 raft_consensus.cc:3037] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.232898 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c3800fb2784a1dba12d01c56c226e5" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:39.233863 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0501575a84ec454780f48574bf3155fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.235002 23861 leader_election.cc:302] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:39.237721 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a5a410b6314818b10927e35032f95d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.239176 24551 raft_consensus.cc:2726] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:39.241734 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8910bc67c6234a538c8160e7c020653c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.245828 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f1ab141585441db46d3027833932b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.249265 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92adebc0c69e4a52b975253656132495. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.253347 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a198c33a3e314b5b81a08244862b281a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.257684 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96b8f21c64e343d88d84e9acd0bc69b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.261716 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb96ad2289754e428a1c109b2d59c53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.265722 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e645bb6e2948e091e08433944a52f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.287546 24489 raft_consensus.cc:482] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:39.287729 24489 raft_consensus.cc:504] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.288431 24489 leader_election.cc:288] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:39.296061 23924 raft_consensus.cc:2454] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:39.296341 24483 raft_consensus.cc:2741] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240502 08:16:39.297430 24175 tablet_bootstrap.cc:492] T 5738cd77bbe147b49215ba9639d7f547 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:39.297595 24175 ts_tablet_manager.cc:1367] T 5738cd77bbe147b49215ba9639d7f547 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.204s user 0.006s sys 0.001s
I20240502 08:16:39.298395 24175 raft_consensus.cc:348] T 5738cd77bbe147b49215ba9639d7f547 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.298609 24175 raft_consensus.cc:374] T 5738cd77bbe147b49215ba9639d7f547 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.298704 24175 raft_consensus.cc:729] T 5738cd77bbe147b49215ba9639d7f547 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.298965 24175 consensus_queue.cc:260] T 5738cd77bbe147b49215ba9639d7f547 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.299518 24175 ts_tablet_manager.cc:1398] T 5738cd77bbe147b49215ba9639d7f547 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:16:39.299887 24175 tablet_bootstrap.cc:492] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:39.307427 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e66f5d1a72b347f7a7b39bddbf74b263" } modify_external_catalogs: true
I20240502 08:16:39.307734 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e66f5d1a72b347f7a7b39bddbf74b263" } modify_external_catalogs: true
W20240502 08:16:39.309197 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec10ff7251a5442a8ecc1881b34db74d consensus state has no leader
I20240502 08:16:39.309875 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5509aa65d43f4547853d26666042d04d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:39.310505 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5509aa65d43f4547853d26666042d04d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:39.312045 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 08183523bf4a4d1c9020eb693aa07878 consensus state has no leader
I20240502 08:16:39.312971 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e8b4ffe8cc7942d89f906ce2a0f524f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:39.313182 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bc9b016da1ce42b6a0509fbb664c8548" } modify_external_catalogs: true
I20240502 08:16:39.313356 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bc9b016da1ce42b6a0509fbb664c8548" } modify_external_catalogs: true
I20240502 08:16:39.316236 24489 raft_consensus.cc:482] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:39.316417 24489 raft_consensus.cc:504] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.317268 24489 leader_election.cc:288] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:39.319687 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25c089f0c1b14e419f250d03d4eaad12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.320315 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3d4b4584dd45b58ff0daa4f1c09a0a" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:39.320588 23786 raft_consensus.cc:2454] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:39.320780 24173 tablet_bootstrap.cc:654] T 43e870c3a0e340d1bad1c1fa0380a762 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.320806 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08183523bf4a4d1c9020eb693aa07878" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:39.321058 23788 raft_consensus.cc:2379] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:16:39.329651 24039 tablet_service.cc:1467] Processing CreateTablet for tablet ee022e8e86f8477a8fa1e58d52644b03 (DEFAULT_TABLE table=default.Table_d6f5ca82488a40a69121f1b779d8f874 [id=0422356f66a74ad29c2e13e1ad36b654]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:39.313491 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8b4ffe8cc7942d89f906ce2a0f524f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.313834 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e9f712275e34498c9f78493dc08b37f5" } new_table_name: "default.Table_c67035cbec4e42109cb3d466e0a6ad18" modify_external_catalogs: true
I20240502 08:16:39.314185 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 128c88af54d24ffd895e9de19cdb4c8a
I20240502 08:16:39.333484 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47459b5f5bb4ecb96f76240fefa1f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.314491 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 090bec30f5664421b9bfe3f2810afd14
I20240502 08:16:39.315044 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0287506f7f94418eb19163c272a30fb8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:39.335384 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0287506f7f94418eb19163c272a30fb8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.315608 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet a3043c5ed98c401b82c6e967ca3f3613 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:39.318534 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 223c92ae8f63402f8643e03d914e8e7e (table default.Table_00eb74cd24954c52b5484751f035f4c1 [id=b42e15a2350f4c9a8097ccb0cf1aaddc]) successfully deleted
I20240502 08:16:39.340772 24355 raft_consensus.cc:482] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:16:39.342123 23723 leader_election.cc:341] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: a7c3800fb2784a1dba12d01c56c226e5
I20240502 08:16:39.342654 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet 090bec30f5664421b9bfe3f2810afd14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.343200 24029 tablet_service.cc:1467] Processing CreateTablet for tablet df08f0d3af4c4c7e8abce656714b1b7f (DEFAULT_TABLE table=default.Table_59f82dbde55b406ba5271520c5bab104 [id=8e9acf4c0dbf4590b82aaee07243ed0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.343609 24511 raft_consensus.cc:482] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:39.343750 24511 raft_consensus.cc:504] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.345309 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb2d0a4898544f182ab8070d29a7940" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:39.346993 24025 tablet_service.cc:1467] Processing CreateTablet for tablet ba3aa2621fdb43fd8299f81af486ef1e (DEFAULT_TABLE table=default.Table_30df43df108e43d4b44b20c60eeb8287 [id=f8b12b3bb4904c18be60b567cb05b739]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.347623 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee022e8e86f8477a8fa1e58d52644b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.347878 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet 128c88af54d24ffd895e9de19cdb4c8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:39.350530 23993 leader_election.cc:341] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:39.351464 23723 leader_election.cc:341] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 203ed05d0bdc4b1f99207423b7fc848b
I20240502 08:16:39.356810 24511 leader_election.cc:288] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:39.357404 24174 tablet_bootstrap.cc:492] T f36ceb0222ef473a828ca795f393174d P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:39.357550 24174 ts_tablet_manager.cc:1367] T f36ceb0222ef473a828ca795f393174d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.215s user 0.004s sys 0.001s
I20240502 08:16:39.357736 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd7244b9f5741a8aa1c6835fe544676. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.358588 24174 raft_consensus.cc:348] T f36ceb0222ef473a828ca795f393174d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.358814 24174 raft_consensus.cc:374] T f36ceb0222ef473a828ca795f393174d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.358891 24174 raft_consensus.cc:729] T f36ceb0222ef473a828ca795f393174d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.359103 24174 consensus_queue.cc:260] T f36ceb0222ef473a828ca795f393174d P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.360944 24174 ts_tablet_manager.cc:1398] T f36ceb0222ef473a828ca795f393174d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:16:39.361213 24174 tablet_bootstrap.cc:492] T b63fe5b6d4db4750800ecd429fa97ed1 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:39.362726 24174 tablet_bootstrap.cc:654] T b63fe5b6d4db4750800ecd429fa97ed1 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.366793 24180 ts_tablet_manager.cc:1888] T 1df565c1c6564efd88fa557c50522d7e P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:39.366957 24180 log.cc:1196] T 1df565c1c6564efd88fa557c50522d7e P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/1df565c1c6564efd88fa557c50522d7e
I20240502 08:16:39.367383 24180 ts_tablet_manager.cc:1909] T 1df565c1c6564efd88fa557c50522d7e P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:39.368249 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 274) took 1155 ms. Trace:
I20240502 08:16:39.368376 24180 rpcz_store.cc:276] 0502 08:16:38.212760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.216839 (+ 4079us) service_pool.cc:225] Handling call
0502 08:16:39.148787 (+931948us) ts_tablet_manager.cc:1120] Deleting tablet 1df565c1c6564efd88fa557c50522d7e
0502 08:16:39.148798 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:39.352817 (+204019us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:39.366581 (+ 13764us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:39.368206 (+ 1625us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":931736}
I20240502 08:16:39.368952 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 1df565c1c6564efd88fa557c50522d7e (table default.Table_2107771cb030402280a54f03d8b2bce4 [id=0a35939b821f468c9c7c86425fd688a5]) successfully deleted
W20240502 08:16:39.370198 23723 leader_election.cc:341] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 5738cd77bbe147b49215ba9639d7f547
I20240502 08:16:39.371618 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 275) took 1049 ms. Trace:
I20240502 08:16:39.371724 24180 rpcz_store.cc:276] 0502 08:16:38.321789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.321859 (+ 70us) service_pool.cc:225] Handling call
0502 08:16:39.371477 (+1049618us) ts_tablet_manager.cc:1120] Deleting tablet b1fac7df16fc41d7a3297055a7258270
0502 08:16:39.371487 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:39.371588 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1047977}
I20240502 08:16:39.372179 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df08f0d3af4c4c7e8abce656714b1b7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.373123 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 276) took 1030 ms. Trace:
I20240502 08:16:39.356638 24355 raft_consensus.cc:504] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.373241 24180 rpcz_store.cc:276] 0502 08:16:38.342842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.342890 (+ 48us) service_pool.cc:225] Handling call
0502 08:16:39.372944 (+1030054us) ts_tablet_manager.cc:1120] Deleting tablet dcdd765895254a39b86b692b5d812c83
0502 08:16:39.372955 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:39.373088 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1029858}
W20240502 08:16:39.373940 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet dcdd765895254a39b86b692b5d812c83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dcdd765895254a39b86b692b5d812c83 already in progress: creating tablet
W20240502 08:16:39.375056 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet da4be1d381a64b7baeb1c60cb17d924a because the tablet was not found. No further retry: Not found: Tablet not found: da4be1d381a64b7baeb1c60cb17d924a
W20240502 08:16:39.376371 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet c5d28098dcbc4761a34ee65769d6fd7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5d28098dcbc4761a34ee65769d6fd7c already in progress: creating tablet
I20240502 08:16:39.378260 24180 tablet_replica.cc:330] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:39.378569 24180 raft_consensus.cc:2229] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:39.378733 24180 raft_consensus.cc:2258] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:39.379309 24180 ts_tablet_manager.cc:1875] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:39.264436 24551 raft_consensus.cc:482] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:39.383100 24174 tablet_bootstrap.cc:492] T b63fe5b6d4db4750800ecd429fa97ed1 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:39.383252 24174 ts_tablet_manager.cc:1367] T b63fe5b6d4db4750800ecd429fa97ed1 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.003s
I20240502 08:16:39.383919 24174 raft_consensus.cc:348] T b63fe5b6d4db4750800ecd429fa97ed1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.384130 24174 raft_consensus.cc:374] T b63fe5b6d4db4750800ecd429fa97ed1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.384182 24174 raft_consensus.cc:729] T b63fe5b6d4db4750800ecd429fa97ed1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.384470 24174 consensus_queue.cc:260] T b63fe5b6d4db4750800ecd429fa97ed1 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.386291 24174 ts_tablet_manager.cc:1398] T b63fe5b6d4db4750800ecd429fa97ed1 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:16:39.386587 24174 tablet_bootstrap.cc:492] T b742c51f6204449a804889815604d5cb P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:39.387993 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51b26043ef44b30846498cd1497607c" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:39.388701 24174 tablet_bootstrap.cc:654] T b742c51f6204449a804889815604d5cb P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.390069 24022 tablet_service.cc:1467] Processing CreateTablet for tablet e0b0d4871dd24607854100a4ef762e25 (DEFAULT_TABLE table=default.Table_5dcb3ffd44ea42de93aad8dcced74ed4 [id=910c7cae36264e39bbfca0be919538e0]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:39.398017 23993 leader_election.cc:341] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:39.399068 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3aa2621fdb43fd8299f81af486ef1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.399904 24174 tablet_bootstrap.cc:492] T b742c51f6204449a804889815604d5cb P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:39.400048 24174 ts_tablet_manager.cc:1367] T b742c51f6204449a804889815604d5cb P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20240502 08:16:39.400727 24174 raft_consensus.cc:348] T b742c51f6204449a804889815604d5cb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.400925 24174 raft_consensus.cc:374] T b742c51f6204449a804889815604d5cb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.401003 24174 raft_consensus.cc:729] T b742c51f6204449a804889815604d5cb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.401285 24174 consensus_queue.cc:260] T b742c51f6204449a804889815604d5cb P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.405619 23723 leader_election.cc:302] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
W20240502 08:16:39.405827 23723 raft_consensus.cc:2670] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:16:39.406849 24489 consensus_queue.cc:1035] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:39.407088 23723 leader_election.cc:302] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:39.407508 24489 raft_consensus.cc:3037] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.407663 24355 leader_election.cc:288] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:39.410681 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 89f57e3a39d54b7ab372d4b145b7a696 (DEFAULT_TABLE table=default.Table_28b3620523074c23866e772b117bf0ee [id=b09ad60d787b4740a493cb73c22b94a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.410938 24489 raft_consensus.cc:2726] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:39.411675 24174 ts_tablet_manager.cc:1398] T b742c51f6204449a804889815604d5cb P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240502 08:16:39.411983 24174 tablet_bootstrap.cc:492] T 6bb2d0a4898544f182ab8070d29a7940 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:39.412408 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18d284f71937459db0c0676ce4568175. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.413336 24034 tablet_service.cc:1467] Processing CreateTablet for tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 (DEFAULT_TABLE table=default.Table_e9f712275e34498c9f78493dc08b37f5 [id=b2cb0ed2926d4be7ade0790203b57e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.416301 24180 ts_tablet_manager.cc:1888] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:39.416455 24180 log.cc:1196] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/ba3d4b4584dd45b58ff0daa4f1c09a0a
I20240502 08:16:39.416802 24180 ts_tablet_manager.cc:1909] T ba3d4b4584dd45b58ff0daa4f1c09a0a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:39.417459 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51b26043ef44b30846498cd1497607c" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:39.417752 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 090bec30f5664421b9bfe3f2810afd14 (DEFAULT_TABLE table=default.Table_70034c04442f49f6a42cb66dccfd1cc1 [id=74be9e9508024ebeb4623c488e21faf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.418002 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 18d284f71937459db0c0676ce4568175 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.419059 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet ba3d4b4584dd45b58ff0daa4f1c09a0a (table default.Table_4e40fa1cf0ab4dedb2c6aadb2418901b [id=fd781805eb4c45f09fe6460144472816]) successfully deleted
I20240502 08:16:39.419276 24174 tablet_bootstrap.cc:654] T 6bb2d0a4898544f182ab8070d29a7940 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.419652 24180 tablet_replica.cc:330] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:39.419829 24180 raft_consensus.cc:2229] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:39.419984 24180 raft_consensus.cc:2258] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:39.420477 24180 ts_tablet_manager.cc:1875] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:39.421836 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb2d0a4898544f182ab8070d29a7940" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:39.422783 24551 raft_consensus.cc:504] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.426559 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f57e3a39d54b7ab372d4b145b7a696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.426662 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 5453cc0b93ac4cbb90a0588bb2100d60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.426851 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb2d0a4898544f182ab8070d29a7940 with delete_type TABLET_DATA_DELETED (replaced by tablet 1cd8338500f54604abcd6d42702c2eca) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.427305 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 4c395a29c7a1485eab39ca107dd91fca (DEFAULT_TABLE table=default.Table_3c57dba4df6d4a139583cb8ccb6e6ee9 [id=272acbb9a6ca4f2dbcbfea6aacbe9cfc]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:39.427644 23996 leader_election.cc:341] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:39.427779 23996 leader_election.cc:302] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:39.427305 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6a19bb8705c45c0bfc537d3a6bf8ae2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:39.428877 23996 leader_election.cc:341] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:39.429049 23996 leader_election.cc:302] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:39.433908 24180 ts_tablet_manager.cc:1888] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:39.434078 24180 log.cc:1196] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/5738cd77bbe147b49215ba9639d7f547
I20240502 08:16:39.434489 24180 ts_tablet_manager.cc:1909] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:39.434798 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c395a29c7a1485eab39ca107dd91fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.435307 24174 tablet_bootstrap.cc:492] T 6bb2d0a4898544f182ab8070d29a7940 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:39.435281 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 5738cd77bbe147b49215ba9639d7f547 (table default.Table_c84ff62b97d44dd2ad4a1d9927383457 [id=1bfba58d949847308b00af3275167658]) successfully deleted
I20240502 08:16:39.435484 24174 ts_tablet_manager.cc:1367] T 6bb2d0a4898544f182ab8070d29a7940 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.001s
I20240502 08:16:39.435593 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:39.436158 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 1e5de793e74247dd9e73b7b1d58027d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e5de793e74247dd9e73b7b1d58027d9 already in progress: creating tablet
I20240502 08:16:39.436386 24174 raft_consensus.cc:348] T 6bb2d0a4898544f182ab8070d29a7940 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.436617 24174 raft_consensus.cc:374] T 6bb2d0a4898544f182ab8070d29a7940 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.436694 24174 raft_consensus.cc:729] T 6bb2d0a4898544f182ab8070d29a7940 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
W20240502 08:16:39.436733 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 848a0865eb2b4828a2594a17d96869d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 848a0865eb2b4828a2594a17d96869d0 already in progress: creating tablet
I20240502 08:16:39.436921 24174 consensus_queue.cc:260] T 6bb2d0a4898544f182ab8070d29a7940 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.437395 24174 ts_tablet_manager.cc:1398] T 6bb2d0a4898544f182ab8070d29a7940 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:39.437705 24174 tablet_bootstrap.cc:492] T d5bab1b7c48248a7973c27d0a865b34b P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:39.439085 24355 raft_consensus.cc:2726] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:39.439327 24174 tablet_bootstrap.cc:654] T d5bab1b7c48248a7973c27d0a865b34b P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.439342 24021 tablet_service.cc:1467] Processing CreateTablet for tablet 55f3f09fc1f94c00a47b9087b98e221d (DEFAULT_TABLE table=default.Table_0d21bbe9f22f46dabc7701e182ea31b8 [id=106dd88de5eb49e89f0042beb7920736]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.441218 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 63afd6e4fee24edeaba1209105f1c3e6 (DEFAULT_TABLE table=default.Table_3e9140f5996646cbb7cad1837c94d43e [id=84e36915119b4125a1b0d64b2d1647fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.442072 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 156c7356ece342aea0d8db667ebad89c (DEFAULT_TABLE table=default.Table_39406aba7a1b46719724dece66b69fdd [id=15b52f0688c24df2b6f3fda3732ceeb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.442682 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b0d4871dd24607854100a4ef762e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.443738 24571 raft_consensus.cc:2726] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:39.444734 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 0882b7509a74475780eb700818aab398 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0882b7509a74475780eb700818aab398 already in progress: creating tablet
I20240502 08:16:39.445027 24023 tablet_service.cc:1467] Processing CreateTablet for tablet ec10ff7251a5442a8ecc1881b34db74d (DEFAULT_TABLE table=default.Table_dc7d5494bb5d436486375f565bc4353b [id=1d7a43116fe74c35b505eccd165f99cd]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:39.446223 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 71aa916247c449fea718b35869a6df20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71aa916247c449fea718b35869a6df20 already in progress: creating tablet
W20240502 08:16:39.446926 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet b11f504b93574e42a9bf30fde6d08207 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b11f504b93574e42a9bf30fde6d08207 already in progress: creating tablet
W20240502 08:16:39.447799 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 7c374f42f353497fa1d745164e7fbf61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c374f42f353497fa1d745164e7fbf61 already in progress: creating tablet
W20240502 08:16:39.448431 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 5a1e208733844829b13222e00a432439 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a1e208733844829b13222e00a432439 already in progress: creating tablet
I20240502 08:16:39.449040 24175 tablet_bootstrap.cc:654] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:39.449138 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet b6417aafbdcf416ea7bff56bce5e6ddb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6417aafbdcf416ea7bff56bce5e6ddb already in progress: creating tablet
W20240502 08:16:39.449855 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet dd3078cc1f6247609f8bd017ed076b45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd3078cc1f6247609f8bd017ed076b45 already in progress: creating tablet
I20240502 08:16:39.450438 24180 tablet_replica.cc:330] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:39.450771 24180 raft_consensus.cc:2229] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:39.450974 24180 raft_consensus.cc:2258] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:39.451670 24180 ts_tablet_manager.cc:1875] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:39.452081 23723 leader_election.cc:341] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:39.452260 23723 leader_election.cc:302] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:39.452525 24489 raft_consensus.cc:2726] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:39.454342 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d7af10ea88e34ce9b9e4565cb73c02de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:39.454658 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 090bec30f5664421b9bfe3f2810afd14. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:39.455077 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7af10ea88e34ce9b9e4565cb73c02de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.456146 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55f3f09fc1f94c00a47b9087b98e221d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.458786 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 7a97427e819b4826b1ed89e6d021c8d4
I20240502 08:16:39.458875 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_70822394abb947128d1014970c8ef471"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:39.459403 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70822394abb947128d1014970c8ef471 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.460458 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec10ff7251a5442a8ecc1881b34db74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.460810 24551 leader_election.cc:288] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:39.461760 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet df9402cf8abc4b4d893d42f9a3a99016
I20240502 08:16:39.462131 23608 catalog_manager.cc:6554] ReplaceTablet: tablet df9402cf8abc4b4d893d42f9a3a99016 deleted and replaced by tablet fddac683eda14ec1a0140926380f2c07
I20240502 08:16:39.463783 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a6b10e6dd2db4b5eb3e3d639cac19f54" } new_table_name: "default.Table_897d3dc97c8241d58a1a9a5329a1f892" modify_external_catalogs: true
I20240502 08:16:39.465092 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_37ab1c2d8931468482c9d99dd6536fe5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:39.465682 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37ab1c2d8931468482c9d99dd6536fe5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.465855 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a97427e819b4826b1ed89e6d021c8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.466122 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet f36ceb0222ef473a828ca795f393174d with delete_type TABLET_DATA_DELETED (replaced by tablet df9402cf8abc4b4d893d42f9a3a99016) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.466403 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet df9402cf8abc4b4d893d42f9a3a99016 with delete_type TABLET_DATA_DELETED (replaced by tablet fddac683eda14ec1a0140926380f2c07) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.476123 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 74b4c563d6bd4a44bb3bab58a3fcdc07 (DEFAULT_TABLE table=default.Table_f96549ae28214964b5530d879c8f04b6 [id=a9c009feee0e486794a8500121fe5487]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.477543 24174 tablet_bootstrap.cc:492] T d5bab1b7c48248a7973c27d0a865b34b P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:39.477818 24174 ts_tablet_manager.cc:1367] T d5bab1b7c48248a7973c27d0a865b34b P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.013s
I20240502 08:16:39.478675 24174 raft_consensus.cc:348] T d5bab1b7c48248a7973c27d0a865b34b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.479012 24174 raft_consensus.cc:374] T d5bab1b7c48248a7973c27d0a865b34b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.479264 24174 raft_consensus.cc:729] T d5bab1b7c48248a7973c27d0a865b34b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.479751 24174 consensus_queue.cc:260] T d5bab1b7c48248a7973c27d0a865b34b P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.481065 24174 ts_tablet_manager.cc:1398] T d5bab1b7c48248a7973c27d0a865b34b P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:16:39.481580 24174 tablet_bootstrap.cc:492] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:39.483650 24174 tablet_bootstrap.cc:654] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.486960 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 592eeecbf5514215b6464bd030081e4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.487522 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b4c563d6bd4a44bb3bab58a3fcdc07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.487725 24179 ts_tablet_manager.cc:1888] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240502 08:16:39.488698 23861 leader_election.cc:341] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: b63fe5b6d4db4750800ecd429fa97ed1
I20240502 08:16:39.491395 24179 log.cc:1196] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/405ea360814e4593a289d753821277ff
I20240502 08:16:39.492149 24180 ts_tablet_manager.cc:1888] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:39.492322 24180 log.cc:1196] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/e9592a041a9543e8b41f043927807131
I20240502 08:16:39.492617 24179 ts_tablet_manager.cc:1909] T 405ea360814e4593a289d753821277ff P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:39.492699 24180 ts_tablet_manager.cc:1909] T e9592a041a9543e8b41f043927807131 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:39.493140 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 256) took 1277 ms. Trace:
I20240502 08:16:39.497397 24179 rpcz_store.cc:276] 0502 08:16:38.215758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.215806 (+ 48us) service_pool.cc:225] Handling call
0502 08:16:39.187557 (+971751us) ts_tablet_manager.cc:1120] Deleting tablet 405ea360814e4593a289d753821277ff
0502 08:16:39.187566 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:39.246251 (+ 58685us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:39.467313 (+221062us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:39.493085 (+ 25772us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":971482}
I20240502 08:16:39.498054 24179 tablet_replica.cc:330] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:39.499083 24179 raft_consensus.cc:2229] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:39.500360 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 18d284f71937459db0c0676ce4568175 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.503948 23723 leader_election.cc:302] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:39.504379 24489 raft_consensus.cc:2781] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:39.504513 24489 raft_consensus.cc:686] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:39.504812 24489 consensus_queue.cc:237] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.509369 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.514348 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet e9592a041a9543e8b41f043927807131 (table default.Table_6c584db923e147d380281ecb1acd11a1 [id=467a7f6db73641ca85a1898b251938fb]) successfully deleted
W20240502 08:16:39.516294 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 already in progress: creating tablet
W20240502 08:16:39.517673 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 5453cc0b93ac4cbb90a0588bb2100d60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5453cc0b93ac4cbb90a0588bb2100d60 already in progress: creating tablet
W20240502 08:16:39.518617 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 18d284f71937459db0c0676ce4568175 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18d284f71937459db0c0676ce4568175 already in progress: creating tablet
I20240502 08:16:39.518924 24180 tablet_replica.cc:330] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:39.519141 24180 raft_consensus.cc:2229] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:39.519337 24180 raft_consensus.cc:2258] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:39.525342 24179 raft_consensus.cc:2258] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:39.526377 24173 tablet_bootstrap.cc:492] T 43e870c3a0e340d1bad1c1fa0380a762 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:39.526558 24180 ts_tablet_manager.cc:1875] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:39.526906 24179 ts_tablet_manager.cc:1875] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:39.533970 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 5453cc0b93ac4cbb90a0588bb2100d60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.539929 24180 ts_tablet_manager.cc:1888] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:39.540082 24180 log.cc:1196] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/a7c3800fb2784a1dba12d01c56c226e5
I20240502 08:16:39.540551 24180 ts_tablet_manager.cc:1909] T a7c3800fb2784a1dba12d01c56c226e5 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:39.541621 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet a7c3800fb2784a1dba12d01c56c226e5 (table default.Table_295f41167e6e481ea2443d5f8d02a998 [id=73559755c4594bcfa591d054964154ae]) successfully deleted
I20240502 08:16:39.545984 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 4c395a29c7a1485eab39ca107dd91fca (DEFAULT_TABLE table=default.Table_3c57dba4df6d4a139583cb8ccb6e6ee9 [id=272acbb9a6ca4f2dbcbfea6aacbe9cfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.546536 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c395a29c7a1485eab39ca107dd91fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.549417 24180 tablet_replica.cc:330] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:39.549680 24180 raft_consensus.cc:2229] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:39.549863 24180 raft_consensus.cc:2258] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:39.550659 24180 ts_tablet_manager.cc:1875] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:39.552928 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07344b78894c42809391d0fe2a9cd8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.554154 24509 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f87df6a45ad94428a9aa6b2796a0fd25. Latest consensus state: current_term: 1 leader_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } } }
I20240502 08:16:39.554455 24509 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:39.554462 24029 tablet_service.cc:1467] Processing CreateTablet for tablet a4d151770ee44130adfd822518b9b017 (DEFAULT_TABLE table=default.Table_fb39fa7949f3412c83974132184b5d9f [id=8b62b02f324d4b6cbc7095de109955b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.555531 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 203ed05d0bdc4b1f99207423b7fc848b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.556041 24564 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f87df6a45ad94428a9aa6b2796a0fd25)
I20240502 08:16:39.556200 24564 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:39.556592 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 63afd6e4fee24edeaba1209105f1c3e6 (DEFAULT_TABLE table=default.Table_3e9140f5996646cbb7cad1837c94d43e [id=84e36915119b4125a1b0d64b2d1647fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.557142 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63afd6e4fee24edeaba1209105f1c3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.557178 24564 leader_election.cc:288] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84f3c07a31414c4f8fb61c44dc9f16e1 (127.22.225.125:36189), f87df6a45ad94428a9aa6b2796a0fd25 (127.22.225.124:41745)
I20240502 08:16:39.558040 23620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" candidate_term: 2 candidate_status { last_received { term: 1 index: 392 } } ignore_live_leader: false dest_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" is_pre_election: true
I20240502 08:16:39.558498 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 156c7356ece342aea0d8db667ebad89c (DEFAULT_TABLE table=default.Table_39406aba7a1b46719724dece66b69fdd [id=15b52f0688c24df2b6f3fda3732ceeb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.558835 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156c7356ece342aea0d8db667ebad89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.560549 23548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" candidate_term: 2 candidate_status { last_received { term: 1 index: 392 } } ignore_live_leader: false dest_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" is_pre_election: true
W20240502 08:16:39.561054 24409 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.225.65:59489 (request call id 15) took 791 ms (client timeout 1000 ms). Trace:
I20240502 08:16:39.561221 24371 leader_election.cc:302] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcc8a323e6b9429eaaf4e0c1698ad077; no voters: 84f3c07a31414c4f8fb61c44dc9f16e1, f87df6a45ad94428a9aa6b2796a0fd25
I20240502 08:16:39.561686 24564 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:16:39.561174 24409 rpcz_store.cc:269] 0502 08:16:38.769675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.769743 (+ 68us) service_pool.cc:225] Handling call
0502 08:16:39.560536 (+790793us) spinlock_profiling.cc:243] Waited 790 ms on lock 0x616000050554. stack: 00007fedae942756 00007fedae942510 00007fedad97548d 0000000000955804 00000000009557cd 00007fedbd48115d 00007fedbd481253 00007fedba1b3d41 00007fedc1d9f34d 00007fedc1f26668 00007fedb8d11786 00007fedb8d114b5 00007fedb564a04e 00007fedb5648f2e 00007fedb564f405 00007fedb564fd2f
0502 08:16:39.560980 (+ 444us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1816699136}
I20240502 08:16:39.562312 23892 tablet_service.cc:1467] Processing CreateTablet for tablet a4d151770ee44130adfd822518b9b017 (DEFAULT_TABLE table=default.Table_fb39fa7949f3412c83974132184b5d9f [id=8b62b02f324d4b6cbc7095de109955b5]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:39.566399 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 consensus state has no leader
I20240502 08:16:39.566772 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6f4d25ef5b90455ca233af7b70b9073a" } modify_external_catalogs: true
I20240502 08:16:39.566951 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6f4d25ef5b90455ca233af7b70b9073a" } modify_external_catalogs: true
I20240502 08:16:39.568657 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_63545ead8a7a4aae8b2daa7af0fe5113"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:39.569203 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63545ead8a7a4aae8b2daa7af0fe5113 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.569831 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cb7682ca6f0f4d93922aa35e427c9cfa" } new_table_name: "default.Table_1600e0e09856405ea0d90ac82aa47fda" modify_external_catalogs: true
I20240502 08:16:39.570436 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet a3043c5ed98c401b82c6e967ca3f3613
I20240502 08:16:39.570832 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 5bcc7169796e4ee0910f5e7084849012
I20240502 08:16:39.570873 23604 catalog_manager.cc:6554] ReplaceTablet: tablet a3043c5ed98c401b82c6e967ca3f3613 deleted and replaced by tablet 0281c8123eab4c48bddc377f5695c6e0
I20240502 08:16:39.562669 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 55f3f09fc1f94c00a47b9087b98e221d (DEFAULT_TABLE table=default.Table_0d21bbe9f22f46dabc7701e182ea31b8 [id=106dd88de5eb49e89f0042beb7920736]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.571152 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_725d048135a04e32bd016e60ac974715" } new_table_name: "default.Table_42d6187c77b347e58022b68ed044f3fe" modify_external_catalogs: true
I20240502 08:16:39.571488 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_01e045a6ce334744b9ab0c8d6d390054"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:39.571892 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01e045a6ce334744b9ab0c8d6d390054 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.572335 23610 master_service.cc:946] ReplaceTablet: received request to replace tablet aacdf506221044ffada0fec0c7d42d0b from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:39.575997 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet b63fe5b6d4db4750800ecd429fa97ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.576272 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet b742c51f6204449a804889815604d5cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.576465 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb2d0a4898544f182ab8070d29a7940 with delete_type TABLET_DATA_DELETED (replaced by tablet 1cd8338500f54604abcd6d42702c2eca) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.576161 23904 tablet_service.cc:1467] Processing CreateTablet for tablet b0e8982fe35f47cda376d9646b52e961 (DEFAULT_TABLE table=default.Table_b9be11b881924c8380270cc629f05765 [id=2459b7ca8e62437889f0c01eb20262ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.576426 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4d151770ee44130adfd822518b9b017. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.576673 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet a3043c5ed98c401b82c6e967ca3f3613 with delete_type TABLET_DATA_DELETED (replaced by tablet 0281c8123eab4c48bddc377f5695c6e0) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.576865 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bcc7169796e4ee0910f5e7084849012 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.576956 23893 tablet_service.cc:1467] Processing CreateTablet for tablet ec10ff7251a5442a8ecc1881b34db74d (DEFAULT_TABLE table=default.Table_dc7d5494bb5d436486375f565bc4353b [id=1d7a43116fe74c35b505eccd165f99cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.577126 24180 ts_tablet_manager.cc:1888] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:39.578173 24180 log.cc:1196] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/203ed05d0bdc4b1f99207423b7fc848b
I20240502 08:16:39.578542 24180 ts_tablet_manager.cc:1909] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:39.579473 24180 tablet_replica.cc:330] T f36ceb0222ef473a828ca795f393174d P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:39.579701 24180 raft_consensus.cc:2229] T f36ceb0222ef473a828ca795f393174d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:39.579900 24180 raft_consensus.cc:2258] T f36ceb0222ef473a828ca795f393174d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:39.579877 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 203ed05d0bdc4b1f99207423b7fc848b (table default.Table_68bf91a14ffa42cdbb10a1d7ca56cb08 [id=59d945a73daa4918ba17063eae0b9b93]) successfully deleted
W20240502 08:16:39.580169 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 090bec30f5664421b9bfe3f2810afd14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 090bec30f5664421b9bfe3f2810afd14 already in progress: creating tablet
W20240502 08:16:39.580471 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 128c88af54d24ffd895e9de19cdb4c8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 128c88af54d24ffd895e9de19cdb4c8a already in progress: creating tablet
I20240502 08:16:39.580577 23886 tablet_service.cc:1467] Processing CreateTablet for tablet e5f5597402374527bf0e5e50178df713 (DEFAULT_TABLE table=default.Table_c44e58448899483fb6f36f8ee844e61d [id=ca51b24a648a4f2ea766c671f9058a1c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:39.580670 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 7a97427e819b4826b1ed89e6d021c8d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a97427e819b4826b1ed89e6d021c8d4 already in progress: creating tablet
I20240502 08:16:39.580865 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 9467974d67c64c7eb5e488ac51f1240d (DEFAULT_TABLE table=default.Table_c1886293aee349b2b9e7177c1cfd6d26 [id=0853ffc4d48549bc93e251a5dbc55901]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.566956 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 1ce5ccfd29724dc49afd0892b99f3169 (table default.Table_bdbe807258a5454795eaff8c27bb17dd [id=3a9d7e8fe2c84cb4942fee9053a012b0]). Expected version 1 got 0
I20240502 08:16:39.583403 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55f3f09fc1f94c00a47b9087b98e221d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.583745 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9467974d67c64c7eb5e488ac51f1240d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.583925 24174 tablet_bootstrap.cc:492] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:39.584110 24174 ts_tablet_manager.cc:1367] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.103s user 0.003s sys 0.011s
I20240502 08:16:39.584363 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5f5597402374527bf0e5e50178df713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.585034 24174 raft_consensus.cc:348] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.585256 24174 raft_consensus.cc:374] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.585332 24174 raft_consensus.cc:729] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.585593 23897 tablet_service.cc:1467] Processing CreateTablet for tablet 63e645bb6e2948e091e08433944a52f4 (DEFAULT_TABLE table=default.Table_5b51e3b192594b119d6719a685fe3513 [id=47b70f46b4d345618054c69267d33c95]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.585618 24174 consensus_queue.cc:260] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.585880 23889 tablet_service.cc:1467] Processing CreateTablet for tablet a5a5a410b6314818b10927e35032f95d (DEFAULT_TABLE table=default.Table_6bdd34511d19462d91feac0f19089b68 [id=e5b9ab35a16943af94e5108dc0d29426]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.586109 24174 ts_tablet_manager.cc:1398] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:39.586120 23901 tablet_service.cc:1467] Processing CreateTablet for tablet eb96ad2289754e428a1c109b2d59c53b (DEFAULT_TABLE table=default.Table_bc1e9d346aa846f8abca7504185f7cdb [id=8d27878c6c8f4234908a2c78cb8bbd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.586395 24174 tablet_bootstrap.cc:492] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:39.586381 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 96b8f21c64e343d88d84e9acd0bc69b4 (DEFAULT_TABLE table=default.Table_9d372763eb03435f8b7074a802d86ffe [id=9ff8ea75a9bc4de194a70fe2088a94bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.586517 23903 tablet_service.cc:1514] Processing DeleteTablet for tablet 128c88af54d24ffd895e9de19cdb4c8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.586714 23903 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a97427e819b4826b1ed89e6d021c8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.586951 23895 tablet_service.cc:1514] Processing DeleteTablet for tablet 090bec30f5664421b9bfe3f2810afd14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.586915 23894 tablet_service.cc:1467] Processing CreateTablet for tablet a198c33a3e314b5b81a08244862b281a (DEFAULT_TABLE table=default.Table_22b226147b894b50a865f33b9f41491c [id=337cf81e0af74e8ca4333e4b1bfe4065]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.587146 23895 tablet_service.cc:1514] Processing DeleteTablet for tablet 5738cd77bbe147b49215ba9639d7f547 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.587270 23903 tablet_service.cc:1514] Processing DeleteTablet for tablet df9402cf8abc4b4d893d42f9a3a99016 with delete_type TABLET_DATA_DELETED (replaced by tablet fddac683eda14ec1a0140926380f2c07) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.587373 23895 tablet_service.cc:1514] Processing DeleteTablet for tablet a3043c5ed98c401b82c6e967ca3f3613 with delete_type TABLET_DATA_DELETED (replaced by tablet 0281c8123eab4c48bddc377f5695c6e0) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.587941 24174 tablet_bootstrap.cc:654] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.585602 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e8982fe35f47cda376d9646b52e961. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.590921 24180 ts_tablet_manager.cc:1875] T f36ceb0222ef473a828ca795f393174d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:39.591754 23596 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 751808dfc1964b539f6f836d786156aa (table default.Table_ad81523678c3450c880efd1ecfc5f95e [id=911d4d5a86e54a5696d37a2bd9f69733], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 751808dfc1964b539f6f836d786156aa consensus state has no leader
I20240502 08:16:39.592414 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 7fe583cbf4af493e82d3ec0508ea0f0b (DEFAULT_TABLE table=default.Table_0a4a6e3b1dd044f1a0e88c6c03f8c8b6 [id=8dc5018c97434885a14739188b199dc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.592859 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe583cbf4af493e82d3ec0508ea0f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.593737 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb96ad2289754e428a1c109b2d59c53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.594707 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e645bb6e2948e091e08433944a52f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.597064 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5738cd77bbe147b49215ba9639d7f547" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:39.597294 23924 raft_consensus.cc:2454] T 5738cd77bbe147b49215ba9639d7f547 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:39.597681 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203ed05d0bdc4b1f99207423b7fc848b" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:39.597931 23925 raft_consensus.cc:2454] T 203ed05d0bdc4b1f99207423b7fc848b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:39.598215 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a5a410b6314818b10927e35032f95d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.600817 24180 ts_tablet_manager.cc:1888] T f36ceb0222ef473a828ca795f393174d P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:39.600934 24180 log.cc:1196] T f36ceb0222ef473a828ca795f393174d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/f36ceb0222ef473a828ca795f393174d
I20240502 08:16:39.601263 24180 ts_tablet_manager.cc:1909] T f36ceb0222ef473a828ca795f393174d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:39.601940 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet f36ceb0222ef473a828ca795f393174d (table default.Table_4b967ef2b2fc4e2caa4219afcae841a6 [id=c32fc96889554c32bdf1d4550633e91f]) successfully deleted
I20240502 08:16:39.602335 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96b8f21c64e343d88d84e9acd0bc69b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.605736 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a198c33a3e314b5b81a08244862b281a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.605839 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 405ea360814e4593a289d753821277ff (table default.Table_3602f20179c64041a63ab185d9359de6 [id=6d4996e259e34d51b7ee13c8efa7e2bd]) successfully deleted
I20240502 08:16:39.606395 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b63fe5b6d4db4750800ecd429fa97ed1" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:39.606671 23788 raft_consensus.cc:2454] T b63fe5b6d4db4750800ecd429fa97ed1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:39.609707 23723 leader_election.cc:302] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
W20240502 08:16:39.609975 23723 raft_consensus.cc:2670] T 5738cd77bbe147b49215ba9639d7f547 P a297a6c7c2504ca68e87b24f6691ffb2: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240502 08:16:39.610569 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet df9402cf8abc4b4d893d42f9a3a99016 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df9402cf8abc4b4d893d42f9a3a99016 already in progress: creating tablet
I20240502 08:16:39.610734 23723 leader_election.cc:302] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:39.610996 24180 tablet_replica.cc:330] T b63fe5b6d4db4750800ecd429fa97ed1 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:39.611235 24180 raft_consensus.cc:2229] T b63fe5b6d4db4750800ecd429fa97ed1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:39.611413 24180 raft_consensus.cc:2258] T b63fe5b6d4db4750800ecd429fa97ed1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:39.612756 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 0501575a84ec454780f48574bf3155fe (DEFAULT_TABLE table=default.Table_0dc68fd335024f2dbce6b1fcc698f1be [id=5cee878b3a31426c879a1bf95045bb09]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.613155 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0501575a84ec454780f48574bf3155fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.614266 24040 tablet_service.cc:1467] Processing CreateTablet for tablet b0e8982fe35f47cda376d9646b52e961 (DEFAULT_TABLE table=default.Table_b9be11b881924c8380270cc629f05765 [id=2459b7ca8e62437889f0c01eb20262ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.615068 24180 ts_tablet_manager.cc:1875] T b63fe5b6d4db4750800ecd429fa97ed1 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:39.612337 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 91f1ab141585441db46d3027833932b0 (DEFAULT_TABLE table=default.Table_b141c2b67ffe43ff97d32c9fff1d1fe0 [id=b2163faeeed146d9936a6e1a5cfbc46e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.616376 24181 ts_tablet_manager.cc:1888] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:39.611922 23858 leader_election.cc:302] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:39.618429 24551 raft_consensus.cc:2781] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:39.618548 24551 raft_consensus.cc:482] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:39.618674 24551 raft_consensus.cc:3037] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.621388 24329 raft_consensus.cc:482] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:39.621585 24329 raft_consensus.cc:504] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.622246 24329 leader_election.cc:288] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:39.625620 24173 ts_tablet_manager.cc:1367] T 43e870c3a0e340d1bad1c1fa0380a762 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.558s user 0.024s sys 0.000s
I20240502 08:16:39.625818 24200 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } } }
I20240502 08:16:39.626070 24200 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:39.626410 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4d151770ee44130adfd822518b9b017. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:39.626492 23723 raft_consensus.cc:2670] T 203ed05d0bdc4b1f99207423b7fc848b P a297a6c7c2504ca68e87b24f6691ffb2: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:16:39.627354 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e8982fe35f47cda376d9646b52e961. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.623628 24551 raft_consensus.cc:504] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.628713 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8633864e248f416cb651da3b0c2aa936" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:39.629851 23861 leader_election.cc:341] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: ba3d4b4584dd45b58ff0daa4f1c09a0a
I20240502 08:16:39.630785 24551 leader_election.cc:288] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:39.623821 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f1ab141585441db46d3027833932b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.619953 23858 leader_election.cc:302] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:39.633513 24551 raft_consensus.cc:2781] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:39.633643 24551 raft_consensus.cc:482] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:39.633724 24551 raft_consensus.cc:3037] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.637881 24551 raft_consensus.cc:504] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.638839 24551 leader_election.cc:288] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:39.640357 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8633864e248f416cb651da3b0c2aa936" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:39.641633 23723 leader_election.cc:341] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: Tablet not RUNNING: STOPPED
I20240502 08:16:39.641841 24528 catalog_manager.cc:1259] Loaded cluster ID: d0e52a60b5c54ea9a5f0ea385bd7e9c9
I20240502 08:16:39.644038 24528 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077: loading cluster ID for follower catalog manager: success
I20240502 08:16:39.645673 24181 log.cc:1196] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/8633864e248f416cb651da3b0c2aa936
I20240502 08:16:39.646112 24181 ts_tablet_manager.cc:1909] T 8633864e248f416cb651da3b0c2aa936 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
W20240502 08:16:39.646404 23861 leader_election.cc:341] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: b63fe5b6d4db4750800ecd429fa97ed1
I20240502 08:16:39.649664 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b63fe5b6d4db4750800ecd429fa97ed1" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:39.650601 24024 tablet_service.cc:1467] Processing CreateTablet for tablet e5f5597402374527bf0e5e50178df713 (DEFAULT_TABLE table=default.Table_c44e58448899483fb6f36f8ee844e61d [id=ca51b24a648a4f2ea766c671f9058a1c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:39.654261 23723 leader_election.cc:341] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:39.654650 23858 leader_election.cc:341] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: ba3d4b4584dd45b58ff0daa4f1c09a0a
I20240502 08:16:39.656289 23858 leader_election.cc:302] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:39.648087 24180 ts_tablet_manager.cc:1888] T b63fe5b6d4db4750800ecd429fa97ed1 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:39.656823 24180 log.cc:1196] T b63fe5b6d4db4750800ecd429fa97ed1 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/b63fe5b6d4db4750800ecd429fa97ed1
I20240502 08:16:39.657281 24180 ts_tablet_manager.cc:1909] T b63fe5b6d4db4750800ecd429fa97ed1 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:39.657322 24551 raft_consensus.cc:2726] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:16:39.657866 23886 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bcc7169796e4ee0910f5e7084849012 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.657939 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet b63fe5b6d4db4750800ecd429fa97ed1 (table default.Table_07371eaf91cb4ab199af62382b640d64 [id=0026ee826a3e489087f873579507c110]) successfully deleted
I20240502 08:16:39.657126 23723 leader_election.cc:302] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:39.659080 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 8910bc67c6234a538c8160e7c020653c (DEFAULT_TABLE table=default.Table_7c04460336ea48e79db571d95cdb1bcc [id=8ea9f7fff43846dfa8310549e8e57f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.658473 24180 tablet_replica.cc:330] T b742c51f6204449a804889815604d5cb P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:39.659641 24180 raft_consensus.cc:2229] T b742c51f6204449a804889815604d5cb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:39.659775 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8910bc67c6234a538c8160e7c020653c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.659834 24180 raft_consensus.cc:2258] T b742c51f6204449a804889815604d5cb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:39.659845 23858 leader_election.cc:341] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: Tablet not RUNNING: STOPPED
I20240502 08:16:39.659950 23858 leader_election.cc:302] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:39.660408 24551 raft_consensus.cc:2726] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:16:39.661116 24180 ts_tablet_manager.cc:1875] T b742c51f6204449a804889815604d5cb P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:39.635799 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 7fe583cbf4af493e82d3ec0508ea0f0b (DEFAULT_TABLE table=default.Table_0a4a6e3b1dd044f1a0e88c6c03f8c8b6 [id=8dc5018c97434885a14739188b199dc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.663709 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 276) took 1557 ms. Trace:
I20240502 08:16:39.663862 24181 rpcz_store.cc:276] 0502 08:16:38.089267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.089416 (+ 149us) service_pool.cc:225] Handling call
0502 08:16:38.892389 (+802973us) ts_tablet_manager.cc:1120] Deleting tablet 8633864e248f416cb651da3b0c2aa936
0502 08:16:38.892397 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:39.184288 (+291891us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:39.616251 (+431963us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:39.646900 (+ 30649us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":802714}
I20240502 08:16:39.664651 24173 raft_consensus.cc:348] T 43e870c3a0e340d1bad1c1fa0380a762 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.666541 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 8633864e248f416cb651da3b0c2aa936 (table default.Table_8ba240c391da43978f93a27dce48e557 [id=1b5536950db244d1978b154da0be33e0]) successfully deleted
I20240502 08:16:39.674019 24489 raft_consensus.cc:2726] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:39.677058 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec10ff7251a5442a8ecc1881b34db74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.680080 24180 ts_tablet_manager.cc:1888] T b742c51f6204449a804889815604d5cb P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:39.680258 24180 log.cc:1196] T b742c51f6204449a804889815604d5cb P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/b742c51f6204449a804889815604d5cb
I20240502 08:16:39.681089 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 92adebc0c69e4a52b975253656132495 (DEFAULT_TABLE table=default.Table_60a54110edb74227a1ae3a379d85e68d [id=07b4ec205e8a47d2a5d3b46ce99e8a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.681351 24180 ts_tablet_manager.cc:1909] T b742c51f6204449a804889815604d5cb P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:39.681793 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 74b4c563d6bd4a44bb3bab58a3fcdc07 (DEFAULT_TABLE table=default.Table_f96549ae28214964b5530d879c8f04b6 [id=a9c009feee0e486794a8500121fe5487]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.682597 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet b742c51f6204449a804889815604d5cb (table default.Table_9507475756c3401fa75fc34d4a4b7814 [id=f42bb435dc484af7990e8cf4671e1c85]) successfully deleted
I20240502 08:16:39.683154 24180 tablet_replica.cc:330] T 6bb2d0a4898544f182ab8070d29a7940 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:39.683385 24180 raft_consensus.cc:2229] T 6bb2d0a4898544f182ab8070d29a7940 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:39.683590 24180 raft_consensus.cc:2258] T 6bb2d0a4898544f182ab8070d29a7940 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:39.681581 24528 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077: acquiring CA information for follower catalog manager: success
I20240502 08:16:39.687809 24180 ts_tablet_manager.cc:1875] T 6bb2d0a4898544f182ab8070d29a7940 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:39.695163 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156c7356ece342aea0d8db667ebad89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.695624 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 92adebc0c69e4a52b975253656132495 (DEFAULT_TABLE table=default.Table_60a54110edb74227a1ae3a379d85e68d [id=07b4ec205e8a47d2a5d3b46ce99e8a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.696031 24181 tablet_replica.cc:330] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:39.696198 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 9467974d67c64c7eb5e488ac51f1240d (DEFAULT_TABLE table=default.Table_c1886293aee349b2b9e7177c1cfd6d26 [id=0853ffc4d48549bc93e251a5dbc55901]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.696280 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 91f1ab141585441db46d3027833932b0 (DEFAULT_TABLE table=default.Table_b141c2b67ffe43ff97d32c9fff1d1fe0 [id=b2163faeeed146d9936a6e1a5cfbc46e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.698033 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 96b8f21c64e343d88d84e9acd0bc69b4 (DEFAULT_TABLE table=default.Table_9d372763eb03435f8b7074a802d86ffe [id=9ff8ea75a9bc4de194a70fe2088a94bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.700151 24026 tablet_service.cc:1467] Processing CreateTablet for tablet a5a5a410b6314818b10927e35032f95d (DEFAULT_TABLE table=default.Table_6bdd34511d19462d91feac0f19089b68 [id=e5b9ab35a16943af94e5108dc0d29426]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.700542 24181 raft_consensus.cc:2229] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:39.701238 24181 raft_consensus.cc:2258] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:39.702016 24181 ts_tablet_manager.cc:1875] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:39.702334 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5f5597402374527bf0e5e50178df713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.700589 24173 raft_consensus.cc:374] T 43e870c3a0e340d1bad1c1fa0380a762 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.703433 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe583cbf4af493e82d3ec0508ea0f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.703873 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f1ab141585441db46d3027833932b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.704030 24173 raft_consensus.cc:729] T 43e870c3a0e340d1bad1c1fa0380a762 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
W20240502 08:16:39.704697 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 577b149f52e44d5d9a9f6b9d85a6864d consensus state has no leader
W20240502 08:16:39.705132 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f36aadfa1d264f23bcafa91744aed36d consensus state has no leader
I20240502 08:16:39.705216 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0d21bbe9f22f46dabc7701e182ea31b8" } new_table_name: "default.Table_9771b72641674c9497872784242b435d" modify_external_catalogs: true
I20240502 08:16:39.705650 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a320ac94a297479885cba64075ce2ae2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:39.706121 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a320ac94a297479885cba64075ce2ae2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.707134 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a5a410b6314818b10927e35032f95d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.707381 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5fe628d3da624233941f440e3743da88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:39.707554 23607 master_service.cc:946] ReplaceTablet: received request to replace tablet bcfacc302ff04178bfffc48111cd7a3f from {username='slave'} at 127.0.0.1:36668
W20240502 08:16:39.707959 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fe628d3da624233941f440e3743da88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.709015 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d6f5ca82488a40a69121f1b779d8f874" } modify_external_catalogs: true
I20240502 08:16:39.709184 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d6f5ca82488a40a69121f1b779d8f874" } modify_external_catalogs: true
I20240502 08:16:39.711138 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 654ce33d1e3d472eb6ae6a726eb803bb
I20240502 08:16:39.713624 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet 654ce33d1e3d472eb6ae6a726eb803bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.713840 24580 tablet.cc:1722] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:39.714370 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96b8f21c64e343d88d84e9acd0bc69b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.715029 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_dbe385cc3f6b48d39bee298ca9490248" } modify_external_catalogs: true
I20240502 08:16:39.715252 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_dbe385cc3f6b48d39bee298ca9490248" } modify_external_catalogs: true
W20240502 08:16:39.719246 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1e746013fde4de4862f213792e235ed consensus state has no leader
I20240502 08:16:39.719594 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet 25c089f0c1b14e419f250d03d4eaad12 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:39.720394 24180 ts_tablet_manager.cc:1888] T 6bb2d0a4898544f182ab8070d29a7940 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:39.720592 24180 log.cc:1196] T 6bb2d0a4898544f182ab8070d29a7940 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/6bb2d0a4898544f182ab8070d29a7940
I20240502 08:16:39.720978 24180 ts_tablet_manager.cc:1909] T 6bb2d0a4898544f182ab8070d29a7940 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:39.723260 24528 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240502 08:16:39.724550 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 6bb2d0a4898544f182ab8070d29a7940 (table default.Table_482a1b0b7a064e02805ac3b86fc62f99 [id=3c1d614616374739aeb928e797479df4]) successfully deleted
I20240502 08:16:39.727672 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 0501575a84ec454780f48574bf3155fe (DEFAULT_TABLE table=default.Table_0dc68fd335024f2dbce6b1fcc698f1be [id=5cee878b3a31426c879a1bf95045bb09]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.729389 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3256fb22eb624bc9bf0343e5e8477544" } modify_external_catalogs: true
I20240502 08:16:39.729620 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3256fb22eb624bc9bf0343e5e8477544" } modify_external_catalogs: true
W20240502 08:16:39.729672 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet a3043c5ed98c401b82c6e967ca3f3613 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3043c5ed98c401b82c6e967ca3f3613 already in progress: creating tablet
W20240502 08:16:39.730391 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 5bcc7169796e4ee0910f5e7084849012 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bcc7169796e4ee0910f5e7084849012 already in progress: creating tablet
I20240502 08:16:39.730777 24174 tablet_bootstrap.cc:492] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:39.730958 24174 ts_tablet_manager.cc:1367] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.145s user 0.012s sys 0.019s
W20240502 08:16:39.730993 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 654ce33d1e3d472eb6ae6a726eb803bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 654ce33d1e3d472eb6ae6a726eb803bb already in progress: creating tablet
I20240502 08:16:39.731889 24174 raft_consensus.cc:348] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.732105 24174 raft_consensus.cc:374] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.732208 24174 raft_consensus.cc:729] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.732429 24174 consensus_queue.cc:260] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.732976 24174 ts_tablet_manager.cc:1398] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:39.733273 24174 tablet_bootstrap.cc:492] T f51b26043ef44b30846498cd1497607c P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:39.735005 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92adebc0c69e4a52b975253656132495. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.735394 24174 tablet_bootstrap.cc:654] T f51b26043ef44b30846498cd1497607c P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.735355 24032 tablet_service.cc:1467] Processing CreateTablet for tablet a198c33a3e314b5b81a08244862b281a (DEFAULT_TABLE table=default.Table_22b226147b894b50a865f33b9f41491c [id=337cf81e0af74e8ca4333e4b1bfe4065]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.739125 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0501575a84ec454780f48574bf3155fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.740818 24173 consensus_queue.cc:260] T 43e870c3a0e340d1bad1c1fa0380a762 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.749637 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet aacdf506221044ffada0fec0c7d42d0b
I20240502 08:16:39.750254 23610 catalog_manager.cc:6554] ReplaceTablet: tablet aacdf506221044ffada0fec0c7d42d0b deleted and replaced by tablet 90d4e2f23c9649a4b1dfb6a179ca7be5
I20240502 08:16:39.750777 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9bb2be9b511a49eca4228b582c5ac739"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:39.751327 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bb2be9b511a49eca4228b582c5ac739 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.752991 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet aacdf506221044ffada0fec0c7d42d0b with delete_type TABLET_DATA_DELETED (replaced by tablet 90d4e2f23c9649a4b1dfb6a179ca7be5) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.754209 24173 ts_tablet_manager.cc:1398] T 43e870c3a0e340d1bad1c1fa0380a762 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.128s user 0.002s sys 0.001s
I20240502 08:16:39.755745 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63afd6e4fee24edeaba1209105f1c3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.756837 24174 tablet_bootstrap.cc:492] T f51b26043ef44b30846498cd1497607c P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:39.757138 24174 ts_tablet_manager.cc:1367] T f51b26043ef44b30846498cd1497607c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.002s
W20240502 08:16:39.760471 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 55f3f09fc1f94c00a47b9087b98e221d consensus state has no leader
I20240502 08:16:39.760881 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet bcfacc302ff04178bfffc48111cd7a3f
I20240502 08:16:39.761325 23607 catalog_manager.cc:6554] ReplaceTablet: tablet bcfacc302ff04178bfffc48111cd7a3f deleted and replaced by tablet b0cf1a6de3d649b984c50f45393ec0f1
I20240502 08:16:39.762663 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet bcfacc302ff04178bfffc48111cd7a3f with delete_type TABLET_DATA_DELETED (replaced by tablet b0cf1a6de3d649b984c50f45393ec0f1) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.763336 24174 raft_consensus.cc:348] T f51b26043ef44b30846498cd1497607c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.764428 24174 raft_consensus.cc:374] T f51b26043ef44b30846498cd1497607c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.764540 24174 raft_consensus.cc:729] T f51b26043ef44b30846498cd1497607c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.765126 24174 consensus_queue.cc:260] T f51b26043ef44b30846498cd1497607c P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.765791 24174 ts_tablet_manager.cc:1398] T f51b26043ef44b30846498cd1497607c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 08:16:39.766619 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet ee022e8e86f8477a8fa1e58d52644b03
I20240502 08:16:39.767529 23602 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 8708e201c2c54521922de7351bd64c6b
I20240502 08:16:39.768767 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet ee022e8e86f8477a8fa1e58d52644b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.770571 24174 tablet_bootstrap.cc:492] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:39.777307 24173 tablet_bootstrap.cc:492] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:39.778542 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a198c33a3e314b5b81a08244862b281a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.778627 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a44933d855bd48aa8865c11eaaa5cbe4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:39.778687 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_32836076d604482a9e0963986e34af91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:39.779163 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32836076d604482a9e0963986e34af91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.779372 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9a479fbf2ac24ac1838fc0599feb088a" } new_table_name: "default.Table_f8db538a85994d0dbee1b1cbbabf5e16" modify_external_catalogs: true
I20240502 08:16:39.780208 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d035b24dfb644b7aa77b160ee28d36e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:39.780597 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d035b24dfb644b7aa77b160ee28d36e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.781735 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ad81523678c3450c880efd1ecfc5f95e" } modify_external_catalogs: true
I20240502 08:16:39.781925 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ad81523678c3450c880efd1ecfc5f95e" } modify_external_catalogs: true
I20240502 08:16:39.785233 24021 tablet_service.cc:1467] Processing CreateTablet for tablet 8910bc67c6234a538c8160e7c020653c (DEFAULT_TABLE table=default.Table_7c04460336ea48e79db571d95cdb1bcc [id=8ea9f7fff43846dfa8310549e8e57f2f]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:39.779160 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a44933d855bd48aa8865c11eaaa5cbe4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.789271 23608 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 43e870c3a0e340d1bad1c1fa0380a762 (table default.Table_299ae144ccc6402abe33d07ef693e5b4 [id=7e0c798bc77a4770a4bbe8658f181ffa]). Expected version 3 got 0
I20240502 08:16:39.789291 23601 catalog_manager.cc:5484] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:39.790498 23601 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 08183523bf4a4d1c9020eb693aa07878 (table default.Table_c72b83e8da844cb6a570c14f50293f34 [id=0a2f8dbb62a64b0a96931ab5f8c17540]). Expected version 2 got 0
I20240502 08:16:39.793644 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 63e645bb6e2948e091e08433944a52f4 (DEFAULT_TABLE table=default.Table_5b51e3b192594b119d6719a685fe3513 [id=47b70f46b4d345618054c69267d33c95]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:39.793838 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet aacdf506221044ffada0fec0c7d42d0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aacdf506221044ffada0fec0c7d42d0b already in progress: creating tablet
W20240502 08:16:39.795727 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5c377e037894728ac084b6500ad20de consensus state has no leader
I20240502 08:16:39.792361 23763 tablet_service.cc:1467] Processing CreateTablet for tablet ac50774b2dd241de9cae8096578d183e (DEFAULT_TABLE table=default.Table_f0695a0f3e5c48c7aa4f998a18be36e1 [id=30defee3de2445f7bc5017fb23ab2ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.797463 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac50774b2dd241de9cae8096578d183e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:39.799108 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet bcfacc302ff04178bfffc48111cd7a3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcfacc302ff04178bfffc48111cd7a3f already in progress: creating tablet
I20240502 08:16:39.792157 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 41f2e98ef2884b05994d02114e992e80 (DEFAULT_TABLE table=default.Table_da999521d34e478fab2c024a89fa0025 [id=b6dee89ecfb443d99267917d4df78e4d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:39.802069 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet ee022e8e86f8477a8fa1e58d52644b03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee022e8e86f8477a8fa1e58d52644b03 already in progress: creating tablet
I20240502 08:16:39.802173 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f2e98ef2884b05994d02114e992e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.806319 24173 tablet_bootstrap.cc:654] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.811344 24585 tablet.cc:1722] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:39.814399 24179 ts_tablet_manager.cc:1888] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:39.776340 24174 tablet_bootstrap.cc:654] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.818575 23606 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 43e870c3a0e340d1bad1c1fa0380a762 (table default.Table_299ae144ccc6402abe33d07ef693e5b4 [id=7e0c798bc77a4770a4bbe8658f181ffa]). Expected version 3 got 0
I20240502 08:16:39.823155 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b4c563d6bd4a44bb3bab58a3fcdc07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.823773 24175 tablet_bootstrap.cc:492] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:39.824515 24029 tablet_service.cc:1467] Processing CreateTablet for tablet eb96ad2289754e428a1c109b2d59c53b (DEFAULT_TABLE table=default.Table_bc1e9d346aa846f8abca7504185f7cdb [id=8d27878c6c8f4234908a2c78cb8bbd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.825058 24179 log.cc:1196] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/c316e1aeab224634b760641a4ce188c8
I20240502 08:16:39.829656 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 090bec30f5664421b9bfe3f2810afd14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.829900 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a97427e819b4826b1ed89e6d021c8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.832080 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 4b8d81f071d74b218fcfd4f4e8c20bab
I20240502 08:16:39.832880 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet 128c88af54d24ffd895e9de19cdb4c8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.833063 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b8d81f071d74b218fcfd4f4e8c20bab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.834479 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet a3043c5ed98c401b82c6e967ca3f3613 with delete_type TABLET_DATA_DELETED (replaced by tablet 0281c8123eab4c48bddc377f5695c6e0) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.834698 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bcc7169796e4ee0910f5e7084849012 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.834941 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92adebc0c69e4a52b975253656132495. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.836108 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet 654ce33d1e3d472eb6ae6a726eb803bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.836313 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet bcfacc302ff04178bfffc48111cd7a3f with delete_type TABLET_DATA_DELETED (replaced by tablet b0cf1a6de3d649b984c50f45393ec0f1) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.836701 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet ee022e8e86f8477a8fa1e58d52644b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.837070 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 41f2e98ef2884b05994d02114e992e80 (DEFAULT_TABLE table=default.Table_da999521d34e478fab2c024a89fa0025 [id=b6dee89ecfb443d99267917d4df78e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.838698 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet df9402cf8abc4b4d893d42f9a3a99016 with delete_type TABLET_DATA_DELETED (replaced by tablet fddac683eda14ec1a0140926380f2c07) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.839869 24174 tablet_bootstrap.cc:492] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:39.840029 24174 ts_tablet_manager.cc:1367] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.070s user 0.005s sys 0.000s
I20240502 08:16:39.840790 24174 raft_consensus.cc:348] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.841012 24174 raft_consensus.cc:374] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.841109 24174 raft_consensus.cc:729] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.840961 24034 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b8d81f071d74b218fcfd4f4e8c20bab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.841349 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet 654ce33d1e3d472eb6ae6a726eb803bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.841311 24174 consensus_queue.cc:260] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.844748 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet ee022e8e86f8477a8fa1e58d52644b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.845196 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 25c089f0c1b14e419f250d03d4eaad12
I20240502 08:16:39.845410 24039 tablet_service.cc:1467] Processing CreateTablet for tablet ac50774b2dd241de9cae8096578d183e (DEFAULT_TABLE table=default.Table_f0695a0f3e5c48c7aa4f998a18be36e1 [id=30defee3de2445f7bc5017fb23ab2ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.845870 23604 catalog_manager.cc:6554] ReplaceTablet: tablet 25c089f0c1b14e419f250d03d4eaad12 deleted and replaced by tablet 5e56af7b42cb481a8363d60f49ee3f61
I20240502 08:16:39.848871 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 751808dfc1964b539f6f836d786156aa
I20240502 08:16:39.850421 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet 25c089f0c1b14e419f250d03d4eaad12 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e56af7b42cb481a8363d60f49ee3f61) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.850497 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet 751808dfc1964b539f6f836d786156aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.850754 24026 tablet_service.cc:1514] Processing DeleteTablet for tablet aacdf506221044ffada0fec0c7d42d0b with delete_type TABLET_DATA_DELETED (replaced by tablet 90d4e2f23c9649a4b1dfb6a179ca7be5) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.851442 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 25c089f0c1b14e419f250d03d4eaad12 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e56af7b42cb481a8363d60f49ee3f61) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:39.851801 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet aacdf506221044ffada0fec0c7d42d0b with delete_type TABLET_DATA_DELETED (replaced by tablet 90d4e2f23c9649a4b1dfb6a179ca7be5) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.852002 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet d5bab1b7c48248a7973c27d0a865b34b with delete_type TABLET_DATA_DELETED (replaced by tablet 7c374f42f353497fa1d745164e7fbf61) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.852085 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet 751808dfc1964b539f6f836d786156aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.852578 23886 tablet_service.cc:1514] Processing DeleteTablet for tablet bcfacc302ff04178bfffc48111cd7a3f with delete_type TABLET_DATA_DELETED (replaced by tablet b0cf1a6de3d649b984c50f45393ec0f1) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.852783 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b8d81f071d74b218fcfd4f4e8c20bab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:39.853017 23888 tablet_service.cc:1467] Processing CreateTablet for tablet ac50774b2dd241de9cae8096578d183e (DEFAULT_TABLE table=default.Table_f0695a0f3e5c48c7aa4f998a18be36e1 [id=30defee3de2445f7bc5017fb23ab2ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.853487 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac50774b2dd241de9cae8096578d183e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.853520 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af6efeebaebc495aba017a922e243c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.855595 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb96ad2289754e428a1c109b2d59c53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.859441 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e645bb6e2948e091e08433944a52f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.863420 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet 25c089f0c1b14e419f250d03d4eaad12 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e56af7b42cb481a8363d60f49ee3f61) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.864810 24181 ts_tablet_manager.cc:1888] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:39.864965 24181 log.cc:1196] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/3cc30014a1ac45de937751eed36877a9
I20240502 08:16:39.865350 24181 ts_tablet_manager.cc:1909] T 3cc30014a1ac45de937751eed36877a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:39.866848 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8910bc67c6234a538c8160e7c020653c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.867376 24175 ts_tablet_manager.cc:1367] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.524s user 0.006s sys 0.007s
I20240502 08:16:39.867496 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac50774b2dd241de9cae8096578d183e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.868482 24175 raft_consensus.cc:348] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.868714 24175 raft_consensus.cc:374] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.868790 24175 raft_consensus.cc:729] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.869071 24175 consensus_queue.cc:260] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.870137 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f2e98ef2884b05994d02114e992e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.870147 24175 ts_tablet_manager.cc:1398] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:16:39.870616 24175 tablet_bootstrap.cc:492] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:39.872402 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet 751808dfc1964b539f6f836d786156aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:39.876612 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 41f2e98ef2884b05994d02114e992e80 (DEFAULT_TABLE table=default.Table_da999521d34e478fab2c024a89fa0025 [id=b6dee89ecfb443d99267917d4df78e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.877020 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f2e98ef2884b05994d02114e992e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.877266 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9467974d67c64c7eb5e488ac51f1240d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.876612 24179 ts_tablet_manager.cc:1909] T c316e1aeab224634b760641a4ce188c8 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:39.878396 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 257) took 1657 ms. Trace:
I20240502 08:16:39.878495 24179 rpcz_store.cc:276] 0502 08:16:38.221190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.221240 (+ 50us) service_pool.cc:225] Handling call
0502 08:16:39.497985 (+1276745us) ts_tablet_manager.cc:1120] Deleting tablet c316e1aeab224634b760641a4ce188c8
0502 08:16:39.497994 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:39.575234 (+ 77240us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:39.686993 (+111759us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:39.878345 (+191352us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1276343}
I20240502 08:16:39.878984 24179 tablet_replica.cc:330] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:39.879206 24179 raft_consensus.cc:2229] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:39.879376 24179 raft_consensus.cc:2258] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:39.879940 24179 ts_tablet_manager.cc:1875] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:39.882874 24173 tablet_bootstrap.cc:492] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:39.883059 24173 ts_tablet_manager.cc:1367] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.125s user 0.004s sys 0.001s
I20240502 08:16:39.886682 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 279) took 1561 ms. Trace:
I20240502 08:16:39.888078 24180 tablet_replica.cc:330] T d5bab1b7c48248a7973c27d0a865b34b P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:39.888293 24180 raft_consensus.cc:2229] T d5bab1b7c48248a7973c27d0a865b34b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240502 08:16:39.895668 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 4b8d81f071d74b218fcfd4f4e8c20bab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b8d81f071d74b218fcfd4f4e8c20bab already in progress: creating tablet
I20240502 08:16:39.897449 24175 tablet_bootstrap.cc:654] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.900884 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet eb96ad2289754e428a1c109b2d59c53b from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:39.906906 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet d8f7b5f018d84680ac5533bf99102d31 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:39.907896 24556 raft_consensus.cc:482] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:39.908114 24556 raft_consensus.cc:504] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.908775 24556 leader_election.cc:288] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:39.909752 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e870c3a0e340d1bad1c1fa0380a762" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:39.910295 23861 consensus_peers.cc:485] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1df565c1c6564efd88fa557c50522d7e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:39.911113 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c29e7767625846bc83fdd0c850f54966" } new_table_name: "default.Table_d4bd37f8c6e54ecf82cd247846aa06d0" modify_external_catalogs: true
I20240502 08:16:39.910759 24173 raft_consensus.cc:348] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.911801 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b76e1d838fa444179e6dbaa30058595a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:39.912304 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b76e1d838fa444179e6dbaa30058595a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.914721 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e870c3a0e340d1bad1c1fa0380a762" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:39.913555 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_55ca9dedb821436ea68a7980e085adb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:39.914968 24060 raft_consensus.cc:2454] T 43e870c3a0e340d1bad1c1fa0380a762 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
W20240502 08:16:39.915230 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55ca9dedb821436ea68a7980e085adb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.915406 24173 raft_consensus.cc:374] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.915818 24173 raft_consensus.cc:729] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.916071 24173 consensus_queue.cc:260] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.917138 24173 ts_tablet_manager.cc:1398] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20240502 08:16:39.917950 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_20c5ace604d64b6cabead6037cdc04e1" } modify_external_catalogs: true
I20240502 08:16:39.918023 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7929569b25404ea08efa4875a3c241a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:39.918171 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_20c5ace604d64b6cabead6037cdc04e1" } modify_external_catalogs: true
W20240502 08:16:39.910230 23723 leader_election.cc:341] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:39.918531 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7929569b25404ea08efa4875a3c241a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.918603 23723 leader_election.cc:302] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:39.918876 24329 raft_consensus.cc:2781] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:39.918980 24329 raft_consensus.cc:482] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:39.919081 24329 raft_consensus.cc:3037] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.911623 24174 ts_tablet_manager.cc:1398] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.071s user 0.002s sys 0.000s
I20240502 08:16:39.921337 24174 tablet_bootstrap.cc:492] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:39.922432 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6f82c95f8aa44cadabb6c55876c6f241" } new_table_name: "default.Table_d416b40fe3d24dafa8db5ee61d13acc0" modify_external_catalogs: true
I20240502 08:16:39.914467 24180 raft_consensus.cc:2258] T d5bab1b7c48248a7973c27d0a865b34b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:39.916241 24181 rpcz_store.cc:276] 0502 08:16:38.318201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.318267 (+ 66us) service_pool.cc:225] Handling call
0502 08:16:39.665920 (+1347653us) ts_tablet_manager.cc:1120] Deleting tablet 3cc30014a1ac45de937751eed36877a9
0502 08:16:39.665931 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:39.818426 (+152495us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:39.864631 (+ 46205us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:39.879509 (+ 14878us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1299,"tablet-delete.queue_time_us":1347395}
I20240502 08:16:39.925086 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 3cc30014a1ac45de937751eed36877a9 (table default.Table_d8aa2cd6c42444c48c383bf13e3728fd [id=9cbc066db19142d3bf0fd50cbe8fc91a]) successfully deleted
I20240502 08:16:39.925360 24329 raft_consensus.cc:504] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:39.925693 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8b1fa17dad143f0a7f1751e651eb81a consensus state has no leader
I20240502 08:16:39.926052 24329 leader_election.cc:288] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:39.926031 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_39406aba7a1b46719724dece66b69fdd" } new_table_name: "default.Table_d8c6955b89724769bbfd81db2b5be455" modify_external_catalogs: true
I20240502 08:16:39.926796 24180 ts_tablet_manager.cc:1875] T d5bab1b7c48248a7973c27d0a865b34b P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:39.927349 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e870c3a0e340d1bad1c1fa0380a762" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:39.927770 24173 tablet_bootstrap.cc:492] T ce758c4a9258427987e8e83f579a2aa2 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:39.930078 23723 leader_election.cc:341] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:39.934404 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b9b22cc686024294ab1843b25cb74af7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:39.934604 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 280) took 1602 ms. Trace:
I20240502 08:16:39.934788 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e870c3a0e340d1bad1c1fa0380a762" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:39.935030 24060 raft_consensus.cc:3037] T 43e870c3a0e340d1bad1c1fa0380a762 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:39.935127 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9b22cc686024294ab1843b25cb74af7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.936198 24180 ts_tablet_manager.cc:1888] T d5bab1b7c48248a7973c27d0a865b34b P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:39.936331 24180 log.cc:1196] T d5bab1b7c48248a7973c27d0a865b34b P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/d5bab1b7c48248a7973c27d0a865b34b
I20240502 08:16:39.936707 24180 ts_tablet_manager.cc:1909] T d5bab1b7c48248a7973c27d0a865b34b P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:39.938140 24060 raft_consensus.cc:2454] T 43e870c3a0e340d1bad1c1fa0380a762 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:39.940097 24181 rpcz_store.cc:276] 0502 08:16:38.322937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.322980 (+ 43us) service_pool.cc:225] Handling call
0502 08:16:39.925359 (+1602379us) ts_tablet_manager.cc:1120] Deleting tablet b1fac7df16fc41d7a3297055a7258270
0502 08:16:39.925369 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:39.925675 (+ 306us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1602204}
I20240502 08:16:39.940361 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet d5bab1b7c48248a7973c27d0a865b34b (table default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]) successfully deleted
I20240502 08:16:39.940599 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 281) took 1594 ms. Trace:
I20240502 08:16:39.940697 24181 rpcz_store.cc:276] 0502 08:16:38.346048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.346111 (+ 63us) service_pool.cc:225] Handling call
0502 08:16:39.940423 (+1594312us) ts_tablet_manager.cc:1120] Deleting tablet dcdd765895254a39b86b692b5d812c83
0502 08:16:39.940431 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:39.940554 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1555478}
I20240502 08:16:39.941007 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 282) took 1527 ms. Trace:
I20240502 08:16:39.941108 24181 rpcz_store.cc:276] 0502 08:16:38.413905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.413970 (+ 65us) service_pool.cc:225] Handling call
0502 08:16:39.940877 (+1526907us) ts_tablet_manager.cc:1120] Deleting tablet c5d28098dcbc4761a34ee65769d6fd7c
0502 08:16:39.940883 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:39.940969 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1526657}
I20240502 08:16:39.941371 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 315) took 1143 ms. Trace:
I20240502 08:16:39.941453 24181 rpcz_store.cc:276] 0502 08:16:38.797686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.007969 (+210283us) service_pool.cc:225] Handling call
0502 08:16:39.941272 (+933303us) ts_tablet_manager.cc:1120] Deleting tablet 848a0865eb2b4828a2594a17d96869d0
0502 08:16:39.941279 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:39.941341 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":933094}
I20240502 08:16:39.941794 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 316) took 1135 ms. Trace:
I20240502 08:16:39.941886 24181 rpcz_store.cc:276] 0502 08:16:38.805866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.008196 (+202330us) service_pool.cc:225] Handling call
0502 08:16:39.941687 (+933491us) ts_tablet_manager.cc:1120] Deleting tablet 0882b7509a74475780eb700818aab398
0502 08:16:39.941694 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:39.941764 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":933294}
I20240502 08:16:39.942185 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 317) took 1136 ms. Trace:
I20240502 08:16:39.942276 24181 rpcz_store.cc:276] 0502 08:16:38.805997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.008406 (+202409us) service_pool.cc:225] Handling call
0502 08:16:39.942072 (+933666us) ts_tablet_manager.cc:1120] Deleting tablet 71aa916247c449fea718b35869a6df20
0502 08:16:39.942081 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:39.942156 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":933510}
I20240502 08:16:39.942572 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 318) took 1136 ms. Trace:
I20240502 08:16:39.942662 24181 rpcz_store.cc:276] 0502 08:16:38.806070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.008572 (+202502us) service_pool.cc:225] Handling call
0502 08:16:39.942454 (+933882us) ts_tablet_manager.cc:1120] Deleting tablet b11f504b93574e42a9bf30fde6d08207
0502 08:16:39.942460 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:39.942544 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":933703}
I20240502 08:16:39.942883 24181 tablet_replica.cc:330] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:39.943068 24181 raft_consensus.cc:2229] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:39.943207 24181 raft_consensus.cc:2258] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:39.943702 24181 ts_tablet_manager.cc:1875] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:39.944643 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet dcdd765895254a39b86b692b5d812c83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dcdd765895254a39b86b692b5d812c83 already in progress: creating tablet
W20240502 08:16:39.944900 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet c5d28098dcbc4761a34ee65769d6fd7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5d28098dcbc4761a34ee65769d6fd7c already in progress: creating tablet
W20240502 08:16:39.945097 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 848a0865eb2b4828a2594a17d96869d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 848a0865eb2b4828a2594a17d96869d0 already in progress: creating tablet
W20240502 08:16:39.945250 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 0882b7509a74475780eb700818aab398 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0882b7509a74475780eb700818aab398 already in progress: creating tablet
W20240502 08:16:39.945403 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 71aa916247c449fea718b35869a6df20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71aa916247c449fea718b35869a6df20 already in progress: creating tablet
W20240502 08:16:39.945612 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet b11f504b93574e42a9bf30fde6d08207 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b11f504b93574e42a9bf30fde6d08207 already in progress: creating tablet
W20240502 08:16:39.946717 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 25c089f0c1b14e419f250d03d4eaad12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25c089f0c1b14e419f250d03d4eaad12 already in progress: creating tablet
I20240502 08:16:39.946971 24329 raft_consensus.cc:482] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:39.947122 24329 raft_consensus.cc:504] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.947850 24329 leader_election.cc:288] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:39.948522 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 751808dfc1964b539f6f836d786156aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 751808dfc1964b539f6f836d786156aa already in progress: creating tablet
I20240502 08:16:39.950237 23723 leader_election.cc:302] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:39.950402 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c52f1cae3a4ad38a9f95bfaf055e76" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:39.950659 23924 raft_consensus.cc:1261] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:39.950690 24329 raft_consensus.cc:2781] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:39.950831 24329 raft_consensus.cc:686] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:39.951150 24329 consensus_queue.cc:237] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:39.951375 23723 leader_election.cc:341] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:39.951766 24556 consensus_queue.cc:1035] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:39.963501 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_810beecf9ff143d38f72f76e5548a1db" } new_table_name: "default.Table_236150d3903d437d95f67b0edec36091" modify_external_catalogs: true
I20240502 08:16:39.964123 24031 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 327) took 1052 ms. Trace:
I20240502 08:16:39.964267 24031 rpcz_store.cc:276] 0502 08:16:38.904947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.095388 (+190441us) service_pool.cc:225] Handling call
0502 08:16:39.175399 (+ 80011us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:39.175429 (+ 30us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:39.896311 (+720882us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:39.956668 (+ 60357us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:39.956866 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:39.971122 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c52f1cae3a4ad38a9f95bfaf055e76" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:39.971356 24057 raft_consensus.cc:2454] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:39.971940 24580 tablet.cc:1722] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:39.971968 23750 tablet_service.cc:1467] Processing CreateTablet for tablet f1510777d3cb4ae899542796b5a493ad (DEFAULT_TABLE table=default.Table_32836076d604482a9e0963986e34af91 [id=edae4b0f01754e09a93a0d1d6697610d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.972313 24060 raft_consensus.cc:3037] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.972726 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1510777d3cb4ae899542796b5a493ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.974121 24175 tablet_bootstrap.cc:492] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:39.974340 24175 ts_tablet_manager.cc:1367] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.104s user 0.003s sys 0.000s
I20240502 08:16:39.975418 24175 raft_consensus.cc:348] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.975677 24175 raft_consensus.cc:374] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.975776 24175 raft_consensus.cc:729] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.975975 24175 consensus_queue.cc:260] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.976750 24175 ts_tablet_manager.cc:1398] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:39.977154 24175 tablet_bootstrap.cc:492] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:39.977299 24489 raft_consensus.cc:482] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:39.977442 24489 raft_consensus.cc:504] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:39.978091 24489 leader_election.cc:288] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:39.980062 24173 tablet_bootstrap.cc:654] T ce758c4a9258427987e8e83f579a2aa2 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.980165 23602 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet b6c52f1cae3a4ad38a9f95bfaf055e76 (table default.Table_35b808b0487f463dbcfa4a773a2a20f0 [id=922fb979b5894023a68820aece0b8c5e]). Expected version 1 got 0
I20240502 08:16:39.981604 23604 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 1ce5ccfd29724dc49afd0892b99f3169 (table default.Table_bdbe807258a5454795eaff8c27bb17dd [id=3a9d7e8fe2c84cb4942fee9053a012b0]). Expected version 1 got 0
I20240502 08:16:39.982635 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c22209bdcbfd4dfa9e643ab9c98dc005"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:39.983193 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c22209bdcbfd4dfa9e643ab9c98dc005 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:39.985517 23603 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet b6c52f1cae3a4ad38a9f95bfaf055e76 (table default.Table_35b808b0487f463dbcfa4a773a2a20f0 [id=922fb979b5894023a68820aece0b8c5e]). Expected version 1 got 0
I20240502 08:16:39.986272 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_307041e730f248a5bd4d265f08aad5e4" } modify_external_catalogs: true
I20240502 08:16:39.986479 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_307041e730f248a5bd4d265f08aad5e4" } modify_external_catalogs: true
I20240502 08:16:39.990517 23723 leader_election.cc:302] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:39.990918 24556 raft_consensus.cc:2781] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:39.991046 24556 raft_consensus.cc:482] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:39.991127 24556 raft_consensus.cc:3037] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.992283 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642744194e22461f8316a78689fd99ff" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:39.992568 24057 raft_consensus.cc:2454] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:39.992843 24060 raft_consensus.cc:1261] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:39.994860 24556 raft_consensus.cc:504] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:39.995745 24556 leader_election.cc:288] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:39.996639 23723 leader_election.cc:302] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:39.997090 24556 raft_consensus.cc:2781] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:39.997241 24556 raft_consensus.cc:482] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:39.997429 24556 raft_consensus.cc:3037] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:40.000164 24556 raft_consensus.cc:504] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:40.000325 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c52f1cae3a4ad38a9f95bfaf055e76" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:40.001101 24556 leader_election.cc:288] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:40.001219 24060 raft_consensus.cc:3037] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:40.001642 24556 consensus_queue.cc:1035] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:40.004609 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642744194e22461f8316a78689fd99ff" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:40.004818 24057 raft_consensus.cc:3037] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.923017 24174 tablet_bootstrap.cc:654] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.018759 24057 raft_consensus.cc:2454] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:40.020187 24025 tablet_service.cc:1467] Processing CreateTablet for tablet f1510777d3cb4ae899542796b5a493ad (DEFAULT_TABLE table=default.Table_32836076d604482a9e0963986e34af91 [id=edae4b0f01754e09a93a0d1d6697610d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.021013 23723 leader_election.cc:302] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:40.021036 24181 ts_tablet_manager.cc:1888] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.022609 24585 tablet.cc:1722] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:40.023269 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1510777d3cb4ae899542796b5a493ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.024468 24181 log.cc:1196] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/f51b26043ef44b30846498cd1497607c
I20240502 08:16:40.027640 24556 raft_consensus.cc:2781] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:40.014585 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c52f1cae3a4ad38a9f95bfaf055e76" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:40.043706 24181 ts_tablet_manager.cc:1909] T f51b26043ef44b30846498cd1497607c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:40.044093 24556 raft_consensus.cc:686] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:40.044523 24556 consensus_queue.cc:237] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:40.047073 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 42e11d405a2e4e518a10bf383e9ae6c4 (DEFAULT_TABLE table=default.Table_01e045a6ce334744b9ab0c8d6d390054 [id=15cd2d29f86a49e295158ba17e364a23]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.047564 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e11d405a2e4e518a10bf383e9ae6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.026728 23921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642744194e22461f8316a78689fd99ff" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:40.060283 23605 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet b6c52f1cae3a4ad38a9f95bfaf055e76 (table default.Table_35b808b0487f463dbcfa4a773a2a20f0 [id=922fb979b5894023a68820aece0b8c5e]). Expected version 1 got 0
I20240502 08:16:40.060688 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 2b98cc8728c042f4b7f6496fcc5ef913 (DEFAULT_TABLE table=default.Table_5fe628d3da624233941f440e3743da88 [id=ca161ce6780b446fa5743d3974cfa27b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.061161 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b98cc8728c042f4b7f6496fcc5ef913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.061204 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c72b83e8da844cb6a570c14f50293f34" } modify_external_catalogs: true
I20240502 08:16:40.061451 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c72b83e8da844cb6a570c14f50293f34" } modify_external_catalogs: true
W20240502 08:16:40.070360 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0cf1a6de3d649b984c50f45393ec0f1 consensus state has no leader
W20240502 08:16:40.071163 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 156c7356ece342aea0d8db667ebad89c consensus state has no leader
W20240502 08:16:40.072044 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec10ff7251a5442a8ecc1881b34db74d consensus state has no leader
I20240502 08:16:40.072779 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 18625a1d971c43549b9f80e6046efc9c (DEFAULT_TABLE table=default.Table_70822394abb947128d1014970c8ef471 [id=62ad2ebd7db541958a77f09c07ab946e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.073213 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18625a1d971c43549b9f80e6046efc9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.083803 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 1b175e1b55bf4102804666b6fffccead (DEFAULT_TABLE table=default.Table_a320ac94a297479885cba64075ce2ae2 [id=5a9755c4cebd4ad2ab00ce225b25334c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.084383 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b175e1b55bf4102804666b6fffccead. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.089179 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 1305dbdac3994224a047df139cf8d518 (DEFAULT_TABLE table=default.Table_37ab1c2d8931468482c9d99dd6536fe5 [id=47b742f54bb14af8a878ac416a991240]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.089928 23758 tablet_service.cc:1467] Processing CreateTablet for tablet c45aa0dd26524e0fba27e4eaf17d704e (DEFAULT_TABLE table=default.Table_d7af10ea88e34ce9b9e4565cb73c02de [id=589deb067c2b4cd086d0277ecfed5126]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.090385 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45aa0dd26524e0fba27e4eaf17d704e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.091084 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1305dbdac3994224a047df139cf8d518. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.092844 24174 tablet_bootstrap.cc:492] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:40.093044 24174 ts_tablet_manager.cc:1367] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.172s user 0.004s sys 0.000s
I20240502 08:16:40.094123 24174 raft_consensus.cc:348] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.094591 24174 raft_consensus.cc:374] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.094705 24174 raft_consensus.cc:729] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.095032 24174 consensus_queue.cc:260] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.095872 24174 ts_tablet_manager.cc:1398] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:16:40.096350 24174 tablet_bootstrap.cc:492] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:40.098817 24173 tablet_bootstrap.cc:492] T ce758c4a9258427987e8e83f579a2aa2 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:40.098877 24038 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 326) took 1196 ms. Trace:
I20240502 08:16:40.100772 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 665bbee2d15b4112b651844431535d0e (DEFAULT_TABLE table=default.Table_5509aa65d43f4547853d26666042d04d [id=427121bc94c247eaa315b6c830770d61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.101240 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 665bbee2d15b4112b651844431535d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.103583 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2d9b44d9d9d34aa184855169eb278e94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:40.104142 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d9b44d9d9d34aa184855169eb278e94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:40.104990 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_86918f47dba74a9eb36be5145a017f5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:40.105541 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86918f47dba74a9eb36be5145a017f5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:40.106600 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c466cab9104c4631a59e1d5e481c5fe0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:40.106959 23921 raft_consensus.cc:3037] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:40.107030 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c466cab9104c4631a59e1d5e481c5fe0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:40.108599 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e01d80203b2347cb9f8941844cdcc6fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:40.109055 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e01d80203b2347cb9f8941844cdcc6fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:40.110744 23723 leader_election.cc:341] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:40.115098 24038 rpcz_store.cc:276] 0502 08:16:38.901949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.021197 (+119248us) service_pool.cc:225] Handling call
0502 08:16:39.136385 (+115188us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:39.136434 (+ 49us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:40.002251 (+865817us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.098671 (+ 96420us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:40.098831 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:40.115482 24060 raft_consensus.cc:2454] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:40.113191 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642744194e22461f8316a78689fd99ff" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:40.121976 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 07a9d13d3a2d4e6dbc04998c3dd7e1a8 (DEFAULT_TABLE table=default.Table_c538b531b58147b89d720628ee7556dd [id=7a49ccfdd6884281ba9b882debd04871]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.122437 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a9d13d3a2d4e6dbc04998c3dd7e1a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.125360 24175 tablet_bootstrap.cc:654] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.126722 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 6088f160a13c4ff09f72bed72f69eaf8 (DEFAULT_TABLE table=default.Table_0287506f7f94418eb19163c272a30fb8 [id=4fecac7b0d9042b3b4f674f3fc1338ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.130402 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 9c072eb45e1b41eeb5aafa6d5a786944
I20240502 08:16:40.131995 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet eb96ad2289754e428a1c109b2d59c53b
I20240502 08:16:40.132426 23601 catalog_manager.cc:6554] ReplaceTablet: tablet eb96ad2289754e428a1c109b2d59c53b deleted and replaced by tablet 6ee764398b364d029ae7f0f575a30d64
I20240502 08:16:40.132715 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0dc68fd335024f2dbce6b1fcc698f1be" } new_table_name: "default.Table_9a9569245562493dafbeb639c6492826" modify_external_catalogs: true
I20240502 08:16:40.132899 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 42e11d405a2e4e518a10bf383e9ae6c4 (DEFAULT_TABLE table=default.Table_01e045a6ce334744b9ab0c8d6d390054 [id=15cd2d29f86a49e295158ba17e364a23]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.133414 23723 leader_election.cc:302] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:40.134128 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 24a4104f4cb64889ad66810788ad6b29
I20240502 08:16:40.134217 24489 raft_consensus.cc:2781] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:40.134399 24489 raft_consensus.cc:686] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:40.134701 24489 consensus_queue.cc:237] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.134935 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9feed0e1d82c472aaf0e9b92cd9173d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:40.135443 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9feed0e1d82c472aaf0e9b92cd9173d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:40.136538 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c072eb45e1b41eeb5aafa6d5a786944 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:40.136765 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet eb96ad2289754e428a1c109b2d59c53b with delete_type TABLET_DATA_DELETED (replaced by tablet 6ee764398b364d029ae7f0f575a30d64) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:40.136983 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a4104f4cb64889ad66810788ad6b29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:40.137233 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 9c072eb45e1b41eeb5aafa6d5a786944 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c072eb45e1b41eeb5aafa6d5a786944 already in progress: creating tablet
W20240502 08:16:40.137732 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet eb96ad2289754e428a1c109b2d59c53b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb96ad2289754e428a1c109b2d59c53b already in progress: creating tablet
W20240502 08:16:40.137991 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 24a4104f4cb64889ad66810788ad6b29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24a4104f4cb64889ad66810788ad6b29 already in progress: creating tablet
I20240502 08:16:40.140220 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6088f160a13c4ff09f72bed72f69eaf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.142097 24174 tablet_bootstrap.cc:654] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.142298 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 2b98cc8728c042f4b7f6496fcc5ef913 (DEFAULT_TABLE table=default.Table_5fe628d3da624233941f440e3743da88 [id=ca161ce6780b446fa5743d3974cfa27b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.144831 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 18625a1d971c43549b9f80e6046efc9c (DEFAULT_TABLE table=default.Table_70822394abb947128d1014970c8ef471 [id=62ad2ebd7db541958a77f09c07ab946e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.145290 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18625a1d971c43549b9f80e6046efc9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.146160 23888 tablet_service.cc:1467] Processing CreateTablet for tablet f1510777d3cb4ae899542796b5a493ad (DEFAULT_TABLE table=default.Table_32836076d604482a9e0963986e34af91 [id=edae4b0f01754e09a93a0d1d6697610d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.147861 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 42e11d405a2e4e518a10bf383e9ae6c4 (DEFAULT_TABLE table=default.Table_01e045a6ce334744b9ab0c8d6d390054 [id=15cd2d29f86a49e295158ba17e364a23]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.137481 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1704983c4496461d9fa8ea54781f5cb1" } modify_external_catalogs: true
I20240502 08:16:40.149791 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1704983c4496461d9fa8ea54781f5cb1" } modify_external_catalogs: true
I20240502 08:16:40.149950 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 07a9d13d3a2d4e6dbc04998c3dd7e1a8 (DEFAULT_TABLE table=default.Table_c538b531b58147b89d720628ee7556dd [id=7a49ccfdd6884281ba9b882debd04871]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.150374 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c072eb45e1b41eeb5aafa6d5a786944 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:40.150602 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet eb96ad2289754e428a1c109b2d59c53b with delete_type TABLET_DATA_DELETED (replaced by tablet 6ee764398b364d029ae7f0f575a30d64) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:40.150909 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 665bbee2d15b4112b651844431535d0e (DEFAULT_TABLE table=default.Table_5509aa65d43f4547853d26666042d04d [id=427121bc94c247eaa315b6c830770d61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.151382 23894 tablet_service.cc:1467] Processing CreateTablet for tablet c45aa0dd26524e0fba27e4eaf17d704e (DEFAULT_TABLE table=default.Table_d7af10ea88e34ce9b9e4565cb73c02de [id=589deb067c2b4cd086d0277ecfed5126]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.152421 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 1b175e1b55bf4102804666b6fffccead (DEFAULT_TABLE table=default.Table_a320ac94a297479885cba64075ce2ae2 [id=5a9755c4cebd4ad2ab00ce225b25334c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.152720 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b98cc8728c042f4b7f6496fcc5ef913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.155304 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 1305dbdac3994224a047df139cf8d518 (DEFAULT_TABLE table=default.Table_37ab1c2d8931468482c9d99dd6536fe5 [id=47b742f54bb14af8a878ac416a991240]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.155689 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 2b98cc8728c042f4b7f6496fcc5ef913 (DEFAULT_TABLE table=default.Table_5fe628d3da624233941f440e3743da88 [id=ca161ce6780b446fa5743d3974cfa27b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.156180 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b98cc8728c042f4b7f6496fcc5ef913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.158488 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 320) took 1322 ms. Trace:
I20240502 08:16:40.158612 24181 rpcz_store.cc:276] 0502 08:16:38.806247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.012655 (+206408us) service_pool.cc:225] Handling call
0502 08:16:39.942828 (+930173us) ts_tablet_manager.cc:1120] Deleting tablet f51b26043ef44b30846498cd1497607c
0502 08:16:39.942834 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:39.960899 (+ 18065us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.020881 (+ 59982us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.128764 (+107883us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":929950}
I20240502 08:16:40.159394 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 18625a1d971c43549b9f80e6046efc9c (DEFAULT_TABLE table=default.Table_70822394abb947128d1014970c8ef471 [id=62ad2ebd7db541958a77f09c07ab946e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.159794 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a4104f4cb64889ad66810788ad6b29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:40.159972 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a9d13d3a2d4e6dbc04998c3dd7e1a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.160570 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet f51b26043ef44b30846498cd1497607c (table default.Table_da5e259b0d0c47cf94ab30e3e98d1540 [id=956913447d9b4f55920aa4382692d608]) successfully deleted
I20240502 08:16:40.161201 24029 tablet_service.cc:1467] Processing CreateTablet for tablet c45aa0dd26524e0fba27e4eaf17d704e (DEFAULT_TABLE table=default.Table_d7af10ea88e34ce9b9e4565cb73c02de [id=589deb067c2b4cd086d0277ecfed5126]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.161594 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet eb96ad2289754e428a1c109b2d59c53b with delete_type TABLET_DATA_DELETED (replaced by tablet 6ee764398b364d029ae7f0f575a30d64) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.160235 24173 ts_tablet_manager.cc:1367] T ce758c4a9258427987e8e83f579a2aa2 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.180s user 0.005s sys 0.006s
I20240502 08:16:40.161191 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 07a9d13d3a2d4e6dbc04998c3dd7e1a8 (DEFAULT_TABLE table=default.Table_c538b531b58147b89d720628ee7556dd [id=7a49ccfdd6884281ba9b882debd04871]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.168092 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 1305dbdac3994224a047df139cf8d518 (DEFAULT_TABLE table=default.Table_37ab1c2d8931468482c9d99dd6536fe5 [id=47b742f54bb14af8a878ac416a991240]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.168668 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e11d405a2e4e518a10bf383e9ae6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.169639 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a4104f4cb64889ad66810788ad6b29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.169849 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c072eb45e1b41eeb5aafa6d5a786944 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.170085 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45aa0dd26524e0fba27e4eaf17d704e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.170915 23921 raft_consensus.cc:2454] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:40.171536 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 6088f160a13c4ff09f72bed72f69eaf8 (DEFAULT_TABLE table=default.Table_0287506f7f94418eb19163c272a30fb8 [id=4fecac7b0d9042b3b4f674f3fc1338ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.172668 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b175e1b55bf4102804666b6fffccead. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.173707 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1305dbdac3994224a047df139cf8d518. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.175609 24551 raft_consensus.cc:482] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:40.175751 24551 raft_consensus.cc:504] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:40.177853 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18625a1d971c43549b9f80e6046efc9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.178009 24551 leader_election.cc:288] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
W20240502 08:16:40.178928 23861 leader_election.cc:341] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: f36ceb0222ef473a828ca795f393174d
I20240502 08:16:40.180902 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1510777d3cb4ae899542796b5a493ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.182385 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1305dbdac3994224a047df139cf8d518. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.183066 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e11d405a2e4e518a10bf383e9ae6c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.161697 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 665bbee2d15b4112b651844431535d0e (DEFAULT_TABLE table=default.Table_5509aa65d43f4547853d26666042d04d [id=427121bc94c247eaa315b6c830770d61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.167943 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 1b175e1b55bf4102804666b6fffccead (DEFAULT_TABLE table=default.Table_a320ac94a297479885cba64075ce2ae2 [id=5a9755c4cebd4ad2ab00ce225b25334c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.188133 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 665bbee2d15b4112b651844431535d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.189742 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 665bbee2d15b4112b651844431535d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.192629 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a9d13d3a2d4e6dbc04998c3dd7e1a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.193356 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6088f160a13c4ff09f72bed72f69eaf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.194201 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 321) took 1352 ms. Trace:
I20240502 08:16:40.195036 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b175e1b55bf4102804666b6fffccead. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.195113 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45aa0dd26524e0fba27e4eaf17d704e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.196230 24181 rpcz_store.cc:276] 0502 08:16:38.806322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.012899 (+206577us) service_pool.cc:225] Handling call
0502 08:16:40.158894 (+1145995us) ts_tablet_manager.cc:1120] Deleting tablet 7c374f42f353497fa1d745164e7fbf61
0502 08:16:40.158904 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.159034 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1145826}
W20240502 08:16:40.197521 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 7c374f42f353497fa1d745164e7fbf61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c374f42f353497fa1d745164e7fbf61 already in progress: creating tablet
I20240502 08:16:40.200484 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 322) took 1393 ms. Trace:
I20240502 08:16:40.200594 24181 rpcz_store.cc:276] 0502 08:16:38.806396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.013079 (+206683us) service_pool.cc:225] Handling call
0502 08:16:40.196577 (+1183498us) ts_tablet_manager.cc:1120] Deleting tablet 5a1e208733844829b13222e00a432439
0502 08:16:40.196589 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.200160 (+ 3571us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1183324}
W20240502 08:16:40.201292 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 5a1e208733844829b13222e00a432439 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a1e208733844829b13222e00a432439 already in progress: creating tablet
I20240502 08:16:40.201844 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 323) took 1394 ms. Trace:
I20240502 08:16:40.201946 24181 rpcz_store.cc:276] 0502 08:16:38.806470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.013246 (+206776us) service_pool.cc:225] Handling call
0502 08:16:40.200861 (+1187615us) ts_tablet_manager.cc:1120] Deleting tablet b6417aafbdcf416ea7bff56bce5e6ddb
0502 08:16:40.200870 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.200964 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1187461}
W20240502 08:16:40.202600 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet b6417aafbdcf416ea7bff56bce5e6ddb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6417aafbdcf416ea7bff56bce5e6ddb already in progress: creating tablet
I20240502 08:16:40.203089 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 314) took 1425 ms. Trace:
I20240502 08:16:40.203164 24181 rpcz_store.cc:276] 0502 08:16:38.796258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.001019 (+204761us) service_pool.cc:225] Handling call
0502 08:16:40.202161 (+1201142us) ts_tablet_manager.cc:1120] Deleting tablet 1e5de793e74247dd9e73b7b1d58027d9
0502 08:16:40.202169 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.202254 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1106823}
W20240502 08:16:40.203728 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 1e5de793e74247dd9e73b7b1d58027d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e5de793e74247dd9e73b7b1d58027d9 already in progress: creating tablet
I20240502 08:16:40.202877 24173 raft_consensus.cc:348] T ce758c4a9258427987e8e83f579a2aa2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:40.208623 24173 raft_consensus.cc:374] T ce758c4a9258427987e8e83f579a2aa2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.208729 24173 raft_consensus.cc:729] T ce758c4a9258427987e8e83f579a2aa2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.210165 24173 consensus_queue.cc:260] T ce758c4a9258427987e8e83f579a2aa2 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:40.217849 24173 ts_tablet_manager.cc:1398] T ce758c4a9258427987e8e83f579a2aa2 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.050s user 0.001s sys 0.001s
I20240502 08:16:40.219513 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 324) took 1396 ms. Trace:
I20240502 08:16:40.219811 24173 tablet_bootstrap.cc:492] T 6788ffd2eb2248c3b84e474de56c37a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:40.226431 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 6088f160a13c4ff09f72bed72f69eaf8 (DEFAULT_TABLE table=default.Table_0287506f7f94418eb19163c272a30fb8 [id=4fecac7b0d9042b3b4f674f3fc1338ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.226913 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6088f160a13c4ff09f72bed72f69eaf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.227037 24181 rpcz_store.cc:276] 0502 08:16:38.806594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.013405 (+206811us) service_pool.cc:225] Handling call
0502 08:16:40.203369 (+1189964us) ts_tablet_manager.cc:1120] Deleting tablet dd3078cc1f6247609f8bd017ed076b45
0502 08:16:40.203377 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.203451 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1104901}
W20240502 08:16:40.228240 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet dd3078cc1f6247609f8bd017ed076b45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd3078cc1f6247609f8bd017ed076b45 already in progress: creating tablet
I20240502 08:16:40.228925 24181 tablet_replica.cc:330] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:40.229141 24181 raft_consensus.cc:2229] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.229918 24181 raft_consensus.cc:2258] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.230501 24181 ts_tablet_manager.cc:1875] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.228329 24600 tablet.cc:1722] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:40.258188 24173 tablet_bootstrap.cc:654] T 6788ffd2eb2248c3b84e474de56c37a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.264264 24181 ts_tablet_manager.cc:1888] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.264616 24181 log.cc:1196] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/b742c51f6204449a804889815604d5cb
I20240502 08:16:40.265148 24181 ts_tablet_manager.cc:1909] T b742c51f6204449a804889815604d5cb P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:40.274765 24173 tablet_bootstrap.cc:492] T 6788ffd2eb2248c3b84e474de56c37a9 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:40.275063 24173 ts_tablet_manager.cc:1367] T 6788ffd2eb2248c3b84e474de56c37a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.002s
I20240502 08:16:40.262637 24591 tablet.cc:1722] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:40.275962 24173 raft_consensus.cc:348] T 6788ffd2eb2248c3b84e474de56c37a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:40.278419 24173 raft_consensus.cc:374] T 6788ffd2eb2248c3b84e474de56c37a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.278604 24173 raft_consensus.cc:729] T 6788ffd2eb2248c3b84e474de56c37a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.278908 24173 consensus_queue.cc:260] T 6788ffd2eb2248c3b84e474de56c37a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:40.283864 24173 ts_tablet_manager.cc:1398] T 6788ffd2eb2248c3b84e474de56c37a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 08:16:40.286934 24179 ts_tablet_manager.cc:1888] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.287099 24179 log.cc:1196] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/ab70513985b54416856a29b32cecae4d
I20240502 08:16:40.290171 24173 tablet_bootstrap.cc:492] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:40.291724 24173 tablet_bootstrap.cc:654] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.292161 24179 ts_tablet_manager.cc:1909] T ab70513985b54416856a29b32cecae4d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:40.293869 24039 tablet_service.cc:1467] Processing CreateTablet for tablet a4d96cc17d9b4bcf8ce7c4414a63f9cb (DEFAULT_TABLE table=default.Table_e7935e6f9887457f9d33f01909abdcb5 [id=6aec8183dea443498fbd449595177a06]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.294474 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4d96cc17d9b4bcf8ce7c4414a63f9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.301738 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 319) took 1495 ms. Trace:
I20240502 08:16:40.301980 24181 rpcz_store.cc:276] 0502 08:16:38.806180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.008752 (+202572us) service_pool.cc:225] Handling call
0502 08:16:40.227850 (+1219098us) ts_tablet_manager.cc:1120] Deleting tablet b742c51f6204449a804889815604d5cb
0502 08:16:40.227859 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.237994 (+ 10135us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.264126 (+ 26132us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.301683 (+ 37557us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1112985}
I20240502 08:16:40.302594 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet b742c51f6204449a804889815604d5cb (table default.Table_9507475756c3401fa75fc34d4a4b7814 [id=f42bb435dc484af7990e8cf4671e1c85]) successfully deleted
I20240502 08:16:40.307565 23886 tablet_service.cc:1467] Processing CreateTablet for tablet a4d96cc17d9b4bcf8ce7c4414a63f9cb (DEFAULT_TABLE table=default.Table_e7935e6f9887457f9d33f01909abdcb5 [id=6aec8183dea443498fbd449595177a06]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.310756 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4d96cc17d9b4bcf8ce7c4414a63f9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.313905 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0b7c860d457f421294ca1d37248e40b5" } modify_external_catalogs: true
I20240502 08:16:40.314131 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0b7c860d457f421294ca1d37248e40b5" } modify_external_catalogs: true
I20240502 08:16:40.317142 23749 tablet_service.cc:1467] Processing CreateTablet for tablet a4d96cc17d9b4bcf8ce7c4414a63f9cb (DEFAULT_TABLE table=default.Table_e7935e6f9887457f9d33f01909abdcb5 [id=6aec8183dea443498fbd449595177a06]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.324986 23602 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_bdbe807258a5454795eaff8c27bb17dd [id=3a9d7e8fe2c84cb4942fee9053a012b0] alter complete (version 1)
I20240502 08:16:40.326660 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 08183523bf4a4d1c9020eb693aa07878
I20240502 08:16:40.331976 23888 tablet_service.cc:1514] Processing DeleteTablet for tablet 08183523bf4a4d1c9020eb693aa07878 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:40.333164 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 08183523bf4a4d1c9020eb693aa07878 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.338053 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 339) took 1344 ms. Trace:
I20240502 08:16:40.338179 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 90d4e2f23c9649a4b1dfb6a179ca7be5 (DEFAULT_TABLE table=default.Table_102eeba031aa4389820d28985935dd4c [id=9480d59a76fc401e959a1e41caafc845]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.338233 24181 rpcz_store.cc:276] 0502 08:16:38.994063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.409748 (+415685us) service_pool.cc:225] Handling call
0502 08:16:40.337695 (+927947us) ts_tablet_manager.cc:1120] Deleting tablet 18d284f71937459db0c0676ce4568175
0502 08:16:40.337709 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.338013 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":919555}
I20240502 08:16:40.339804 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 258) took 2117 ms. Trace:
I20240502 08:16:40.339917 24179 rpcz_store.cc:276] 0502 08:16:38.221846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.221895 (+ 49us) service_pool.cc:225] Handling call
0502 08:16:39.878896 (+1657001us) ts_tablet_manager.cc:1120] Deleting tablet ab70513985b54416856a29b32cecae4d
0502 08:16:39.878905 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:39.905726 (+ 26821us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.286781 (+381055us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.339764 (+ 52983us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1656681}
W20240502 08:16:40.342521 23858 leader_election.cc:341] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: f36ceb0222ef473a828ca795f393174d
I20240502 08:16:40.342689 23858 leader_election.cc:302] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:40.342892 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 18d284f71937459db0c0676ce4568175 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18d284f71937459db0c0676ce4568175 already in progress: creating tablet
I20240502 08:16:40.343000 24551 raft_consensus.cc:2726] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:40.343392 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet ab70513985b54416856a29b32cecae4d (table default.Table_d9b6d863d79341ebb994770c34ae2174 [id=259cb14e2ef6427f8dfba46f188050dd]) successfully deleted
W20240502 08:16:40.343467 23858 consensus_peers.cc:485] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1df565c1c6564efd88fa557c50522d7e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:40.348639 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4d96cc17d9b4bcf8ce7c4414a63f9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.349541 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d4e2f23c9649a4b1dfb6a179ca7be5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.350245 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 90d4e2f23c9649a4b1dfb6a179ca7be5 (DEFAULT_TABLE table=default.Table_102eeba031aa4389820d28985935dd4c [id=9480d59a76fc401e959a1e41caafc845]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.350718 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 90d4e2f23c9649a4b1dfb6a179ca7be5 (DEFAULT_TABLE table=default.Table_102eeba031aa4389820d28985935dd4c [id=9480d59a76fc401e959a1e41caafc845]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.351187 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d4e2f23c9649a4b1dfb6a179ca7be5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.354102 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 341) took 1335 ms. Trace:
I20240502 08:16:40.354285 24181 rpcz_store.cc:276] 0502 08:16:39.018269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.425397 (+407128us) service_pool.cc:225] Handling call
0502 08:16:40.353958 (+928561us) ts_tablet_manager.cc:1120] Deleting tablet 5453cc0b93ac4cbb90a0588bb2100d60
0502 08:16:40.353967 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.354070 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":927179}
W20240502 08:16:40.354879 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 5453cc0b93ac4cbb90a0588bb2100d60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5453cc0b93ac4cbb90a0588bb2100d60 already in progress: creating tablet
I20240502 08:16:40.356662 24040 tablet_service.cc:1467] Processing CreateTablet for tablet fba8f6e017e848d59239ae98432b1837 (DEFAULT_TABLE table=default.Table_63545ead8a7a4aae8b2daa7af0fe5113 [id=3deaa06a1e88491f9dda6abda2cbdd9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.357035 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba8f6e017e848d59239ae98432b1837. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.360989 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d4e2f23c9649a4b1dfb6a179ca7be5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.370373 24179 tablet_replica.cc:330] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:40.371474 23602 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 1ce5ccfd29724dc49afd0892b99f3169 (table default.Table_bdbe807258a5454795eaff8c27bb17dd [id=3a9d7e8fe2c84cb4942fee9053a012b0]). Expected version 1 got 0
I20240502 08:16:40.371729 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_70822394abb947128d1014970c8ef471" } new_table_name: "default.Table_e02044df8b8c4ca9a9bdccaaf14a2eb4" modify_external_catalogs: true
I20240502 08:16:40.372179 24179 raft_consensus.cc:2229] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.373391 24179 raft_consensus.cc:2258] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.374027 24179 ts_tablet_manager.cc:1875] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.377578 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet f36ceb0222ef473a828ca795f393174d with delete_type TABLET_DATA_DELETED (replaced by tablet df9402cf8abc4b4d893d42f9a3a99016) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:40.379266 24181 tablet_replica.cc:330] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:40.379523 24181 raft_consensus.cc:2229] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.379711 24181 raft_consensus.cc:2258] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.380316 24181 ts_tablet_manager.cc:1875] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.380296 23758 tablet_service.cc:1467] Processing CreateTablet for tablet fba8f6e017e848d59239ae98432b1837 (DEFAULT_TABLE table=default.Table_63545ead8a7a4aae8b2daa7af0fe5113 [id=3deaa06a1e88491f9dda6abda2cbdd9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.380450 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet 08183523bf4a4d1c9020eb693aa07878 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:40.380823 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet f51b26043ef44b30846498cd1497607c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:40.381379 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba8f6e017e848d59239ae98432b1837. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.381603 24180 tablet_replica.cc:330] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:40.382092 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d3c66c36f4cd494585b1e4cb114b81a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:40.382540 24180 raft_consensus.cc:2229] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
W20240502 08:16:40.382669 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3c66c36f4cd494585b1e4cb114b81a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:40.382916 24180 raft_consensus.cc:2258] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.385180 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet 8633864e248f416cb651da3b0c2aa936 with delete_type TABLET_DATA_DELETED (replaced by tablet 9761df0bbdee44af90135bffb8b5828f) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:40.386371 24180 ts_tablet_manager.cc:1875] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.390816 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet ce758c4a9258427987e8e83f579a2aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.393025 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5ad32437311f45d3a50a7fe4b134adb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:40.394503 24179 ts_tablet_manager.cc:1888] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.395785 24179 log.cc:1196] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/da4be1d381a64b7baeb1c60cb17d924a
I20240502 08:16:40.396396 24179 ts_tablet_manager.cc:1909] T da4be1d381a64b7baeb1c60cb17d924a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:40.397207 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 259) took 2174 ms. Trace:
W20240502 08:16:40.397523 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ad32437311f45d3a50a7fe4b134adb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:40.397827 24179 rpcz_store.cc:276] 0502 08:16:38.222411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.222451 (+ 40us) service_pool.cc:225] Handling call
0502 08:16:40.370305 (+2147854us) ts_tablet_manager.cc:1120] Deleting tablet da4be1d381a64b7baeb1c60cb17d924a
0502 08:16:40.370314 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.376562 (+ 6248us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.394359 (+ 17797us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.397157 (+ 2798us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2147593}
I20240502 08:16:40.398761 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 260) took 2175 ms. Trace:
I20240502 08:16:40.399257 24179 rpcz_store.cc:276] 0502 08:16:38.223139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.223277 (+ 138us) service_pool.cc:225] Handling call
0502 08:16:40.398588 (+2175311us) ts_tablet_manager.cc:1120] Deleting tablet 319fabf91458438ba5e40ab68b323055
0502 08:16:40.398598 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.398711 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2175088}
I20240502 08:16:40.400108 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 261) took 2176 ms. Trace:
I20240502 08:16:40.400543 24179 rpcz_store.cc:276] 0502 08:16:38.223242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.223616 (+ 374us) service_pool.cc:225] Handling call
0502 08:16:40.399950 (+2176334us) ts_tablet_manager.cc:1120] Deleting tablet 4a57ed188315493198e95cefb6907ab6
0502 08:16:40.399958 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.400060 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2175973}
I20240502 08:16:40.401298 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 262) took 2177 ms. Trace:
I20240502 08:16:40.401697 24179 rpcz_store.cc:276] 0502 08:16:38.224130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.224192 (+ 62us) service_pool.cc:225] Handling call
0502 08:16:40.401159 (+2176967us) ts_tablet_manager.cc:1120] Deleting tablet bd4a31e5aa834adeb9f59267247c6980
0502 08:16:40.401169 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.401251 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2176690}
I20240502 08:16:40.402406 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 263) took 2177 ms. Trace:
I20240502 08:16:40.402839 24179 rpcz_store.cc:276] 0502 08:16:38.224646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.224693 (+ 47us) service_pool.cc:225] Handling call
0502 08:16:40.402272 (+2177579us) ts_tablet_manager.cc:1120] Deleting tablet 98a96292b28a4b0aad88fe0a96730a68
0502 08:16:40.402279 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.402360 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2177242}
I20240502 08:16:40.403549 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 264) took 2178 ms. Trace:
I20240502 08:16:40.403998 24179 rpcz_store.cc:276] 0502 08:16:38.225185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.272878 (+ 47693us) service_pool.cc:225] Handling call
0502 08:16:40.403394 (+2130516us) ts_tablet_manager.cc:1120] Deleting tablet b742c51f6204449a804889815604d5cb
0502 08:16:40.403401 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.403493 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2130228}
I20240502 08:16:40.404733 24179 tablet_replica.cc:330] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:40.405184 24179 raft_consensus.cc:2229] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.405808 24179 raft_consensus.cc:2258] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.406558 24179 ts_tablet_manager.cc:1875] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.412607 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 0f306a288b90413b9d8ea8b76d23363a (DEFAULT_TABLE table=default.Table_9bb2be9b511a49eca4228b582c5ac739 [id=97c877d344bd44d896a989e63768579b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.412598 24030 tablet_service.cc:1467] Processing CreateTablet for tablet fddac683eda14ec1a0140926380f2c07 (DEFAULT_TABLE table=default.Table_4b967ef2b2fc4e2caa4219afcae841a6 [id=c32fc96889554c32bdf1d4550633e91f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.412654 23899 tablet_service.cc:1467] Processing CreateTablet for tablet fba8f6e017e848d59239ae98432b1837 (DEFAULT_TABLE table=default.Table_63545ead8a7a4aae8b2daa7af0fe5113 [id=3deaa06a1e88491f9dda6abda2cbdd9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.413075 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fddac683eda14ec1a0140926380f2c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.413357 23898 tablet_service.cc:1467] Processing CreateTablet for tablet fddac683eda14ec1a0140926380f2c07 (DEFAULT_TABLE table=default.Table_4b967ef2b2fc4e2caa4219afcae841a6 [id=c32fc96889554c32bdf1d4550633e91f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.413878 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f306a288b90413b9d8ea8b76d23363a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.414340 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba8f6e017e848d59239ae98432b1837. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.420356 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fddac683eda14ec1a0140926380f2c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.421332 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 0f306a288b90413b9d8ea8b76d23363a (DEFAULT_TABLE table=default.Table_9bb2be9b511a49eca4228b582c5ac739 [id=97c877d344bd44d896a989e63768579b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.421731 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f306a288b90413b9d8ea8b76d23363a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.424742 24179 ts_tablet_manager.cc:1888] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.424865 24179 log.cc:1196] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/9b1eb762780c450bbaf380d17db564b6
I20240502 08:16:40.425242 24179 ts_tablet_manager.cc:1909] T 9b1eb762780c450bbaf380d17db564b6 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:40.426578 24175 tablet_bootstrap.cc:492] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:40.426946 24175 ts_tablet_manager.cc:1367] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.450s user 0.016s sys 0.011s
I20240502 08:16:40.427837 24175 raft_consensus.cc:348] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:40.428217 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet 6788ffd2eb2248c3b84e474de56c37a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.426748 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 265) took 2152 ms. Trace:
I20240502 08:16:40.429235 24175 raft_consensus.cc:374] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.429630 24175 raft_consensus.cc:729] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.429950 24175 consensus_queue.cc:260] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:40.433320 24175 ts_tablet_manager.cc:1398] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:16:40.434753 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet da4be1d381a64b7baeb1c60cb17d924a (table default.Table_1e7f2516f34b483d913cf5080a8febec [id=d5c976b2debc472fa1d7c28ae73ca32a]) successfully deleted
W20240502 08:16:40.435140 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 319fabf91458438ba5e40ab68b323055 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 319fabf91458438ba5e40ab68b323055 already in progress: creating tablet
I20240502 08:16:40.435776 24175 tablet_bootstrap.cc:492] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:40.435770 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 4a57ed188315493198e95cefb6907ab6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a57ed188315493198e95cefb6907ab6 already in progress: creating tablet
W20240502 08:16:40.435997 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet bd4a31e5aa834adeb9f59267247c6980 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd4a31e5aa834adeb9f59267247c6980 already in progress: creating tablet
I20240502 08:16:40.429395 24179 rpcz_store.cc:276] 0502 08:16:38.273331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.273375 (+ 44us) service_pool.cc:225] Handling call
0502 08:16:40.404650 (+2131275us) ts_tablet_manager.cc:1120] Deleting tablet 9b1eb762780c450bbaf380d17db564b6
0502 08:16:40.404659 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.414790 (+ 10131us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.424606 (+ 9816us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.425565 (+ 959us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2130988}
W20240502 08:16:40.436221 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 98a96292b28a4b0aad88fe0a96730a68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98a96292b28a4b0aad88fe0a96730a68 already in progress: creating tablet
W20240502 08:16:40.436414 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet b742c51f6204449a804889815604d5cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b742c51f6204449a804889815604d5cb already in progress: creating tablet
I20240502 08:16:40.436578 24179 tablet_replica.cc:330] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:40.436733 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 9b1eb762780c450bbaf380d17db564b6 (table default.Table_4dd22967bcf147388ea4ceb9cd24b065 [id=ba56850714c2422eb5dbbc0f49df7ce7]) successfully deleted
I20240502 08:16:40.436861 24179 raft_consensus.cc:2229] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.437069 24179 raft_consensus.cc:2258] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.437776 24179 ts_tablet_manager.cc:1875] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.438668 23602 catalog_manager.cc:5484] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:40.439218 23602 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 43e870c3a0e340d1bad1c1fa0380a762 (table default.Table_299ae144ccc6402abe33d07ef693e5b4 [id=7e0c798bc77a4770a4bbe8658f181ffa]). Expected version 3 got 0
I20240502 08:16:40.439496 23602 catalog_manager.cc:5484] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:40.439915 23602 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet b6c52f1cae3a4ad38a9f95bfaf055e76 (table default.Table_35b808b0487f463dbcfa4a773a2a20f0 [id=922fb979b5894023a68820aece0b8c5e]). Expected version 1 got 0
I20240502 08:16:40.440187 23602 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 23ce83c8001d4b2988b88c0ed9bf948c (table default.Table_190eae2a8c64440f8d19842f8b6ac254 [id=54454efbe9394170822f237dc8e6115e]). Expected version 1 got 0
I20240502 08:16:40.440423 23602 catalog_manager.cc:5484] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:40.441748 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 0f306a288b90413b9d8ea8b76d23363a (DEFAULT_TABLE table=default.Table_9bb2be9b511a49eca4228b582c5ac739 [id=97c877d344bd44d896a989e63768579b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.442315 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f306a288b90413b9d8ea8b76d23363a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.444252 24040 tablet_service.cc:1467] Processing CreateTablet for tablet b0cf1a6de3d649b984c50f45393ec0f1 (DEFAULT_TABLE table=default.Table_d416b40fe3d24dafa8db5ee61d13acc0 [id=fa5100e581a7430ab8f0c19b2daae994]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.444494 23891 tablet_service.cc:1467] Processing CreateTablet for tablet b0cf1a6de3d649b984c50f45393ec0f1 (DEFAULT_TABLE table=default.Table_d416b40fe3d24dafa8db5ee61d13acc0 [id=fa5100e581a7430ab8f0c19b2daae994]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.444862 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0cf1a6de3d649b984c50f45393ec0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.444986 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0cf1a6de3d649b984c50f45393ec0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.445098 23756 tablet_service.cc:1467] Processing CreateTablet for tablet b0cf1a6de3d649b984c50f45393ec0f1 (DEFAULT_TABLE table=default.Table_d416b40fe3d24dafa8db5ee61d13acc0 [id=fa5100e581a7430ab8f0c19b2daae994]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.443257 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 0281c8123eab4c48bddc377f5695c6e0 (DEFAULT_TABLE table=default.Table_827517f3c36f43e585faba823c6d068d [id=d1a317371a5b43f185eb6453861e112a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.445500 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0cf1a6de3d649b984c50f45393ec0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.453274 24180 ts_tablet_manager.cc:1888] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:16:40.453369 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 0281c8123eab4c48bddc377f5695c6e0 (DEFAULT_TABLE table=default.Table_827517f3c36f43e585faba823c6d068d [id=d1a317371a5b43f185eb6453861e112a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.453516 24180 log.cc:1196] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/08183523bf4a4d1c9020eb693aa07878
I20240502 08:16:40.453897 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0281c8123eab4c48bddc377f5695c6e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.454102 24180 ts_tablet_manager.cc:1909] T 08183523bf4a4d1c9020eb693aa07878 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:40.454846 24175 tablet_bootstrap.cc:654] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.455490 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 08183523bf4a4d1c9020eb693aa07878 (table default.Table_c72b83e8da844cb6a570c14f50293f34 [id=0a2f8dbb62a64b0a96931ab5f8c17540]) successfully deleted
I20240502 08:16:40.455950 24173 tablet_bootstrap.cc:492] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:40.456178 24173 ts_tablet_manager.cc:1367] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.166s user 0.002s sys 0.002s
I20240502 08:16:40.456820 23749 tablet_service.cc:1467] Processing CreateTablet for tablet fddac683eda14ec1a0140926380f2c07 (DEFAULT_TABLE table=default.Table_4b967ef2b2fc4e2caa4219afcae841a6 [id=c32fc96889554c32bdf1d4550633e91f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.457269 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fddac683eda14ec1a0140926380f2c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.457259 24173 raft_consensus.cc:348] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.457485 24173 raft_consensus.cc:374] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.457593 24173 raft_consensus.cc:729] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.457908 24173 consensus_queue.cc:260] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.459947 24173 ts_tablet_manager.cc:1398] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:16:40.460664 24173 tablet_bootstrap.cc:492] T 0be4f8d5cc09483d8d1aa1e37679b544 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:40.461751 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 87c36b622cd94f23b6e7ebdd2ab5b6b3 (DEFAULT_TABLE table=default.Table_e8b4ffe8cc7942d89f906ce2a0f524f3 [id=8338775ae95b48a29b9e9686600aab25]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.462319 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c36b622cd94f23b6e7ebdd2ab5b6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.464864 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0281c8123eab4c48bddc377f5695c6e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.471331 24173 tablet_bootstrap.cc:654] T 0be4f8d5cc09483d8d1aa1e37679b544 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.474000 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 0281c8123eab4c48bddc377f5695c6e0 (DEFAULT_TABLE table=default.Table_827517f3c36f43e585faba823c6d068d [id=d1a317371a5b43f185eb6453861e112a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.474495 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0281c8123eab4c48bddc377f5695c6e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.476347 24180 tablet_replica.cc:330] T f51b26043ef44b30846498cd1497607c P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:40.480233 24180 raft_consensus.cc:2229] T f51b26043ef44b30846498cd1497607c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.481639 24180 raft_consensus.cc:2258] T f51b26043ef44b30846498cd1497607c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.482321 24180 ts_tablet_manager.cc:1875] T f51b26043ef44b30846498cd1497607c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.494724 24175 tablet_bootstrap.cc:492] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:40.494921 24175 ts_tablet_manager.cc:1367] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.059s user 0.006s sys 0.002s
I20240502 08:16:40.481251 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 87c36b622cd94f23b6e7ebdd2ab5b6b3 (DEFAULT_TABLE table=default.Table_e8b4ffe8cc7942d89f906ce2a0f524f3 [id=8338775ae95b48a29b9e9686600aab25]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.495931 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c36b622cd94f23b6e7ebdd2ab5b6b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.496258 24174 tablet_bootstrap.cc:492] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:40.496471 24174 ts_tablet_manager.cc:1367] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.400s user 0.004s sys 0.000s
I20240502 08:16:40.497412 24174 raft_consensus.cc:348] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.497742 24174 raft_consensus.cc:374] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.497910 24174 raft_consensus.cc:729] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.498276 24174 consensus_queue.cc:260] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.500660 24174 ts_tablet_manager.cc:1398] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:40.501055 24174 tablet_bootstrap.cc:492] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:40.504055 24175 raft_consensus.cc:348] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.507715 24179 ts_tablet_manager.cc:1888] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.508343 24179 log.cc:1196] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/485c772245fe48a39fc6ceb6377ac0b6
I20240502 08:16:40.508805 24179 ts_tablet_manager.cc:1909] T 485c772245fe48a39fc6ceb6377ac0b6 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:40.508915 24175 raft_consensus.cc:374] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.509584 24175 raft_consensus.cc:729] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.509871 24175 consensus_queue.cc:260] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.511363 24174 tablet_bootstrap.cc:654] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.518369 24175 ts_tablet_manager.cc:1398] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240502 08:16:40.518941 24175 tablet_bootstrap.cc:492] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:40.524395 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 18625a1d971c43549b9f80e6046efc9c consensus state has no leader
I20240502 08:16:40.525889 23606 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet e134e90efb4a48bb82b53a80bda708b3 (table default.Table_cc80d1cedafa4e71906e5e9530fdd828 [id=47682c26a85842c7b8991bf3828a9336]). Expected version 1 got 0
I20240502 08:16:40.526805 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ca3dbc715088456686fa2e20e8c5b3b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:40.527402 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca3dbc715088456686fa2e20e8c5b3b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:40.528743 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 87c36b622cd94f23b6e7ebdd2ab5b6b3 (DEFAULT_TABLE table=default.Table_e8b4ffe8cc7942d89f906ce2a0f524f3 [id=8338775ae95b48a29b9e9686600aab25]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.529229 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_55ca9dedb821436ea68a7980e085adb9" } new_table_name: "default.Table_70eb03c32f654f229093f610d22f9972" modify_external_catalogs: true
I20240502 08:16:40.529461 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c36b622cd94f23b6e7ebdd2ab5b6b3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:40.530561 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0501575a84ec454780f48574bf3155fe consensus state has no leader
I20240502 08:16:40.530894 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5b51e3b192594b119d6719a685fe3513" } new_table_name: "default.Table_92a3c7cbfc16461ba5b08f262bc016d3" modify_external_catalogs: true
I20240502 08:16:40.532509 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 1ba1baa2279045b3adc58d187a12cbf3
I20240502 08:16:40.533407 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 78ddfc7b74644a0bbf8222d5f90e6658
I20240502 08:16:40.535699 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet 78ddfc7b74644a0bbf8222d5f90e6658 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:40.537614 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ba1baa2279045b3adc58d187a12cbf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:40.539832 23886 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ba1baa2279045b3adc58d187a12cbf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:40.541733 23888 tablet_service.cc:1514] Processing DeleteTablet for tablet a7c3800fb2784a1dba12d01c56c226e5 with delete_type TABLET_DATA_DELETED (replaced by tablet f4b291bdb81a4efbac7f6691e5577bbe) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:40.542593 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 268) took 2240 ms. Trace:
I20240502 08:16:40.542703 24179 rpcz_store.cc:276] 0502 08:16:38.282368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.282415 (+ 47us) service_pool.cc:225] Handling call
0502 08:16:40.436462 (+2154047us) ts_tablet_manager.cc:1120] Deleting tablet 485c772245fe48a39fc6ceb6377ac0b6
0502 08:16:40.436470 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.467687 (+ 31217us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.507583 (+ 39896us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.522839 (+ 15256us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2094375}
I20240502 08:16:40.543448 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ba1baa2279045b3adc58d187a12cbf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.543687 24179 tablet_replica.cc:330] T c709efdd2ce74b5a8ab7278b402ebfe9 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:40.543910 24179 raft_consensus.cc:2229] T c709efdd2ce74b5a8ab7278b402ebfe9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.544106 24179 raft_consensus.cc:2258] T c709efdd2ce74b5a8ab7278b402ebfe9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.544668 24179 ts_tablet_manager.cc:1875] T c709efdd2ce74b5a8ab7278b402ebfe9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.545015 24551 raft_consensus.cc:482] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:40.545166 24551 raft_consensus.cc:504] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:40.545881 24551 leader_election.cc:288] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
W20240502 08:16:40.546635 23861 leader_election.cc:341] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: a7c3800fb2784a1dba12d01c56c226e5
W20240502 08:16:40.546761 23858 leader_election.cc:341] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: a7c3800fb2784a1dba12d01c56c226e5
I20240502 08:16:40.546875 23858 leader_election.cc:302] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:40.547282 24551 raft_consensus.cc:2726] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:40.549211 24175 tablet_bootstrap.cc:654] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.549245 23886 tablet_service.cc:1514] Processing DeleteTablet for tablet 78ddfc7b74644a0bbf8222d5f90e6658 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:40.550388 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 485c772245fe48a39fc6ceb6377ac0b6 (table default.Table_0528f5d019fe489db8bed1f71adac3cb [id=7dd2b25a6676488d8f3a8e2e6b65046e]) successfully deleted
I20240502 08:16:40.554167 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1e7f2516f34b483d913cf5080a8febec" } modify_external_catalogs: true
I20240502 08:16:40.554390 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1e7f2516f34b483d913cf5080a8febec" } modify_external_catalogs: true
I20240502 08:16:40.556146 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7202b499dd484f6196154d3bc98435fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:40.556769 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7202b499dd484f6196154d3bc98435fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:40.574321 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5fa34ef38ad04a649c1fb10b958e99e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:40.574791 24179 ts_tablet_manager.cc:1888] T c709efdd2ce74b5a8ab7278b402ebfe9 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.574927 24179 log.cc:1196] T c709efdd2ce74b5a8ab7278b402ebfe9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/c709efdd2ce74b5a8ab7278b402ebfe9
W20240502 08:16:40.574939 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fa34ef38ad04a649c1fb10b958e99e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:40.575307 24179 ts_tablet_manager.cc:1909] T c709efdd2ce74b5a8ab7278b402ebfe9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:40.576320 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e87bd15719d642bfbc0c9ab785ff62d1" } new_table_name: "default.Table_646fc9f7ffec46cf999a349a70c20fe7" modify_external_catalogs: true
I20240502 08:16:40.569515 24180 ts_tablet_manager.cc:1888] T f51b26043ef44b30846498cd1497607c P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.581518 24180 log.cc:1196] T f51b26043ef44b30846498cd1497607c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/f51b26043ef44b30846498cd1497607c
I20240502 08:16:40.582368 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet b742c51f6204449a804889815604d5cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:40.586167 24180 ts_tablet_manager.cc:1909] T f51b26043ef44b30846498cd1497607c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:40.588649 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 270) took 2305 ms. Trace:
I20240502 08:16:40.589892 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet f51b26043ef44b30846498cd1497607c (table default.Table_da5e259b0d0c47cf94ab30e3e98d1540 [id=956913447d9b4f55920aa4382692d608]) successfully deleted
I20240502 08:16:40.590517 24180 tablet_replica.cc:330] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:40.590806 24179 rpcz_store.cc:276] 0502 08:16:38.283179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.284975 (+ 1796us) service_pool.cc:225] Handling call
0502 08:16:40.543602 (+2258627us) ts_tablet_manager.cc:1120] Deleting tablet c709efdd2ce74b5a8ab7278b402ebfe9
0502 08:16:40.543613 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.564522 (+ 20909us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.574647 (+ 10125us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.588607 (+ 13960us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2197799}
I20240502 08:16:40.591440 24180 raft_consensus.cc:2229] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.592185 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet c709efdd2ce74b5a8ab7278b402ebfe9 (table default.Table_34b1b262bb424bde8bd994ada1f03434 [id=322a04a326c24fc4ad2c3f1384db0fd3]) successfully deleted
I20240502 08:16:40.592290 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 271) took 2232 ms. Trace:
I20240502 08:16:40.592411 24179 rpcz_store.cc:276] 0502 08:16:38.361428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.362137 (+ 709us) service_pool.cc:225] Handling call
0502 08:16:40.592122 (+2229985us) ts_tablet_manager.cc:1120] Deleting tablet b1fac7df16fc41d7a3297055a7258270
0502 08:16:40.592135 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.592254 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2229712}
I20240502 08:16:40.592886 24180 raft_consensus.cc:2258] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.593784 24180 ts_tablet_manager.cc:1875] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.594918 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 272) took 2233 ms. Trace:
W20240502 08:16:40.599351 23596 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 18625a1d971c43549b9f80e6046efc9c (table default.Table_e02044df8b8c4ca9a9bdccaaf14a2eb4 [id=62ad2ebd7db541958a77f09c07ab946e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 18625a1d971c43549b9f80e6046efc9c consensus state has no leader
I20240502 08:16:40.597973 24174 tablet_bootstrap.cc:492] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:40.604314 24174 ts_tablet_manager.cc:1367] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.103s user 0.004s sys 0.000s
I20240502 08:16:40.604458 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_299ae144ccc6402abe33d07ef693e5b4" } modify_external_catalogs: true
I20240502 08:16:40.604662 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_299ae144ccc6402abe33d07ef693e5b4" } modify_external_catalogs: true
I20240502 08:16:40.605371 24174 raft_consensus.cc:348] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:40.605624 24174 raft_consensus.cc:374] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.605708 24174 raft_consensus.cc:729] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.606526 23609 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet e134e90efb4a48bb82b53a80bda708b3 (table default.Table_cc80d1cedafa4e71906e5e9530fdd828 [id=47682c26a85842c7b8991bf3828a9336]). Expected version 1 got 0
I20240502 08:16:40.603955 24551 raft_consensus.cc:482] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:40.607067 24551 raft_consensus.cc:504] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.607292 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet f36aadfa1d264f23bcafa91744aed36d from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:40.607810 24551 leader_election.cc:288] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:40.606902 24175 tablet_bootstrap.cc:492] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:40.608388 24175 ts_tablet_manager.cc:1367] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.090s user 0.001s sys 0.010s
I20240502 08:16:40.609223 24175 raft_consensus.cc:348] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:40.609427 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ae9c817c7be42da8c689267c7eb5a06 has no consensus state
I20240502 08:16:40.609485 24175 raft_consensus.cc:374] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.609591 24175 raft_consensus.cc:729] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
W20240502 08:16:40.609851 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 63e645bb6e2948e091e08433944a52f4 consensus state has no leader
I20240502 08:16:40.609869 24175 consensus_queue.cc:260] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.610109 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet 0dff0d73a933427fb1f2686bac33eba1 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:40.610495 24175 ts_tablet_manager.cc:1398] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:40.610527 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet c00f29fb76b44d6ea4f76963d2d0751c from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:40.610782 24175 tablet_bootstrap.cc:492] T 23ce83c8001d4b2988b88c0ed9bf948c P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:40.612452 23858 leader_election.cc:341] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: b742c51f6204449a804889815604d5cb
I20240502 08:16:40.612669 24179 rpcz_store.cc:276] 0502 08:16:38.361600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.380647 (+ 19047us) service_pool.cc:225] Handling call
0502 08:16:40.594731 (+2214084us) ts_tablet_manager.cc:1120] Deleting tablet dcdd765895254a39b86b692b5d812c83
0502 08:16:40.594742 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.594884 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2213875}
I20240502 08:16:40.613056 24174 consensus_queue.cc:260] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
W20240502 08:16:40.614768 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet dcdd765895254a39b86b692b5d812c83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dcdd765895254a39b86b692b5d812c83 already in progress: creating tablet
I20240502 08:16:40.614831 24174 ts_tablet_manager.cc:1398] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 08:16:40.615152 24174 tablet_bootstrap.cc:492] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:40.615732 24179 tablet_replica.cc:330] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:40.615932 24179 raft_consensus.cc:2229] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:40.616353 24179 pending_rounds.cc:62] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b: Trying to abort 1 pending ops.
I20240502 08:16:40.616452 24179 pending_rounds.cc:68] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156425238749184 op_type: NO_OP noop_request { }
I20240502 08:16:40.616585 24179 raft_consensus.cc:2866] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:40.616730 24179 raft_consensus.cc:2258] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.617535 24175 tablet_bootstrap.cc:654] T 23ce83c8001d4b2988b88c0ed9bf948c P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.621052 23607 master_service.cc:946] ReplaceTablet: received request to replace tablet 18d284f71937459db0c0676ce4568175 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:40.621644 24179 ts_tablet_manager.cc:1875] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.621640 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2ecca22d4d934ddbacb4eb1826759e48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:40.622161 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ecca22d4d934ddbacb4eb1826759e48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:40.623503 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_498b372c15a04b70b87bd67b543501cf" } modify_external_catalogs: true
I20240502 08:16:40.623663 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_498b372c15a04b70b87bd67b543501cf" } modify_external_catalogs: true
I20240502 08:16:40.625890 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet f51b26043ef44b30846498cd1497607c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:40.626416 23861 leader_election.cc:341] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: b742c51f6204449a804889815604d5cb
I20240502 08:16:40.626562 23861 leader_election.cc:302] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:40.626807 24551 raft_consensus.cc:2726] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:40.627957 24174 tablet_bootstrap.cc:654] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.631608 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet 78ddfc7b74644a0bbf8222d5f90e6658 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.632786 24181 ts_tablet_manager.cc:1888] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.633033 24181 log.cc:1196] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/6bb2d0a4898544f182ab8070d29a7940
I20240502 08:16:40.633491 24181 ts_tablet_manager.cc:1909] T 6bb2d0a4898544f182ab8070d29a7940 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:40.634424 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 342) took 1497 ms. Trace:
I20240502 08:16:40.634581 24181 rpcz_store.cc:276] 0502 08:16:39.136763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.426787 (+290024us) service_pool.cc:225] Handling call
0502 08:16:40.379195 (+952408us) ts_tablet_manager.cc:1120] Deleting tablet 6bb2d0a4898544f182ab8070d29a7940
0502 08:16:40.379206 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.431582 (+ 52376us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.631191 (+199609us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.634337 (+ 3146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":952255}
I20240502 08:16:40.635162 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 6bb2d0a4898544f182ab8070d29a7940 (table default.Table_482a1b0b7a064e02805ac3b86fc62f99 [id=3c1d614616374739aeb928e797479df4]) successfully deleted
I20240502 08:16:40.635706 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 340) took 1630 ms. Trace:
I20240502 08:16:40.636086 24180 ts_tablet_manager.cc:1888] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.636224 24180 log.cc:1196] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/8633864e248f416cb651da3b0c2aa936
I20240502 08:16:40.636588 24596 raft_consensus.cc:482] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:40.636744 24596 raft_consensus.cc:504] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.637342 24596 leader_election.cc:288] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:40.637851 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cc30014a1ac45de937751eed36877a9" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:40.638123 23723 leader_election.cc:341] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 3cc30014a1ac45de937751eed36877a9
W20240502 08:16:40.638345 23723 leader_election.cc:341] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:40.638463 23723 leader_election.cc:302] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:40.638988 24596 raft_consensus.cc:2726] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:40.640501 24174 tablet_bootstrap.cc:492] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:40.640640 24174 ts_tablet_manager.cc:1367] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.001s
I20240502 08:16:40.641263 24181 rpcz_store.cc:276] 0502 08:16:39.005493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.418168 (+412675us) service_pool.cc:225] Handling call
0502 08:16:40.635522 (+1217354us) ts_tablet_manager.cc:1120] Deleting tablet 9c03a4353c4e46e3bcdf3a36aac04bf5
0502 08:16:40.635532 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.635658 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1198282}
I20240502 08:16:40.641391 24174 raft_consensus.cc:348] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.641598 24174 raft_consensus.cc:374] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.641690 24174 raft_consensus.cc:729] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
W20240502 08:16:40.641873 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 already in progress: creating tablet
I20240502 08:16:40.641893 24174 consensus_queue.cc:260] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.642380 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 364) took 1302 ms. Trace:
I20240502 08:16:40.642499 24181 rpcz_store.cc:276] 0502 08:16:39.339945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.826486 (+486541us) service_pool.cc:225] Handling call
0502 08:16:40.642236 (+815750us) ts_tablet_manager.cc:1120] Deleting tablet 090bec30f5664421b9bfe3f2810afd14
0502 08:16:40.642244 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.642335 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":812422}
I20240502 08:16:40.640102 24173 tablet_bootstrap.cc:492] T 0be4f8d5cc09483d8d1aa1e37679b544 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:40.642828 24173 ts_tablet_manager.cc:1367] T 0be4f8d5cc09483d8d1aa1e37679b544 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.183s user 0.001s sys 0.003s
I20240502 08:16:40.643505 24173 raft_consensus.cc:348] T 0be4f8d5cc09483d8d1aa1e37679b544 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.643743 24173 raft_consensus.cc:374] T 0be4f8d5cc09483d8d1aa1e37679b544 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.643850 24173 raft_consensus.cc:729] T 0be4f8d5cc09483d8d1aa1e37679b544 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.644107 24173 consensus_queue.cc:260] T 0be4f8d5cc09483d8d1aa1e37679b544 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.636587 24180 ts_tablet_manager.cc:1909] T 8633864e248f416cb651da3b0c2aa936 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:40.649616 24175 tablet_bootstrap.cc:492] T 23ce83c8001d4b2988b88c0ed9bf948c P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:40.649765 24175 ts_tablet_manager.cc:1367] T 23ce83c8001d4b2988b88c0ed9bf948c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20240502 08:16:40.650790 24173 ts_tablet_manager.cc:1398] T 0be4f8d5cc09483d8d1aa1e37679b544 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 08:16:40.650758 24175 raft_consensus.cc:348] T 23ce83c8001d4b2988b88c0ed9bf948c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.650966 24175 raft_consensus.cc:374] T 23ce83c8001d4b2988b88c0ed9bf948c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.652045 24175 raft_consensus.cc:729] T 23ce83c8001d4b2988b88c0ed9bf948c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.652333 24175 consensus_queue.cc:260] T 23ce83c8001d4b2988b88c0ed9bf948c P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.653023 24175 ts_tablet_manager.cc:1398] T 23ce83c8001d4b2988b88c0ed9bf948c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:40.653307 24175 tablet_bootstrap.cc:492] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:40.658077 24173 tablet_bootstrap.cc:492] T 1065732d7333413d984ed56275fd61a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:40.659570 24174 ts_tablet_manager.cc:1398] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240502 08:16:40.666347 24174 tablet_bootstrap.cc:492] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:40.667141 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 8633864e248f416cb651da3b0c2aa936 (table default.Table_8ba240c391da43978f93a27dce48e557 [id=1b5536950db244d1978b154da0be33e0]) successfully deleted
I20240502 08:16:40.668990 24174 tablet_bootstrap.cc:654] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.669961 24175 tablet_bootstrap.cc:654] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:40.671205 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 78ddfc7b74644a0bbf8222d5f90e6658 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78ddfc7b74644a0bbf8222d5f90e6658 already in progress: creating tablet
W20240502 08:16:40.671494 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 1ba1baa2279045b3adc58d187a12cbf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ba1baa2279045b3adc58d187a12cbf3 already in progress: creating tablet
I20240502 08:16:40.672025 24174 tablet_bootstrap.cc:492] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:40.672173 24174 ts_tablet_manager.cc:1367] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:16:40.672808 24174 raft_consensus.cc:348] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:40.673043 24174 raft_consensus.cc:374] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.673110 24174 raft_consensus.cc:729] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.673316 24174 consensus_queue.cc:260] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:40.673755 24179 ts_tablet_manager.cc:1888] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:40.674261 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 365) took 1198 ms. Trace:
I20240502 08:16:40.674520 24179 log.cc:1196] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/1df565c1c6564efd88fa557c50522d7e
W20240502 08:16:40.674676 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 090bec30f5664421b9bfe3f2810afd14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 090bec30f5664421b9bfe3f2810afd14 already in progress: creating tablet
I20240502 08:16:40.675237 24179 ts_tablet_manager.cc:1909] T 1df565c1c6564efd88fa557c50522d7e P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:40.674779 24173 tablet_bootstrap.cc:654] T 1065732d7333413d984ed56275fd61a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.675580 24181 rpcz_store.cc:276] 0502 08:16:39.476002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.829842 (+353840us) service_pool.cc:225] Handling call
0502 08:16:40.674087 (+844245us) ts_tablet_manager.cc:1120] Deleting tablet 7a97427e819b4826b1ed89e6d021c8d4
0502 08:16:40.674098 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.674213 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":844081}
I20240502 08:16:40.676903 24175 tablet_bootstrap.cc:492] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:40.676892 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 363) took 1337 ms. Trace:
I20240502 08:16:40.676970 24181 rpcz_store.cc:276] 0502 08:16:39.339841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.821224 (+481383us) service_pool.cc:225] Handling call
0502 08:16:40.676747 (+855523us) ts_tablet_manager.cc:1120] Deleting tablet 128c88af54d24ffd895e9de19cdb4c8a
0502 08:16:40.676755 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.676847 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":843730}
I20240502 08:16:40.677059 24175 ts_tablet_manager.cc:1367] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240502 08:16:40.677292 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 367) took 1095 ms. Trace:
I20240502 08:16:40.677369 24181 rpcz_store.cc:276] 0502 08:16:39.581505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.833034 (+251529us) service_pool.cc:225] Handling call
0502 08:16:40.677160 (+844126us) ts_tablet_manager.cc:1120] Deleting tablet a3043c5ed98c401b82c6e967ca3f3613
0502 08:16:40.677169 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.677249 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":842545}
W20240502 08:16:40.677811 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 7a97427e819b4826b1ed89e6d021c8d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a97427e819b4826b1ed89e6d021c8d4 already in progress: creating tablet
I20240502 08:16:40.677891 24175 raft_consensus.cc:348] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.678078 24175 raft_consensus.cc:374] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 08:16:40.678061 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 128c88af54d24ffd895e9de19cdb4c8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 128c88af54d24ffd895e9de19cdb4c8a already in progress: creating tablet
I20240502 08:16:40.678148 24175 raft_consensus.cc:729] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
W20240502 08:16:40.678268 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet a3043c5ed98c401b82c6e967ca3f3613 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3043c5ed98c401b82c6e967ca3f3613 already in progress: creating tablet
I20240502 08:16:40.678296 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 368) took 1096 ms. Trace:
I20240502 08:16:40.678370 24181 rpcz_store.cc:276] 0502 08:16:39.581645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.834639 (+252994us) service_pool.cc:225] Handling call
0502 08:16:40.678167 (+843528us) ts_tablet_manager.cc:1120] Deleting tablet 5bcc7169796e4ee0910f5e7084849012
0502 08:16:40.678182 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.678255 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":842752}
I20240502 08:16:40.678380 24175 consensus_queue.cc:260] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.678155 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 273) took 2238 ms. Trace:
I20240502 08:16:40.678715 24179 rpcz_store.cc:276] 0502 08:16:38.439571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.439621 (+ 50us) service_pool.cc:225] Handling call
0502 08:16:40.613492 (+2173871us) ts_tablet_manager.cc:1120] Deleting tablet 1df565c1c6564efd88fa557c50522d7e
0502 08:16:40.615655 (+ 2163us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.624923 (+ 9268us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.648186 (+ 23263us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.678108 (+ 29922us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2173598}
W20240502 08:16:40.678855 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 5bcc7169796e4ee0910f5e7084849012 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bcc7169796e4ee0910f5e7084849012 already in progress: creating tablet
W20240502 08:16:40.679364 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 654ce33d1e3d472eb6ae6a726eb803bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 654ce33d1e3d472eb6ae6a726eb803bb already in progress: creating tablet
W20240502 08:16:40.679589 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet bcfacc302ff04178bfffc48111cd7a3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcfacc302ff04178bfffc48111cd7a3f already in progress: creating tablet
I20240502 08:16:40.679705 24179 tablet_replica.cc:330] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
W20240502 08:16:40.679790 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet ee022e8e86f8477a8fa1e58d52644b03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee022e8e86f8477a8fa1e58d52644b03 already in progress: creating tablet
I20240502 08:16:40.679908 24179 raft_consensus.cc:2229] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.679991 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 1df565c1c6564efd88fa557c50522d7e (table default.Table_2107771cb030402280a54f03d8b2bce4 [id=0a35939b821f468c9c7c86425fd688a5]) successfully deleted
I20240502 08:16:40.680065 24179 raft_consensus.cc:2258] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.679349 24175 ts_tablet_manager.cc:1398] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:40.681411 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 366) took 1194 ms. Trace:
I20240502 08:16:40.681505 24181 rpcz_store.cc:276] 0502 08:16:39.486515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.830017 (+343502us) service_pool.cc:225] Handling call
0502 08:16:40.681274 (+851257us) ts_tablet_manager.cc:1120] Deleting tablet df9402cf8abc4b4d893d42f9a3a99016
0502 08:16:40.681283 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.681353 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":840200}
I20240502 08:16:40.683923 24175 tablet_bootstrap.cc:492] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:40.685792 24181 tablet_replica.cc:330] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:40.686031 24181 raft_consensus.cc:2229] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.686316 24179 ts_tablet_manager.cc:1875] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.686457 24181 raft_consensus.cc:2258] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.686789 24174 ts_tablet_manager.cc:1398] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 08:16:40.687291 24181 ts_tablet_manager.cc:1875] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:40.690025 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet df9402cf8abc4b4d893d42f9a3a99016 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df9402cf8abc4b4d893d42f9a3a99016 already in progress: creating tablet
W20240502 08:16:40.690272 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 4b8d81f071d74b218fcfd4f4e8c20bab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b8d81f071d74b218fcfd4f4e8c20bab already in progress: creating tablet
W20240502 08:16:40.690462 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 751808dfc1964b539f6f836d786156aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 751808dfc1964b539f6f836d786156aa already in progress: creating tablet
W20240502 08:16:40.690644 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet aacdf506221044ffada0fec0c7d42d0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aacdf506221044ffada0fec0c7d42d0b already in progress: creating tablet
W20240502 08:16:40.690807 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 25c089f0c1b14e419f250d03d4eaad12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25c089f0c1b14e419f250d03d4eaad12 already in progress: creating tablet
W20240502 08:16:40.690996 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet eb96ad2289754e428a1c109b2d59c53b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb96ad2289754e428a1c109b2d59c53b already in progress: creating tablet
W20240502 08:16:40.691147 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 24a4104f4cb64889ad66810788ad6b29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24a4104f4cb64889ad66810788ad6b29 already in progress: creating tablet
W20240502 08:16:40.691334 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 9c072eb45e1b41eeb5aafa6d5a786944 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c072eb45e1b41eeb5aafa6d5a786944 already in progress: creating tablet
I20240502 08:16:40.692592 24179 ts_tablet_manager.cc:1888] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.692734 24179 log.cc:1196] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/ba3d4b4584dd45b58ff0daa4f1c09a0a
I20240502 08:16:40.693099 24179 ts_tablet_manager.cc:1909] T ba3d4b4584dd45b58ff0daa4f1c09a0a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:40.693161 24181 ts_tablet_manager.cc:1888] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:16:40.693262 24181 log.cc:1196] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/08183523bf4a4d1c9020eb693aa07878
I20240502 08:16:40.693706 24181 ts_tablet_manager.cc:1909] T 08183523bf4a4d1c9020eb693aa07878 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:40.694320 24175 tablet_bootstrap.cc:654] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.694618 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet fbef606ce7ac499d9bd84b49d804621c
I20240502 08:16:40.694924 24181 tablet_replica.cc:330] T ce758c4a9258427987e8e83f579a2aa2 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:40.695122 24181 raft_consensus.cc:2229] T ce758c4a9258427987e8e83f579a2aa2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.695700 24181 raft_consensus.cc:2258] T ce758c4a9258427987e8e83f579a2aa2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.696206 24181 ts_tablet_manager.cc:1875] T ce758c4a9258427987e8e83f579a2aa2 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.697157 24174 tablet_bootstrap.cc:492] T b11f504b93574e42a9bf30fde6d08207 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:40.698217 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet fbef606ce7ac499d9bd84b49d804621c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:40.699141 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet fbef606ce7ac499d9bd84b49d804621c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:40.699580 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet fbef606ce7ac499d9bd84b49d804621c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbef606ce7ac499d9bd84b49d804621c already in progress: creating tablet
I20240502 08:16:40.700081 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 08183523bf4a4d1c9020eb693aa07878 (table default.Table_c72b83e8da844cb6a570c14f50293f34 [id=0a2f8dbb62a64b0a96931ab5f8c17540]) successfully deleted
I20240502 08:16:40.700429 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet fbef606ce7ac499d9bd84b49d804621c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.701277 24174 tablet_bootstrap.cc:654] T b11f504b93574e42a9bf30fde6d08207 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.705193 24181 ts_tablet_manager.cc:1888] T ce758c4a9258427987e8e83f579a2aa2 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.705389 24181 log.cc:1196] T ce758c4a9258427987e8e83f579a2aa2 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/ce758c4a9258427987e8e83f579a2aa2
I20240502 08:16:40.705785 24181 ts_tablet_manager.cc:1909] T ce758c4a9258427987e8e83f579a2aa2 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:40.706704 24181 tablet_replica.cc:330] T 6788ffd2eb2248c3b84e474de56c37a9 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:40.706925 24181 raft_consensus.cc:2229] T 6788ffd2eb2248c3b84e474de56c37a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.707139 24181 raft_consensus.cc:2258] T 6788ffd2eb2248c3b84e474de56c37a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.709196 24181 ts_tablet_manager.cc:1875] T 6788ffd2eb2248c3b84e474de56c37a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.710461 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet ce758c4a9258427987e8e83f579a2aa2 (table default.Table_cb0ea06f5e0b47e5978fc9a5ed8880ce [id=9965057f98614e8284d007156af6fdd0]) successfully deleted
I20240502 08:16:40.710839 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 0be4f8d5cc09483d8d1aa1e37679b544 with delete_type TABLET_DATA_DELETED (replaced by tablet e686db13bb7541c69ae88f0fb7e27061) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.711027 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 275) took 2271 ms. Trace:
I20240502 08:16:40.712464 24179 rpcz_store.cc:276] 0502 08:16:38.439802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.439863 (+ 61us) service_pool.cc:225] Handling call
0502 08:16:40.679634 (+2239771us) ts_tablet_manager.cc:1120] Deleting tablet ba3d4b4584dd45b58ff0daa4f1c09a0a
0502 08:16:40.679644 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.689865 (+ 10221us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.692502 (+ 2637us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.710951 (+ 18449us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2239531}
I20240502 08:16:40.714000 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 274) took 2274 ms. Trace:
I20240502 08:16:40.715083 24179 rpcz_store.cc:276] 0502 08:16:38.439693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.440125 (+ 432us) service_pool.cc:225] Handling call
0502 08:16:40.713778 (+2273653us) ts_tablet_manager.cc:1120] Deleting tablet c5d28098dcbc4761a34ee65769d6fd7c
0502 08:16:40.713791 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.713942 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2273231}
I20240502 08:16:40.715818 24181 ts_tablet_manager.cc:1888] T 6788ffd2eb2248c3b84e474de56c37a9 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.716007 24181 log.cc:1196] T 6788ffd2eb2248c3b84e474de56c37a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/6788ffd2eb2248c3b84e474de56c37a9
I20240502 08:16:40.716158 24179 tablet_replica.cc:330] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:40.717348 24179 raft_consensus.cc:2229] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.717636 24179 raft_consensus.cc:2258] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.718052 24181 ts_tablet_manager.cc:1909] T 6788ffd2eb2248c3b84e474de56c37a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:40.718623 24179 ts_tablet_manager.cc:1875] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.720201 24181 tablet_replica.cc:330] T 0be4f8d5cc09483d8d1aa1e37679b544 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:40.720569 24181 raft_consensus.cc:2229] T 0be4f8d5cc09483d8d1aa1e37679b544 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.720763 24181 raft_consensus.cc:2258] T 0be4f8d5cc09483d8d1aa1e37679b544 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.720824 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 6788ffd2eb2248c3b84e474de56c37a9 (table default.Table_59462d2ccf514f0da864562120217931 [id=62273209104a4a8e9f24a9a77d2a90f0]) successfully deleted
I20240502 08:16:40.721449 24181 ts_tablet_manager.cc:1875] T 0be4f8d5cc09483d8d1aa1e37679b544 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:40.721693 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 1ba1baa2279045b3adc58d187a12cbf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ba1baa2279045b3adc58d187a12cbf3 already in progress: creating tablet
W20240502 08:16:40.722092 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 78ddfc7b74644a0bbf8222d5f90e6658 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78ddfc7b74644a0bbf8222d5f90e6658 already in progress: creating tablet
W20240502 08:16:40.722465 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet fbef606ce7ac499d9bd84b49d804621c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbef606ce7ac499d9bd84b49d804621c already in progress: creating tablet
I20240502 08:16:40.722728 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet ba3d4b4584dd45b58ff0daa4f1c09a0a (table default.Table_4e40fa1cf0ab4dedb2c6aadb2418901b [id=fd781805eb4c45f09fe6460144472816]) successfully deleted
W20240502 08:16:40.723047 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet c5d28098dcbc4761a34ee65769d6fd7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5d28098dcbc4761a34ee65769d6fd7c already in progress: creating tablet
I20240502 08:16:40.729110 24181 ts_tablet_manager.cc:1888] T 0be4f8d5cc09483d8d1aa1e37679b544 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.729336 24181 log.cc:1196] T 0be4f8d5cc09483d8d1aa1e37679b544 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/0be4f8d5cc09483d8d1aa1e37679b544
I20240502 08:16:40.729589 24179 ts_tablet_manager.cc:1888] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.729802 24181 ts_tablet_manager.cc:1909] T 0be4f8d5cc09483d8d1aa1e37679b544 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:40.729874 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_586d3e878d614a3da2f2988f6ad484d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:40.730041 24179 log.cc:1196] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/e9592a041a9543e8b41f043927807131
I20240502 08:16:40.730392 24179 ts_tablet_manager.cc:1909] T e9592a041a9543e8b41f043927807131 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
W20240502 08:16:40.730517 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_586d3e878d614a3da2f2988f6ad484d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:40.731268 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 276) took 2159 ms. Trace:
I20240502 08:16:40.731406 24179 rpcz_store.cc:276] 0502 08:16:38.572253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.572313 (+ 60us) service_pool.cc:225] Handling call
0502 08:16:40.716079 (+2143766us) ts_tablet_manager.cc:1120] Deleting tablet e9592a041a9543e8b41f043927807131
0502 08:16:40.716088 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.726542 (+ 10454us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.729438 (+ 2896us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.731230 (+ 1792us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":528,"tablet-delete.queue_time_us":2143527}
I20240502 08:16:40.731890 24179 tablet_replica.cc:330] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:40.732120 24179 raft_consensus.cc:2229] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.732295 24179 raft_consensus.cc:2258] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.732254 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet e9592a041a9543e8b41f043927807131 (table default.Table_6c584db923e147d380281ecb1acd11a1 [id=467a7f6db73641ca85a1898b251938fb]) successfully deleted
I20240502 08:16:40.732540 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 0be4f8d5cc09483d8d1aa1e37679b544 (table default.Table_178527d31d7044fc98fa41886d651049 [id=6dc275aed42b42118a70099d3eae3c21]) successfully deleted
I20240502 08:16:40.732856 24179 ts_tablet_manager.cc:1875] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.739468 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet 8633864e248f416cb651da3b0c2aa936 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:40.742648 24173 tablet_bootstrap.cc:492] T 1065732d7333413d984ed56275fd61a9 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:40.742831 24173 ts_tablet_manager.cc:1367] T 1065732d7333413d984ed56275fd61a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.085s user 0.006s sys 0.006s
I20240502 08:16:40.743548 24175 tablet_bootstrap.cc:492] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:40.743680 24175 ts_tablet_manager.cc:1367] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.060s user 0.011s sys 0.000s
I20240502 08:16:40.743556 24173 raft_consensus.cc:348] T 1065732d7333413d984ed56275fd61a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.743774 24173 raft_consensus.cc:374] T 1065732d7333413d984ed56275fd61a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.743837 24173 raft_consensus.cc:729] T 1065732d7333413d984ed56275fd61a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.744061 24173 consensus_queue.cc:260] T 1065732d7333413d984ed56275fd61a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.744235 24596 raft_consensus.cc:482] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:40.744424 24596 raft_consensus.cc:504] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:40.745157 24596 leader_election.cc:288] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:40.745803 24173 ts_tablet_manager.cc:1398] T 1065732d7333413d984ed56275fd61a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:16:40.746102 24173 tablet_bootstrap.cc:492] T dcdd765895254a39b86b692b5d812c83 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:40.746557 24174 tablet_bootstrap.cc:492] T b11f504b93574e42a9bf30fde6d08207 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:40.746466 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce758c4a9258427987e8e83f579a2aa2" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:40.746701 24174 ts_tablet_manager.cc:1367] T b11f504b93574e42a9bf30fde6d08207 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.004s
W20240502 08:16:40.746665 23723 leader_election.cc:341] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: ce758c4a9258427987e8e83f579a2aa2
I20240502 08:16:40.747493 24174 raft_consensus.cc:348] T b11f504b93574e42a9bf30fde6d08207 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.747792 24173 tablet_bootstrap.cc:654] T dcdd765895254a39b86b692b5d812c83 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.748072 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3c57dba4df6d4a139583cb8ccb6e6ee9" } new_table_name: "default.Table_ebc89613d91747e1b59b01780555b212" modify_external_catalogs: true
I20240502 08:16:40.749655 24175 raft_consensus.cc:348] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.749897 24175 raft_consensus.cc:374] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.749984 24175 raft_consensus.cc:729] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.750284 24175 consensus_queue.cc:260] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.751010 24175 ts_tablet_manager.cc:1398] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 08:16:40.751360 24175 tablet_bootstrap.cc:492] T 8633864e248f416cb651da3b0c2aa936 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:40.751523 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2da8b9d3f374d8a812336057c0ce7d2 consensus state has no leader
I20240502 08:16:40.756338 24175 tablet_bootstrap.cc:654] T 8633864e248f416cb651da3b0c2aa936 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.757508 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet f36aadfa1d264f23bcafa91744aed36d
I20240502 08:16:40.757987 23609 catalog_manager.cc:6554] ReplaceTablet: tablet f36aadfa1d264f23bcafa91744aed36d deleted and replaced by tablet 2085ced52fff46ebaa98324cbb2d8c11
I20240502 08:16:40.759541 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bd9708a3da8240b4aa9444c8e9ed30d2" } modify_external_catalogs: true
I20240502 08:16:40.759763 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bd9708a3da8240b4aa9444c8e9ed30d2" } modify_external_catalogs: true
I20240502 08:16:40.760535 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_cf8b38c1f7e343bc9d21651af74e76c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:40.761026 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf8b38c1f7e343bc9d21651af74e76c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:40.749492 23723 leader_election.cc:341] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:40.761703 23723 leader_election.cc:302] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:40.758452 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet f36aadfa1d264f23bcafa91744aed36d with delete_type TABLET_DATA_DELETED (replaced by tablet 2085ced52fff46ebaa98324cbb2d8c11) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:40.762354 24596 raft_consensus.cc:2726] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:40.762441 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet f36aadfa1d264f23bcafa91744aed36d with delete_type TABLET_DATA_DELETED (replaced by tablet 2085ced52fff46ebaa98324cbb2d8c11) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:40.762619 24174 raft_consensus.cc:374] T b11f504b93574e42a9bf30fde6d08207 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 08:16:40.762732 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet f36aadfa1d264f23bcafa91744aed36d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f36aadfa1d264f23bcafa91744aed36d already in progress: creating tablet
I20240502 08:16:40.762766 24174 raft_consensus.cc:729] T b11f504b93574e42a9bf30fde6d08207 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.758219 24179 ts_tablet_manager.cc:1888] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.763131 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet 1065732d7333413d984ed56275fd61a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.763263 24179 log.cc:1196] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/b63fe5b6d4db4750800ecd429fa97ed1
I20240502 08:16:40.763170 24174 consensus_queue.cc:260] T b11f504b93574e42a9bf30fde6d08207 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.763656 24179 ts_tablet_manager.cc:1909] T b63fe5b6d4db4750800ecd429fa97ed1 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:40.764168 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet f36aadfa1d264f23bcafa91744aed36d with delete_type TABLET_DATA_DELETED (replaced by tablet 2085ced52fff46ebaa98324cbb2d8c11) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.769959 24174 ts_tablet_manager.cc:1398] T b11f504b93574e42a9bf30fde6d08207 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20240502 08:16:40.771142 24181 tablet_replica.cc:330] T 1065732d7333413d984ed56275fd61a9 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:40.771394 24181 raft_consensus.cc:2229] T 1065732d7333413d984ed56275fd61a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.772217 24181 raft_consensus.cc:2258] T 1065732d7333413d984ed56275fd61a9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.772272 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 305) took 1823 ms. Trace:
I20240502 08:16:40.773066 24181 ts_tablet_manager.cc:1875] T 1065732d7333413d984ed56275fd61a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.773873 24174 tablet_bootstrap.cc:492] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:40.775981 24179 rpcz_store.cc:276] 0502 08:16:38.949183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.953409 (+ 4226us) service_pool.cc:225] Handling call
0502 08:16:40.731829 (+1778420us) ts_tablet_manager.cc:1120] Deleting tablet b63fe5b6d4db4750800ecd429fa97ed1
0502 08:16:40.731838 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.735780 (+ 3942us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.758089 (+ 22309us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.772213 (+ 14124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1664,"tablet-delete.queue_time_us":1735535}
I20240502 08:16:40.777386 24173 tablet_bootstrap.cc:492] T dcdd765895254a39b86b692b5d812c83 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:40.777534 24173 ts_tablet_manager.cc:1367] T dcdd765895254a39b86b692b5d812c83 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.002s
I20240502 08:16:40.778316 24173 raft_consensus.cc:348] T dcdd765895254a39b86b692b5d812c83 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.778510 24173 raft_consensus.cc:374] T dcdd765895254a39b86b692b5d812c83 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.778604 24173 raft_consensus.cc:729] T dcdd765895254a39b86b692b5d812c83 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.778831 24173 consensus_queue.cc:260] T dcdd765895254a39b86b692b5d812c83 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.779402 24173 ts_tablet_manager.cc:1398] T dcdd765895254a39b86b692b5d812c83 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:40.779836 24173 tablet_bootstrap.cc:492] T 10adabdd31f848548ef6f2408ccc2050 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:40.779068 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 309) took 1828 ms. Trace:
I20240502 08:16:40.780256 24179 rpcz_store.cc:276] 0502 08:16:38.950119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.001192 (+ 51073us) service_pool.cc:225] Handling call
0502 08:16:40.778869 (+1777677us) ts_tablet_manager.cc:1120] Deleting tablet 1e5de793e74247dd9e73b7b1d58027d9
0502 08:16:40.778881 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.779013 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1771750}
I20240502 08:16:40.781373 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 310) took 1831 ms. Trace:
I20240502 08:16:40.781464 24179 rpcz_store.cc:276] 0502 08:16:38.950232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.004362 (+ 54130us) service_pool.cc:225] Handling call
0502 08:16:40.781187 (+1776825us) ts_tablet_manager.cc:1120] Deleting tablet 848a0865eb2b4828a2594a17d96869d0
0502 08:16:40.781198 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.781321 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1755373}
I20240502 08:16:40.782464 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet b63fe5b6d4db4750800ecd429fa97ed1 (table default.Table_07371eaf91cb4ab199af62382b640d64 [id=0026ee826a3e489087f873579507c110]) successfully deleted
W20240502 08:16:40.782785 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 1e5de793e74247dd9e73b7b1d58027d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e5de793e74247dd9e73b7b1d58027d9 already in progress: creating tablet
W20240502 08:16:40.783046 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 848a0865eb2b4828a2594a17d96869d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 848a0865eb2b4828a2594a17d96869d0 already in progress: creating tablet
I20240502 08:16:40.784730 24173 tablet_bootstrap.cc:654] T 10adabdd31f848548ef6f2408ccc2050 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.784827 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 312) took 1834 ms. Trace:
I20240502 08:16:40.784924 24179 rpcz_store.cc:276] 0502 08:16:38.950426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.025839 (+ 75413us) service_pool.cc:225] Handling call
0502 08:16:40.784650 (+1758811us) ts_tablet_manager.cc:1120] Deleting tablet 71aa916247c449fea718b35869a6df20
0502 08:16:40.784661 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.784768 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1754236}
I20240502 08:16:40.785444 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 313) took 1834 ms. Trace:
I20240502 08:16:40.785638 24179 rpcz_store.cc:276] 0502 08:16:38.950508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.030440 (+ 79932us) service_pool.cc:225] Handling call
0502 08:16:40.785280 (+1754840us) ts_tablet_manager.cc:1120] Deleting tablet b11f504b93574e42a9bf30fde6d08207
0502 08:16:40.785289 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.785382 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1754674}
I20240502 08:16:40.786147 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 314) took 1835 ms. Trace:
I20240502 08:16:40.786298 24179 rpcz_store.cc:276] 0502 08:16:38.950622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.030616 (+ 79994us) service_pool.cc:225] Handling call
0502 08:16:40.786005 (+1755389us) ts_tablet_manager.cc:1120] Deleting tablet 7c374f42f353497fa1d745164e7fbf61
0502 08:16:40.786013 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.786097 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1755230}
W20240502 08:16:40.786792 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 71aa916247c449fea718b35869a6df20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71aa916247c449fea718b35869a6df20 already in progress: creating tablet
W20240502 08:16:40.787036 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet b11f504b93574e42a9bf30fde6d08207 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b11f504b93574e42a9bf30fde6d08207 already in progress: creating tablet
W20240502 08:16:40.787281 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 7c374f42f353497fa1d745164e7fbf61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c374f42f353497fa1d745164e7fbf61 already in progress: creating tablet
I20240502 08:16:40.788699 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 315) took 1837 ms. Trace:
I20240502 08:16:40.789104 24179 rpcz_store.cc:276] 0502 08:16:38.950940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.030786 (+ 79846us) service_pool.cc:225] Handling call
0502 08:16:40.788553 (+1757767us) ts_tablet_manager.cc:1120] Deleting tablet 5a1e208733844829b13222e00a432439
0502 08:16:40.788563 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.788650 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1757631}
I20240502 08:16:40.790397 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 43e870c3a0e340d1bad1c1fa0380a762
I20240502 08:16:40.789959 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 316) took 1838 ms. Trace:
I20240502 08:16:40.791918 24179 rpcz_store.cc:276] 0502 08:16:38.951098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.030927 (+ 79829us) service_pool.cc:225] Handling call
0502 08:16:40.789791 (+1758864us) ts_tablet_manager.cc:1120] Deleting tablet b6417aafbdcf416ea7bff56bce5e6ddb
0502 08:16:40.789800 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.789910 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1758674}
I20240502 08:16:40.792398 24181 ts_tablet_manager.cc:1888] T 1065732d7333413d984ed56275fd61a9 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.792575 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet c00f29fb76b44d6ea4f76963d2d0751c
I20240502 08:16:40.793066 23601 catalog_manager.cc:6554] ReplaceTablet: tablet c00f29fb76b44d6ea4f76963d2d0751c deleted and replaced by tablet a17fb4f2f02e43dc9f7cfdb7c73f7774
I20240502 08:16:40.793259 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 43e870c3a0e340d1bad1c1fa0380a762 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:40.793320 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 43e870c3a0e340d1bad1c1fa0380a762 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.793828 24181 log.cc:1196] T 1065732d7333413d984ed56275fd61a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/1065732d7333413d984ed56275fd61a9
W20240502 08:16:40.794023 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 5a1e208733844829b13222e00a432439 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a1e208733844829b13222e00a432439 already in progress: creating tablet
I20240502 08:16:40.794222 24181 ts_tablet_manager.cc:1909] T 1065732d7333413d984ed56275fd61a9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
W20240502 08:16:40.794260 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet b6417aafbdcf416ea7bff56bce5e6ddb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6417aafbdcf416ea7bff56bce5e6ddb already in progress: creating tablet
I20240502 08:16:40.794821 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet c00f29fb76b44d6ea4f76963d2d0751c with delete_type TABLET_DATA_DELETED (replaced by tablet a17fb4f2f02e43dc9f7cfdb7c73f7774) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:40.794960 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet c00f29fb76b44d6ea4f76963d2d0751c with delete_type TABLET_DATA_DELETED (replaced by tablet a17fb4f2f02e43dc9f7cfdb7c73f7774) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.795047 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 317) took 1843 ms. Trace:
I20240502 08:16:40.795388 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet c17e1be6ef294769a01981f4d361be21
I20240502 08:16:40.795655 24179 rpcz_store.cc:276] 0502 08:16:38.951195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.031126 (+ 79931us) service_pool.cc:225] Handling call
0502 08:16:40.792839 (+1761713us) ts_tablet_manager.cc:1120] Deleting tablet dd3078cc1f6247609f8bd017ed076b45
0502 08:16:40.792848 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.794997 (+ 2149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1761581}
I20240502 08:16:40.800681 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet 43e870c3a0e340d1bad1c1fa0380a762 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:40.801017 24180 tablet_replica.cc:330] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:40.801106 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 08183523bf4a4d1c9020eb693aa07878 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:40.801254 24180 raft_consensus.cc:2229] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:40.801313 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet c17e1be6ef294769a01981f4d361be21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:40.801527 24180 pending_rounds.cc:62] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:16:40.801633 24180 pending_rounds.cc:68] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156428603432960 op_type: NO_OP noop_request { }
I20240502 08:16:40.801800 24180 raft_consensus.cc:2866] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:40.801903 24180 raft_consensus.cc:2258] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.800978 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet c00f29fb76b44d6ea4f76963d2d0751c with delete_type TABLET_DATA_DELETED (replaced by tablet a17fb4f2f02e43dc9f7cfdb7c73f7774) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:40.802568 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 311) took 1852 ms. Trace:
I20240502 08:16:40.804303 24180 ts_tablet_manager.cc:1875] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.804340 24179 rpcz_store.cc:276] 0502 08:16:38.950307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.007133 (+ 56826us) service_pool.cc:225] Handling call
0502 08:16:40.802371 (+1795238us) ts_tablet_manager.cc:1120] Deleting tablet 0882b7509a74475780eb700818aab398
0502 08:16:40.802383 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.802516 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1766274}
I20240502 08:16:40.805186 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 332) took 1306 ms. Trace:
I20240502 08:16:40.805269 24179 rpcz_store.cc:276] 0502 08:16:39.498899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.498960 (+ 61us) service_pool.cc:225] Handling call
0502 08:16:40.805025 (+1306065us) ts_tablet_manager.cc:1120] Deleting tablet 18d284f71937459db0c0676ce4568175
0502 08:16:40.805036 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.805133 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1303366}
I20240502 08:16:40.806248 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 333) took 1302 ms. Trace:
I20240502 08:16:40.806322 24179 rpcz_store.cc:276] 0502 08:16:39.503429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.503537 (+ 108us) service_pool.cc:225] Handling call
0502 08:16:40.806117 (+1302580us) ts_tablet_manager.cc:1120] Deleting tablet 9c03a4353c4e46e3bcdf3a36aac04bf5
0502 08:16:40.806127 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.806204 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1296588}
I20240502 08:16:40.806655 24181 tablet_replica.cc:330] T 43e870c3a0e340d1bad1c1fa0380a762 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:40.806870 24181 raft_consensus.cc:2229] T 43e870c3a0e340d1bad1c1fa0380a762 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.807014 24181 raft_consensus.cc:2258] T 43e870c3a0e340d1bad1c1fa0380a762 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.807623 24181 ts_tablet_manager.cc:1875] T 43e870c3a0e340d1bad1c1fa0380a762 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.808624 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 334) took 1282 ms. Trace:
I20240502 08:16:40.808694 24179 rpcz_store.cc:276] 0502 08:16:39.526204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.526341 (+ 137us) service_pool.cc:225] Handling call
0502 08:16:40.808500 (+1282159us) ts_tablet_manager.cc:1120] Deleting tablet 5453cc0b93ac4cbb90a0588bb2100d60
0502 08:16:40.808508 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.808582 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1274306}
I20240502 08:16:40.809491 24179 tablet_replica.cc:330] T 203ed05d0bdc4b1f99207423b7fc848b P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:40.809482 24596 raft_consensus.cc:482] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:40.809651 24596 raft_consensus.cc:504] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.809934 24179 raft_consensus.cc:2229] T 203ed05d0bdc4b1f99207423b7fc848b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.810101 24179 raft_consensus.cc:2258] T 203ed05d0bdc4b1f99207423b7fc848b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.810312 24596 leader_election.cc:288] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:40.810662 24179 ts_tablet_manager.cc:1875] T 203ed05d0bdc4b1f99207423b7fc848b P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.810951 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ce83c8001d4b2988b88c0ed9bf948c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:40.811147 23923 raft_consensus.cc:2454] T 23ce83c8001d4b2988b88c0ed9bf948c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:40.811954 23723 leader_election.cc:302] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:40.812405 24489 raft_consensus.cc:2781] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:40.812496 24489 raft_consensus.cc:482] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:40.812587 24489 raft_consensus.cc:3037] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:40.812516 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ce83c8001d4b2988b88c0ed9bf948c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:40.812762 24060 raft_consensus.cc:2454] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:40.815680 24489 raft_consensus.cc:504] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.816365 24489 leader_election.cc:288] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:40.817013 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ce83c8001d4b2988b88c0ed9bf948c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:40.817211 23923 raft_consensus.cc:3037] T 23ce83c8001d4b2988b88c0ed9bf948c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:40.817200 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ce83c8001d4b2988b88c0ed9bf948c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:40.817412 24060 raft_consensus.cc:3037] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:40.817727 24173 tablet_bootstrap.cc:492] T 10adabdd31f848548ef6f2408ccc2050 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:40.817958 24173 ts_tablet_manager.cc:1367] T 10adabdd31f848548ef6f2408ccc2050 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.007s
I20240502 08:16:40.818814 24173 raft_consensus.cc:348] T 10adabdd31f848548ef6f2408ccc2050 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.819032 24173 raft_consensus.cc:374] T 10adabdd31f848548ef6f2408ccc2050 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.819113 24173 raft_consensus.cc:729] T 10adabdd31f848548ef6f2408ccc2050 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.819339 24173 consensus_queue.cc:260] T 10adabdd31f848548ef6f2408ccc2050 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.819734 23923 raft_consensus.cc:2454] T 23ce83c8001d4b2988b88c0ed9bf948c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:40.819845 24173 ts_tablet_manager.cc:1398] T 10adabdd31f848548ef6f2408ccc2050 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:40.821274 24173 tablet_bootstrap.cc:492] T b6417aafbdcf416ea7bff56bce5e6ddb P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:40.822976 24173 tablet_bootstrap.cc:654] T b6417aafbdcf416ea7bff56bce5e6ddb P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.824959 23723 leader_election.cc:302] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:40.828567 24060 raft_consensus.cc:2454] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:40.831568 24489 raft_consensus.cc:2781] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:40.831741 24489 raft_consensus.cc:686] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:40.831997 24489 consensus_queue.cc:237] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.836199 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet c17e1be6ef294769a01981f4d361be21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:40.836613 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 1065732d7333413d984ed56275fd61a9 (table default.Table_c422ad5c5f164908bfae0d2977635bb3 [id=3584daf64dec4be7a42a2eee267b4861]) successfully deleted
W20240502 08:16:40.836910 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet f36aadfa1d264f23bcafa91744aed36d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f36aadfa1d264f23bcafa91744aed36d already in progress: creating tablet
W20240502 08:16:40.837157 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet dd3078cc1f6247609f8bd017ed076b45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd3078cc1f6247609f8bd017ed076b45 already in progress: creating tablet
W20240502 08:16:40.837386 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 0882b7509a74475780eb700818aab398 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0882b7509a74475780eb700818aab398 already in progress: creating tablet
W20240502 08:16:40.837635 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 18d284f71937459db0c0676ce4568175 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18d284f71937459db0c0676ce4568175 already in progress: creating tablet
W20240502 08:16:40.837877 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 already in progress: creating tablet
W20240502 08:16:40.838075 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 5453cc0b93ac4cbb90a0588bb2100d60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5453cc0b93ac4cbb90a0588bb2100d60 already in progress: creating tablet
I20240502 08:16:40.838554 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet c17e1be6ef294769a01981f4d361be21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.815659 24180 ts_tablet_manager.cc:1888] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:40.841429 24180 log.cc:1196] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/43e870c3a0e340d1bad1c1fa0380a762
I20240502 08:16:40.841940 24180 ts_tablet_manager.cc:1909] T 43e870c3a0e340d1bad1c1fa0380a762 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:40.842950 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 43e870c3a0e340d1bad1c1fa0380a762 (table default.Table_299ae144ccc6402abe33d07ef693e5b4 [id=7e0c798bc77a4770a4bbe8658f181ffa]) successfully deleted
I20240502 08:16:40.846185 24173 tablet_bootstrap.cc:492] T b6417aafbdcf416ea7bff56bce5e6ddb P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:40.846335 24173 ts_tablet_manager.cc:1367] T b6417aafbdcf416ea7bff56bce5e6ddb P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.003s
I20240502 08:16:40.847146 24173 raft_consensus.cc:348] T b6417aafbdcf416ea7bff56bce5e6ddb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.847390 24173 raft_consensus.cc:374] T b6417aafbdcf416ea7bff56bce5e6ddb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.847487 24173 raft_consensus.cc:729] T b6417aafbdcf416ea7bff56bce5e6ddb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.847729 24173 consensus_queue.cc:260] T b6417aafbdcf416ea7bff56bce5e6ddb P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.849231 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8bbe9f5123cc4dacb9fd2749d84f9988"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:40.849638 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6ce45bf2f8e64d759b44ee43f3df434d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:40.849900 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8bbe9f5123cc4dacb9fd2749d84f9988 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:40.850046 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ce45bf2f8e64d759b44ee43f3df434d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:40.851137 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ca3dbc715088456686fa2e20e8c5b3b7" } new_table_name: "default.Table_b3289c94e9274ea591f2b4e00a58a84d" modify_external_catalogs: true
I20240502 08:16:40.851792 24173 ts_tablet_manager.cc:1398] T b6417aafbdcf416ea7bff56bce5e6ddb P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:16:40.853721 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b1e5c38f8b45471698ed791873ecdb99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:40.853832 24173 tablet_bootstrap.cc:492] T c470d420c45843a69a63d9800b7cfc97 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:40.854163 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1e5c38f8b45471698ed791873ecdb99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:40.855506 24173 tablet_bootstrap.cc:654] T c470d420c45843a69a63d9800b7cfc97 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.856794 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 1065732d7333413d984ed56275fd61a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.863386 24179 ts_tablet_manager.cc:1888] T 203ed05d0bdc4b1f99207423b7fc848b P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.863550 24179 log.cc:1196] T 203ed05d0bdc4b1f99207423b7fc848b P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/203ed05d0bdc4b1f99207423b7fc848b
I20240502 08:16:40.863925 24179 ts_tablet_manager.cc:1909] T 203ed05d0bdc4b1f99207423b7fc848b P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:40.864764 24181 ts_tablet_manager.cc:1888] T 43e870c3a0e340d1bad1c1fa0380a762 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.864919 24181 log.cc:1196] T 43e870c3a0e340d1bad1c1fa0380a762 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/43e870c3a0e340d1bad1c1fa0380a762
I20240502 08:16:40.865325 24181 ts_tablet_manager.cc:1909] T 43e870c3a0e340d1bad1c1fa0380a762 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:40.866868 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a89a2cdb74684df5bcbe15389816c50b" } modify_external_catalogs: true
I20240502 08:16:40.867125 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a89a2cdb74684df5bcbe15389816c50b" } modify_external_catalogs: true
I20240502 08:16:40.870193 24173 tablet_bootstrap.cc:492] T c470d420c45843a69a63d9800b7cfc97 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:40.870357 24173 ts_tablet_manager.cc:1367] T c470d420c45843a69a63d9800b7cfc97 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.005s
I20240502 08:16:40.871042 24173 raft_consensus.cc:348] T c470d420c45843a69a63d9800b7cfc97 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.871248 24173 raft_consensus.cc:374] T c470d420c45843a69a63d9800b7cfc97 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.871318 24173 raft_consensus.cc:729] T c470d420c45843a69a63d9800b7cfc97 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.871570 24173 consensus_queue.cc:260] T c470d420c45843a69a63d9800b7cfc97 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.875313 24173 ts_tablet_manager.cc:1398] T c470d420c45843a69a63d9800b7cfc97 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:16:40.875972 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 43e870c3a0e340d1bad1c1fa0380a762 (table default.Table_299ae144ccc6402abe33d07ef693e5b4 [id=7e0c798bc77a4770a4bbe8658f181ffa]) successfully deleted
W20240502 08:16:40.876334 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet c00f29fb76b44d6ea4f76963d2d0751c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c00f29fb76b44d6ea4f76963d2d0751c already in progress: creating tablet
I20240502 08:16:40.876818 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet dcdd765895254a39b86b692b5d812c83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.876890 24173 tablet_bootstrap.cc:492] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:40.877312 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet b6417aafbdcf416ea7bff56bce5e6ddb with delete_type TABLET_DATA_DELETED (replaced by tablet 96b8f21c64e343d88d84e9acd0bc69b4) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.877878 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 10adabdd31f848548ef6f2408ccc2050 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.878935 24173 tablet_bootstrap.cc:654] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.881156 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 335) took 1346 ms. Trace:
W20240502 08:16:40.881189 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet c17e1be6ef294769a01981f4d361be21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c17e1be6ef294769a01981f4d361be21 already in progress: creating tablet
W20240502 08:16:40.881472 23597 catalog_manager.cc:4797] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 1065732d7333413d984ed56275fd61a9 because the tablet was not found. No further retry: Not found: Tablet not found: 1065732d7333413d984ed56275fd61a9
I20240502 08:16:40.881867 24181 tablet_replica.cc:330] T dcdd765895254a39b86b692b5d812c83 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:40.882306 24181 raft_consensus.cc:2229] T dcdd765895254a39b86b692b5d812c83 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.882800 24181 raft_consensus.cc:2258] T dcdd765895254a39b86b692b5d812c83 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.883116 24179 rpcz_store.cc:276] 0502 08:16:39.534380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.534927 (+ 547us) service_pool.cc:225] Handling call
0502 08:16:40.809404 (+1274477us) ts_tablet_manager.cc:1120] Deleting tablet 203ed05d0bdc4b1f99207423b7fc848b
0502 08:16:40.809413 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.834039 (+ 24626us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.863259 (+ 29220us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.881112 (+ 17853us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1253734}
I20240502 08:16:40.889549 24174 tablet_bootstrap.cc:654] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.890352 24181 ts_tablet_manager.cc:1875] T dcdd765895254a39b86b692b5d812c83 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.890919 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 356) took 1305 ms. Trace:
I20240502 08:16:40.890959 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 203ed05d0bdc4b1f99207423b7fc848b (table default.Table_68bf91a14ffa42cdbb10a1d7ca56cb08 [id=59d945a73daa4918ba17063eae0b9b93]) successfully deleted
I20240502 08:16:40.891485 23429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2vVFYb/build/asan/bin/kudu pid 23507
W20240502 08:16:40.891743 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 08183523bf4a4d1c9020eb693aa07878 because the tablet was not found. No further retry: Not found: Tablet not found: 08183523bf4a4d1c9020eb693aa07878
I20240502 08:16:40.892047 24179 rpcz_store.cc:276] 0502 08:16:39.585423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.586409 (+ 986us) service_pool.cc:225] Handling call
0502 08:16:40.890743 (+1304334us) ts_tablet_manager.cc:1120] Deleting tablet 128c88af54d24ffd895e9de19cdb4c8a
0502 08:16:40.890756 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.890882 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1304111}
I20240502 08:16:40.891000 24173 tablet_bootstrap.cc:492] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:40.892493 24173 ts_tablet_manager.cc:1367] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.005s
W20240502 08:16:40.892973 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet c17e1be6ef294769a01981f4d361be21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c17e1be6ef294769a01981f4d361be21 already in progress: creating tablet
I20240502 08:16:40.893231 24173 raft_consensus.cc:348] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:40.893433 24173 raft_consensus.cc:374] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.893532 24173 raft_consensus.cc:729] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
W20240502 08:16:40.893519 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 128c88af54d24ffd895e9de19cdb4c8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 128c88af54d24ffd895e9de19cdb4c8a already in progress: creating tablet
I20240502 08:16:40.893792 24173 consensus_queue.cc:260] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:40.894023 23569 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:16:40.894235 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet c00f29fb76b44d6ea4f76963d2d0751c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c00f29fb76b44d6ea4f76963d2d0751c already in progress: creating tablet
I20240502 08:16:40.895694 24173 ts_tablet_manager.cc:1398] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:40.895995 24173 tablet_bootstrap.cc:492] T b11f504b93574e42a9bf30fde6d08207 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:40.897426 24173 tablet_bootstrap.cc:654] T b11f504b93574e42a9bf30fde6d08207 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:40.903007 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c395a29c7a1485eab39ca107dd91fca consensus state has no leader
I20240502 08:16:40.910102 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 358) took 1324 ms. Trace:
I20240502 08:16:40.910948 24173 tablet_bootstrap.cc:492] T b11f504b93574e42a9bf30fde6d08207 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:40.911150 24173 ts_tablet_manager.cc:1367] T b11f504b93574e42a9bf30fde6d08207 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.004s
I20240502 08:16:40.911790 24179 rpcz_store.cc:276] 0502 08:16:39.586000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.586640 (+ 640us) service_pool.cc:225] Handling call
0502 08:16:40.909879 (+1323239us) ts_tablet_manager.cc:1120] Deleting tablet 7a97427e819b4826b1ed89e6d021c8d4
0502 08:16:40.909909 (+ 30us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.910057 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1323042}
I20240502 08:16:40.911998 24173 raft_consensus.cc:348] T b11f504b93574e42a9bf30fde6d08207 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.912235 24173 raft_consensus.cc:374] T b11f504b93574e42a9bf30fde6d08207 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.912307 24173 raft_consensus.cc:729] T b11f504b93574e42a9bf30fde6d08207 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.912525 24173 consensus_queue.cc:260] T b11f504b93574e42a9bf30fde6d08207 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.913393 24173 ts_tablet_manager.cc:1398] T b11f504b93574e42a9bf30fde6d08207 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:40.913812 24173 tablet_bootstrap.cc:492] T d8f7b5f018d84680ac5533bf99102d31 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:40.914531 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 357) took 1328 ms. Trace:
I20240502 08:16:40.915349 24179 rpcz_store.cc:276] 0502 08:16:39.585770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.586873 (+ 1103us) service_pool.cc:225] Handling call
0502 08:16:40.913938 (+1327065us) ts_tablet_manager.cc:1120] Deleting tablet 090bec30f5664421b9bfe3f2810afd14
0502 08:16:40.913954 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.914487 (+ 533us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1326870}
I20240502 08:16:40.916496 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_91aaef5fd943406aa3548482115afc65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:40.917210 24179 tablet_replica.cc:330] T 5738cd77bbe147b49215ba9639d7f547 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
W20240502 08:16:40.917297 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91aaef5fd943406aa3548482115afc65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:40.917524 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5e74a1a8d1ce46018625fb11e1dac26e" } new_table_name: "default.Table_978594faab984d60ba03d29d40328a6c" modify_external_catalogs: true
I20240502 08:16:40.917635 24179 raft_consensus.cc:2229] T 5738cd77bbe147b49215ba9639d7f547 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.917923 24179 raft_consensus.cc:2258] T 5738cd77bbe147b49215ba9639d7f547 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.918596 24179 ts_tablet_manager.cc:1875] T 5738cd77bbe147b49215ba9639d7f547 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.927079 24173 tablet_bootstrap.cc:654] T d8f7b5f018d84680ac5533bf99102d31 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:40.928988 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 7a97427e819b4826b1ed89e6d021c8d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a97427e819b4826b1ed89e6d021c8d4 already in progress: creating tablet
I20240502 08:16:40.929591 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet c470d420c45843a69a63d9800b7cfc97 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:40.930167 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 21832f0151ec4cf48f51f9bd2cb2b5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:40.930657 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 090bec30f5664421b9bfe3f2810afd14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 090bec30f5664421b9bfe3f2810afd14 already in progress: creating tablet
I20240502 08:16:40.932093 24175 tablet_bootstrap.cc:492] T 8633864e248f416cb651da3b0c2aa936 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:40.932291 24175 ts_tablet_manager.cc:1367] T 8633864e248f416cb651da3b0c2aa936 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.181s user 0.018s sys 0.006s
I20240502 08:16:40.933229 24175 raft_consensus.cc:348] T 8633864e248f416cb651da3b0c2aa936 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.933502 24175 raft_consensus.cc:374] T 8633864e248f416cb651da3b0c2aa936 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.933604 24175 raft_consensus.cc:729] T 8633864e248f416cb651da3b0c2aa936 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.933878 24175 consensus_queue.cc:260] T 8633864e248f416cb651da3b0c2aa936 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.934650 24175 ts_tablet_manager.cc:1398] T 8633864e248f416cb651da3b0c2aa936 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 08:16:40.935973 24181 ts_tablet_manager.cc:1888] T dcdd765895254a39b86b692b5d812c83 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.936391 24175 tablet_bootstrap.cc:492] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:40.943149 24175 tablet_bootstrap.cc:654] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.943333 24181 log.cc:1196] T dcdd765895254a39b86b692b5d812c83 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/dcdd765895254a39b86b692b5d812c83
I20240502 08:16:40.943782 24181 ts_tablet_manager.cc:1909] T dcdd765895254a39b86b692b5d812c83 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:40.950359 23605 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 23ce83c8001d4b2988b88c0ed9bf948c (table default.Table_190eae2a8c64440f8d19842f8b6ac254 [id=54454efbe9394170822f237dc8e6115e]). Expected version 1 got 0
I20240502 08:16:40.950701 24181 tablet_replica.cc:330] T b6417aafbdcf416ea7bff56bce5e6ddb P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:40.950978 24181 raft_consensus.cc:2229] T b6417aafbdcf416ea7bff56bce5e6ddb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.951251 24181 raft_consensus.cc:2258] T b6417aafbdcf416ea7bff56bce5e6ddb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.940186 24179 ts_tablet_manager.cc:1888] T 5738cd77bbe147b49215ba9639d7f547 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.951808 24179 log.cc:1196] T 5738cd77bbe147b49215ba9639d7f547 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/5738cd77bbe147b49215ba9639d7f547
I20240502 08:16:40.952243 24179 ts_tablet_manager.cc:1909] T 5738cd77bbe147b49215ba9639d7f547 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:40.953101 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 360) took 1366 ms. Trace:
I20240502 08:16:40.953264 24179 rpcz_store.cc:276] 0502 08:16:39.586392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.587071 (+ 679us) service_pool.cc:225] Handling call
0502 08:16:40.917117 (+1330046us) ts_tablet_manager.cc:1120] Deleting tablet 5738cd77bbe147b49215ba9639d7f547
0502 08:16:40.917132 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.937127 (+ 19995us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.940094 (+ 2967us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:40.953047 (+ 12953us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1329839}
I20240502 08:16:40.953971 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 359) took 1367 ms. Trace:
I20240502 08:16:40.954129 24179 rpcz_store.cc:276] 0502 08:16:39.586179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.586819 (+ 640us) service_pool.cc:225] Handling call
0502 08:16:40.953792 (+1366973us) ts_tablet_manager.cc:1120] Deleting tablet df9402cf8abc4b4d893d42f9a3a99016
0502 08:16:40.953801 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.953918 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1366400}
I20240502 08:16:40.954445 23608 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet e134e90efb4a48bb82b53a80bda708b3 (table default.Table_cc80d1cedafa4e71906e5e9530fdd828 [id=47682c26a85842c7b8991bf3828a9336]). Expected version 1 got 0
I20240502 08:16:40.954854 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 361) took 1368 ms. Trace:
I20240502 08:16:40.956254 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet b11f504b93574e42a9bf30fde6d08207 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:40.957028 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet ce758c4a9258427987e8e83f579a2aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:40.957736 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cc30014a1ac45de937751eed36877a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:40.957988 24179 rpcz_store.cc:276] 0502 08:16:39.586569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.587291 (+ 722us) service_pool.cc:225] Handling call
0502 08:16:40.954674 (+1367383us) ts_tablet_manager.cc:1120] Deleting tablet a3043c5ed98c401b82c6e967ca3f3613
0502 08:16:40.954682 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.954801 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1367199}
I20240502 08:16:40.958204 24181 ts_tablet_manager.cc:1875] T b6417aafbdcf416ea7bff56bce5e6ddb P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.958556 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 362) took 1361 ms. Trace:
I20240502 08:16:40.958781 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 6788ffd2eb2248c3b84e474de56c37a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:40.959528 24179 rpcz_store.cc:276] 0502 08:16:39.596710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.657761 (+ 61051us) service_pool.cc:225] Handling call
0502 08:16:40.958413 (+1300652us) ts_tablet_manager.cc:1120] Deleting tablet 5bcc7169796e4ee0910f5e7084849012
0502 08:16:40.958421 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.958508 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1300420}
I20240502 08:16:40.959417 24175 tablet_bootstrap.cc:492] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:40.960057 24175 ts_tablet_manager.cc:1367] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240502 08:16:40.960902 24175 raft_consensus.cc:348] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.961170 24175 raft_consensus.cc:374] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.961293 24175 raft_consensus.cc:729] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.961552 24175 consensus_queue.cc:260] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:40.963655 23756 tablet_service.cc:1467] Processing CreateTablet for tablet e2792c45255c42adaeefb7aef4e25715 (DEFAULT_TABLE table=default.Table_c466cab9104c4631a59e1d5e481c5fe0 [id=e1658213b68a4cbc91a9435d0b238cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.964092 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2792c45255c42adaeefb7aef4e25715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.965479 24181 ts_tablet_manager.cc:1888] T b6417aafbdcf416ea7bff56bce5e6ddb P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:40.965658 24181 log.cc:1196] T b6417aafbdcf416ea7bff56bce5e6ddb P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/b6417aafbdcf416ea7bff56bce5e6ddb
I20240502 08:16:40.965955 24181 ts_tablet_manager.cc:1909] T b6417aafbdcf416ea7bff56bce5e6ddb P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:40.972330 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 3a74a6eb6d214d858903b3307e09e9e6 (DEFAULT_TABLE table=default.Table_c22209bdcbfd4dfa9e643ab9c98dc005 [id=232529911c8d43e2860c52d5527597d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.972663 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a74a6eb6d214d858903b3307e09e9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.973618 24175 ts_tablet_manager.cc:1398] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:16:40.973904 24175 tablet_bootstrap.cc:492] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:40.979271 23756 tablet_service.cc:1467] Processing CreateTablet for tablet d16110e1b1664802a98182c73f1e41d3 (DEFAULT_TABLE table=default.Table_9feed0e1d82c472aaf0e9b92cd9173d5 [id=0f45bbbc044d4e15b40e0a7f9edf6ef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.979723 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d16110e1b1664802a98182c73f1e41d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.979959 24175 tablet_bootstrap.cc:654] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.981156 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 363) took 1175 ms. Trace:
I20240502 08:16:40.981312 24179 rpcz_store.cc:276] 0502 08:16:39.804891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.806917 (+ 2026us) service_pool.cc:225] Handling call
0502 08:16:40.974186 (+1167269us) ts_tablet_manager.cc:1120] Deleting tablet 654ce33d1e3d472eb6ae6a726eb803bb
0502 08:16:40.974193 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.979883 (+ 5690us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1132599}
I20240502 08:16:40.981875 24181 tablet_replica.cc:330] T 10adabdd31f848548ef6f2408ccc2050 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:40.982228 24181 raft_consensus.cc:2229] T 10adabdd31f848548ef6f2408ccc2050 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.982586 24181 raft_consensus.cc:2258] T 10adabdd31f848548ef6f2408ccc2050 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.983415 24181 ts_tablet_manager.cc:1875] T 10adabdd31f848548ef6f2408ccc2050 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:40.984949 24039 tablet_service.cc:1467] Processing CreateTablet for tablet e2792c45255c42adaeefb7aef4e25715 (DEFAULT_TABLE table=default.Table_c466cab9104c4631a59e1d5e481c5fe0 [id=e1658213b68a4cbc91a9435d0b238cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.985407 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2792c45255c42adaeefb7aef4e25715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.987078 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 5738cd77bbe147b49215ba9639d7f547 (table default.Table_c84ff62b97d44dd2ad4a1d9927383457 [id=1bfba58d949847308b00af3275167658]) successfully deleted
I20240502 08:16:40.987295 23891 tablet_service.cc:1467] Processing CreateTablet for tablet e2792c45255c42adaeefb7aef4e25715 (DEFAULT_TABLE table=default.Table_c466cab9104c4631a59e1d5e481c5fe0 [id=e1658213b68a4cbc91a9435d0b238cb0]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:40.987473 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet df9402cf8abc4b4d893d42f9a3a99016 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df9402cf8abc4b4d893d42f9a3a99016 already in progress: creating tablet
W20240502 08:16:40.987718 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet a3043c5ed98c401b82c6e967ca3f3613 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3043c5ed98c401b82c6e967ca3f3613 already in progress: creating tablet
I20240502 08:16:40.987792 24030 tablet_service.cc:1467] Processing CreateTablet for tablet d16110e1b1664802a98182c73f1e41d3 (DEFAULT_TABLE table=default.Table_9feed0e1d82c472aaf0e9b92cd9173d5 [id=0f45bbbc044d4e15b40e0a7f9edf6ef2]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:40.987918 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 5bcc7169796e4ee0910f5e7084849012 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bcc7169796e4ee0910f5e7084849012 already in progress: creating tablet
I20240502 08:16:40.988142 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2792c45255c42adaeefb7aef4e25715. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:40.988189 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 654ce33d1e3d472eb6ae6a726eb803bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 654ce33d1e3d472eb6ae6a726eb803bb already in progress: creating tablet
I20240502 08:16:40.988479 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d16110e1b1664802a98182c73f1e41d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.988487 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet dcdd765895254a39b86b692b5d812c83 (table default.Table_dd3c838f4a014e178351d135f5d4567d [id=6be0b9c539dd40059dab4bae00b7b180]) successfully deleted
I20240502 08:16:40.988694 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet b6417aafbdcf416ea7bff56bce5e6ddb (table default.Table_9d372763eb03435f8b7074a802d86ffe [id=9ff8ea75a9bc4de194a70fe2088a94bc]) successfully deleted
I20240502 08:16:40.989676 23898 tablet_service.cc:1467] Processing CreateTablet for tablet d16110e1b1664802a98182c73f1e41d3 (DEFAULT_TABLE table=default.Table_9feed0e1d82c472aaf0e9b92cd9173d5 [id=0f45bbbc044d4e15b40e0a7f9edf6ef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.989796 24180 tablet_replica.cc:330] T b11f504b93574e42a9bf30fde6d08207 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:40.990033 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d16110e1b1664802a98182c73f1e41d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.990041 24180 raft_consensus.cc:2229] T b11f504b93574e42a9bf30fde6d08207 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:40.990235 24180 raft_consensus.cc:2258] T b11f504b93574e42a9bf30fde6d08207 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:40.991688 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 76e0f767c6534a608023e6e6d33ca7a8 (DEFAULT_TABLE table=default.Table_7929569b25404ea08efa4875a3c241a3 [id=51db0a1374374028806594b722a42fcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.992128 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e0f767c6534a608023e6e6d33ca7a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:40.992199 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 3a74a6eb6d214d858903b3307e09e9e6 (DEFAULT_TABLE table=default.Table_c22209bdcbfd4dfa9e643ab9c98dc005 [id=232529911c8d43e2860c52d5527597d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:40.992870 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a74a6eb6d214d858903b3307e09e9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.004216 23749 tablet_service.cc:1467] Processing CreateTablet for tablet bb4b4a4d6a364de8af3b54d99a0ae138 (DEFAULT_TABLE table=default.Table_86918f47dba74a9eb36be5145a017f5b [id=8dbcb2bdd1b14db192a3bd5dcc32fc9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.004680 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4b4a4d6a364de8af3b54d99a0ae138. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.011698 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 72ee52bcc61b4816acda5b8140b63504 (DEFAULT_TABLE table=default.Table_d035b24dfb644b7aa77b160ee28d36e5 [id=19ef62dadf8943bdb06f82727634f764]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.011935 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 76e0f767c6534a608023e6e6d33ca7a8 (DEFAULT_TABLE table=default.Table_7929569b25404ea08efa4875a3c241a3 [id=51db0a1374374028806594b722a42fcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.012022 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 72ee52bcc61b4816acda5b8140b63504 (DEFAULT_TABLE table=default.Table_d035b24dfb644b7aa77b160ee28d36e5 [id=19ef62dadf8943bdb06f82727634f764]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.012104 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ee52bcc61b4816acda5b8140b63504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.012431 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e0f767c6534a608023e6e6d33ca7a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.014039 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 76e0f767c6534a608023e6e6d33ca7a8 (DEFAULT_TABLE table=default.Table_7929569b25404ea08efa4875a3c241a3 [id=51db0a1374374028806594b722a42fcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.014577 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e0f767c6534a608023e6e6d33ca7a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.017773 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 72ee52bcc61b4816acda5b8140b63504 (DEFAULT_TABLE table=default.Table_d035b24dfb644b7aa77b160ee28d36e5 [id=19ef62dadf8943bdb06f82727634f764]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.018234 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ee52bcc61b4816acda5b8140b63504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.019582 24029 tablet_service.cc:1467] Processing CreateTablet for tablet e9171ebf74424c9aae5e1c8a5118f768 (DEFAULT_TABLE table=default.Table_b76e1d838fa444179e6dbaa30058595a [id=3711d24a9e544db98c2bda42becc3f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.019999 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9171ebf74424c9aae5e1c8a5118f768. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.020620 24038 tablet_service.cc:1467] Processing CreateTablet for tablet bb4b4a4d6a364de8af3b54d99a0ae138 (DEFAULT_TABLE table=default.Table_86918f47dba74a9eb36be5145a017f5b [id=8dbcb2bdd1b14db192a3bd5dcc32fc9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.021065 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4b4a4d6a364de8af3b54d99a0ae138. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.022068 23899 tablet_service.cc:1467] Processing CreateTablet for tablet bb4b4a4d6a364de8af3b54d99a0ae138 (DEFAULT_TABLE table=default.Table_86918f47dba74a9eb36be5145a017f5b [id=8dbcb2bdd1b14db192a3bd5dcc32fc9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.022454 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4b4a4d6a364de8af3b54d99a0ae138. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.029637 23889 tablet_service.cc:1467] Processing CreateTablet for tablet e9171ebf74424c9aae5e1c8a5118f768 (DEFAULT_TABLE table=default.Table_b76e1d838fa444179e6dbaa30058595a [id=3711d24a9e544db98c2bda42becc3f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.030059 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9171ebf74424c9aae5e1c8a5118f768. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.031356 23608 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 23ce83c8001d4b2988b88c0ed9bf948c (table default.Table_190eae2a8c64440f8d19842f8b6ac254 [id=54454efbe9394170822f237dc8e6115e]). Expected version 1 got 0
I20240502 08:16:41.031611 23608 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet b6c52f1cae3a4ad38a9f95bfaf055e76 (table default.Table_35b808b0487f463dbcfa4a773a2a20f0 [id=922fb979b5894023a68820aece0b8c5e]). Expected version 1 got 0
I20240502 08:16:41.035334 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3c00e5393c47df924bcb5dcf2a0f19 (DEFAULT_TABLE table=default.Table_a44933d855bd48aa8865c11eaaa5cbe4 [id=7227b8f1e8164cc8aee8ded7c5917ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.035849 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3c00e5393c47df924bcb5dcf2a0f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.043373 24031 tablet_service.cc:1467] Processing CreateTablet for tablet b2eac3bde41c4415bdb97fc53a5b395f (DEFAULT_TABLE table=default.Table_2d9b44d9d9d34aa184855169eb278e94 [id=f19f154c950e4a228766d5189c345eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.043743 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2eac3bde41c4415bdb97fc53a5b395f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.053583 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 366) took 1146 ms. Trace:
I20240502 08:16:41.053773 24179 rpcz_store.cc:276] 0502 08:16:39.836072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.836449 (+ 377us) service_pool.cc:225] Handling call
0502 08:16:40.981800 (+1145351us) ts_tablet_manager.cc:1120] Deleting tablet ee022e8e86f8477a8fa1e58d52644b03
0502 08:16:40.981810 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:40.981901 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1136899}
I20240502 08:16:41.054450 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 370) took 1207 ms. Trace:
I20240502 08:16:41.054589 24179 rpcz_store.cc:276] 0502 08:16:39.846837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.847381 (+ 544us) service_pool.cc:225] Handling call
0502 08:16:41.054243 (+1206862us) ts_tablet_manager.cc:1120] Deleting tablet 25c089f0c1b14e419f250d03d4eaad12
0502 08:16:41.054252 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:41.054373 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1203641}
I20240502 08:16:41.055004 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3c00e5393c47df924bcb5dcf2a0f19 (DEFAULT_TABLE table=default.Table_a44933d855bd48aa8865c11eaaa5cbe4 [id=7227b8f1e8164cc8aee8ded7c5917ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.055486 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3c00e5393c47df924bcb5dcf2a0f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.057080 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 3a74a6eb6d214d858903b3307e09e9e6 (DEFAULT_TABLE table=default.Table_c22209bdcbfd4dfa9e643ab9c98dc005 [id=232529911c8d43e2860c52d5527597d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.057529 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a74a6eb6d214d858903b3307e09e9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.060194 24181 ts_tablet_manager.cc:1888] T 10adabdd31f848548ef6f2408ccc2050 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.055119 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 364) took 1231 ms. Trace:
I20240502 08:16:41.061378 24179 rpcz_store.cc:276] 0502 08:16:39.823247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.823451 (+ 204us) service_pool.cc:225] Handling call
0502 08:16:41.054959 (+1231508us) ts_tablet_manager.cc:1120] Deleting tablet aacdf506221044ffada0fec0c7d42d0b
0502 08:16:41.054970 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:41.055066 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1202982}
I20240502 08:16:41.062012 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 371) took 1212 ms. Trace:
I20240502 08:16:41.062171 24179 rpcz_store.cc:276] 0502 08:16:39.849779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.851750 (+ 1971us) service_pool.cc:225] Handling call
0502 08:16:41.061803 (+1210053us) ts_tablet_manager.cc:1120] Deleting tablet 751808dfc1964b539f6f836d786156aa
0502 08:16:41.061809 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:41.061965 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1209494}
I20240502 08:16:41.062333 24181 log.cc:1196] T 10adabdd31f848548ef6f2408ccc2050 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/10adabdd31f848548ef6f2408ccc2050
I20240502 08:16:41.062762 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 365) took 1237 ms. Trace:
I20240502 08:16:41.062861 24181 ts_tablet_manager.cc:1909] T 10adabdd31f848548ef6f2408ccc2050 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:41.062911 24179 rpcz_store.cc:276] 0502 08:16:39.824879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.825012 (+ 133us) service_pool.cc:225] Handling call
0502 08:16:41.062560 (+1237548us) ts_tablet_manager.cc:1120] Deleting tablet bcfacc302ff04178bfffc48111cd7a3f
0502 08:16:41.062568 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:41.062701 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1209845}
I20240502 08:16:41.063458 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 369) took 1226 ms. Trace:
I20240502 08:16:41.063655 24179 rpcz_store.cc:276] 0502 08:16:39.836968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:39.837143 (+ 175us) service_pool.cc:225] Handling call
0502 08:16:41.063305 (+1226162us) ts_tablet_manager.cc:1120] Deleting tablet 4b8d81f071d74b218fcfd4f4e8c20bab
0502 08:16:41.063315 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:41.063407 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1210392}
I20240502 08:16:41.064951 24179 tablet_replica.cc:330] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:41.065208 24179 raft_consensus.cc:2229] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.065497 24179 raft_consensus.cc:2258] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.014993 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3c00e5393c47df924bcb5dcf2a0f19 (DEFAULT_TABLE table=default.Table_a44933d855bd48aa8865c11eaaa5cbe4 [id=7227b8f1e8164cc8aee8ded7c5917ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.016175 23750 tablet_service.cc:1467] Processing CreateTablet for tablet b2eac3bde41c4415bdb97fc53a5b395f (DEFAULT_TABLE table=default.Table_2d9b44d9d9d34aa184855169eb278e94 [id=f19f154c950e4a228766d5189c345eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.017083 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 5e56af7b42cb481a8363d60f49ee3f61 (DEFAULT_TABLE table=default.Table_94efd28ada5b4024ab65e4d5feda0b38 [id=15b42ca7619645c8ac3cff50044f069b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.013796 23749 tablet_service.cc:1467] Processing CreateTablet for tablet e9171ebf74424c9aae5e1c8a5118f768 (DEFAULT_TABLE table=default.Table_b76e1d838fa444179e6dbaa30058595a [id=3711d24a9e544db98c2bda42becc3f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.018653 24174 tablet_bootstrap.cc:492] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:41.064630 24180 ts_tablet_manager.cc:1875] T b11f504b93574e42a9bf30fde6d08207 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.067971 24174 ts_tablet_manager.cc:1367] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.294s user 0.003s sys 0.001s
I20240502 08:16:41.069782 24174 raft_consensus.cc:348] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.070214 24181 tablet_replica.cc:330] T c470d420c45843a69a63d9800b7cfc97 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:41.070411 24174 raft_consensus.cc:374] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.070505 24181 raft_consensus.cc:2229] T c470d420c45843a69a63d9800b7cfc97 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.070721 24181 raft_consensus.cc:2258] T c470d420c45843a69a63d9800b7cfc97 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.070724 24174 raft_consensus.cc:729] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.071545 24181 ts_tablet_manager.cc:1875] T c470d420c45843a69a63d9800b7cfc97 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.071411 24174 consensus_queue.cc:260] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.074940 24175 tablet_bootstrap.cc:492] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:41.075165 24175 ts_tablet_manager.cc:1367] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.101s user 0.001s sys 0.003s
I20240502 08:16:41.076007 24175 raft_consensus.cc:348] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.076313 24175 raft_consensus.cc:374] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.076447 24175 raft_consensus.cc:729] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.076611 24174 ts_tablet_manager.cc:1398] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240502 08:16:41.076896 24175 consensus_queue.cc:260] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.077590 24181 ts_tablet_manager.cc:1888] T c470d420c45843a69a63d9800b7cfc97 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.079916 24174 tablet_bootstrap.cc:492] T 10adabdd31f848548ef6f2408ccc2050 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:41.012378 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ee52bcc61b4816acda5b8140b63504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.080564 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e56af7b42cb481a8363d60f49ee3f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.080739 24175 ts_tablet_manager.cc:1398] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:16:41.085742 24181 log.cc:1196] T c470d420c45843a69a63d9800b7cfc97 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/c470d420c45843a69a63d9800b7cfc97
I20240502 08:16:41.085783 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3c00e5393c47df924bcb5dcf2a0f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.086123 24175 tablet_bootstrap.cc:492] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:41.086207 24181 ts_tablet_manager.cc:1909] T c470d420c45843a69a63d9800b7cfc97 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:41.087369 24181 tablet_replica.cc:330] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:41.087630 24181 raft_consensus.cc:2229] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.087806 24175 tablet_bootstrap.cc:654] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.087852 24181 raft_consensus.cc:2258] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.088675 24181 ts_tablet_manager.cc:1875] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.089890 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2eac3bde41c4415bdb97fc53a5b395f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.095651 24181 ts_tablet_manager.cc:1888] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.097723 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9171ebf74424c9aae5e1c8a5118f768. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.101425 24180 ts_tablet_manager.cc:1888] T b11f504b93574e42a9bf30fde6d08207 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.101650 24180 log.cc:1196] T b11f504b93574e42a9bf30fde6d08207 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/b11f504b93574e42a9bf30fde6d08207
I20240502 08:16:41.102155 24180 ts_tablet_manager.cc:1909] T b11f504b93574e42a9bf30fde6d08207 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:41.102805 24181 log.cc:1196] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/21832f0151ec4cf48f51f9bd2cb2b5ec
I20240502 08:16:41.103116 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet b11f504b93574e42a9bf30fde6d08207 (table default.Table_ec5e04e87fba492ab9a96d4034db240a [id=7900f407998344dfa30c1468428ef42a]) successfully deleted
I20240502 08:16:41.103186 24181 ts_tablet_manager.cc:1909] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:41.104643 24180 tablet_replica.cc:330] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:41.105100 24180 raft_consensus.cc:2229] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.105749 24180 raft_consensus.cc:2258] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.107043 24180 ts_tablet_manager.cc:1875] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.110327 24596 raft_consensus.cc:482] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:41.110467 24596 raft_consensus.cc:504] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.111066 24596 leader_election.cc:288] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:41.112103 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6788ffd2eb2248c3b84e474de56c37a9" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:41.112715 23723 leader_election.cc:341] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:41.113413 23723 leader_election.cc:341] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 6788ffd2eb2248c3b84e474de56c37a9
I20240502 08:16:41.113548 23723 leader_election.cc:302] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:41.114130 24596 raft_consensus.cc:2726] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:41.115350 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 5e56af7b42cb481a8363d60f49ee3f61 (DEFAULT_TABLE table=default.Table_94efd28ada5b4024ab65e4d5feda0b38 [id=15b42ca7619645c8ac3cff50044f069b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.115811 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e56af7b42cb481a8363d60f49ee3f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.116027 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 5e56af7b42cb481a8363d60f49ee3f61 (DEFAULT_TABLE table=default.Table_94efd28ada5b4024ab65e4d5feda0b38 [id=15b42ca7619645c8ac3cff50044f069b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.116421 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e56af7b42cb481a8363d60f49ee3f61. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:41.118151 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet ee022e8e86f8477a8fa1e58d52644b03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee022e8e86f8477a8fa1e58d52644b03 already in progress: creating tablet
W20240502 08:16:41.118465 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 25c089f0c1b14e419f250d03d4eaad12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25c089f0c1b14e419f250d03d4eaad12 already in progress: creating tablet
W20240502 08:16:41.119048 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet aacdf506221044ffada0fec0c7d42d0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aacdf506221044ffada0fec0c7d42d0b already in progress: creating tablet
W20240502 08:16:41.119482 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 751808dfc1964b539f6f836d786156aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 751808dfc1964b539f6f836d786156aa already in progress: creating tablet
W20240502 08:16:41.119735 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet bcfacc302ff04178bfffc48111cd7a3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcfacc302ff04178bfffc48111cd7a3f already in progress: creating tablet
W20240502 08:16:41.119987 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 4b8d81f071d74b218fcfd4f4e8c20bab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b8d81f071d74b218fcfd4f4e8c20bab already in progress: creating tablet
W20240502 08:16:41.120254 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 9c072eb45e1b41eeb5aafa6d5a786944 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c072eb45e1b41eeb5aafa6d5a786944 already in progress: creating tablet
I20240502 08:16:41.121080 23898 tablet_service.cc:1467] Processing CreateTablet for tablet b2eac3bde41c4415bdb97fc53a5b395f (DEFAULT_TABLE table=default.Table_2d9b44d9d9d34aa184855169eb278e94 [id=f19f154c950e4a228766d5189c345eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.121977 24025 tablet_service.cc:1467] Processing CreateTablet for tablet b6d8410ba7cc4b03bb64a138b1f9bb14 (DEFAULT_TABLE table=default.Table_b9b22cc686024294ab1843b25cb74af7 [id=2c5d4e40edf045719a60cb12646c5868]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.122428 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d8410ba7cc4b03bb64a138b1f9bb14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.125962 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet b11f504b93574e42a9bf30fde6d08207 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.126204 24181 tablet_replica.cc:330] T b11f504b93574e42a9bf30fde6d08207 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:41.126394 24181 raft_consensus.cc:2229] T b11f504b93574e42a9bf30fde6d08207 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.126560 24181 raft_consensus.cc:2258] T b11f504b93574e42a9bf30fde6d08207 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.127491 24181 ts_tablet_manager.cc:1875] T b11f504b93574e42a9bf30fde6d08207 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.125774 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 8633864e248f416cb651da3b0c2aa936 with delete_type TABLET_DATA_DELETED (replaced by tablet 9761df0bbdee44af90135bffb8b5828f) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:41.130309 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2eac3bde41c4415bdb97fc53a5b395f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.140504 24181 ts_tablet_manager.cc:1888] T b11f504b93574e42a9bf30fde6d08207 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.143149 24181 log.cc:1196] T b11f504b93574e42a9bf30fde6d08207 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/b11f504b93574e42a9bf30fde6d08207
I20240502 08:16:41.143602 24181 ts_tablet_manager.cc:1909] T b11f504b93574e42a9bf30fde6d08207 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:41.146190 24596 consensus_queue.cc:1035] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240502 08:16:41.151337 24179 ts_tablet_manager.cc:1875] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.154549 24551 raft_consensus.cc:482] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:41.154713 24551 raft_consensus.cc:504] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.155403 24551 leader_election.cc:288] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:41.156545 23858 leader_election.cc:341] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 43e870c3a0e340d1bad1c1fa0380a762
W20240502 08:16:41.156677 23861 leader_election.cc:341] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 43e870c3a0e340d1bad1c1fa0380a762
I20240502 08:16:41.156793 23861 leader_election.cc:302] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:41.157054 24551 raft_consensus.cc:2726] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:41.158546 24174 tablet_bootstrap.cc:654] T 10adabdd31f848548ef6f2408ccc2050 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.161921 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 91a3455e5931488da3134d6b5c18aef2 (DEFAULT_TABLE table=default.Table_d3c66c36f4cd494585b1e4cb114b81a9 [id=c0408e826b4d4bcd819c828dca1bd9cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.162411 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a3455e5931488da3134d6b5c18aef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.169351 23756 tablet_service.cc:1467] Processing CreateTablet for tablet b6d8410ba7cc4b03bb64a138b1f9bb14 (DEFAULT_TABLE table=default.Table_b9b22cc686024294ab1843b25cb74af7 [id=2c5d4e40edf045719a60cb12646c5868]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.169921 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d8410ba7cc4b03bb64a138b1f9bb14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.175824 24489 consensus_queue.cc:1035] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:41.122997 23605 catalog_manager.cc:5484] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:41.188228 24173 tablet_bootstrap.cc:492] T d8f7b5f018d84680ac5533bf99102d31 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:41.188403 24173 ts_tablet_manager.cc:1367] T d8f7b5f018d84680ac5533bf99102d31 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.275s user 0.087s sys 0.063s
I20240502 08:16:41.188402 23605 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 23ce83c8001d4b2988b88c0ed9bf948c (table default.Table_190eae2a8c64440f8d19842f8b6ac254 [id=54454efbe9394170822f237dc8e6115e]). Expected version 1 got 0
I20240502 08:16:41.124907 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9d47c3a9169244efb08e234d37647e64" } new_table_name: "default.Table_2edad641920b44ce9a4d5b7503f0279a" modify_external_catalogs: true
I20240502 08:16:41.189147 24173 raft_consensus.cc:348] T d8f7b5f018d84680ac5533bf99102d31 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.189337 24173 raft_consensus.cc:374] T d8f7b5f018d84680ac5533bf99102d31 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.189406 24173 raft_consensus.cc:729] T d8f7b5f018d84680ac5533bf99102d31 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.129339 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_93bd02b663b8448aa250379544e73dea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:41.189669 24173 consensus_queue.cc:260] T d8f7b5f018d84680ac5533bf99102d31 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:41.190058 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93bd02b663b8448aa250379544e73dea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:41.190233 24173 ts_tablet_manager.cc:1398] T d8f7b5f018d84680ac5533bf99102d31 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:41.190476 24173 tablet_bootstrap.cc:492] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:41.134150 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 045b0882a40f4eaba94935d9bc90da0c has no consensus state
I20240502 08:16:41.191722 24180 ts_tablet_manager.cc:1888] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.191879 24180 log.cc:1196] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/ce758c4a9258427987e8e83f579a2aa2
I20240502 08:16:41.191884 24173 tablet_bootstrap.cc:654] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.192237 24180 ts_tablet_manager.cc:1909] T ce758c4a9258427987e8e83f579a2aa2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:41.135468 23607 master_service.cc:946] ReplaceTablet: received request to replace tablet 5d04312bd81f46ed8c49c42b1af1aec9 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:41.143410 23606 master_service.cc:946] ReplaceTablet: received request to replace tablet 955e7bd2216646c6aef5acce8d5b0d77 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:41.193230 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet ce758c4a9258427987e8e83f579a2aa2 (table default.Table_cb0ea06f5e0b47e5978fc9a5ed8880ce [id=9965057f98614e8284d007156af6fdd0]) successfully deleted
I20240502 08:16:41.193637 24571 raft_consensus.cc:482] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:41.193823 24571 raft_consensus.cc:504] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.194700 24571 leader_election.cc:288] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:41.195688 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e134e90efb4a48bb82b53a80bda708b3" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:41.195963 23786 raft_consensus.cc:2454] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:41.196771 23993 leader_election.cc:302] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:41.199723 24571 raft_consensus.cc:2781] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:41.199826 24571 raft_consensus.cc:482] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:41.199981 24571 raft_consensus.cc:3037] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:41.203090 24571 raft_consensus.cc:504] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.203882 24173 tablet_bootstrap.cc:492] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:41.204056 23921 raft_consensus.cc:3037] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:41.204116 24173 ts_tablet_manager.cc:1367] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240502 08:16:41.204914 24173 raft_consensus.cc:348] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.205138 24173 raft_consensus.cc:374] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.205202 24173 raft_consensus.cc:729] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.205453 24173 consensus_queue.cc:260] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.205667 24180 tablet_replica.cc:330] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:41.205890 24180 raft_consensus.cc:2229] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.208549 24596 consensus_queue.cc:1035] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:41.209183 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e134e90efb4a48bb82b53a80bda708b3" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:41.209398 23786 raft_consensus.cc:3037] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:41.209760 24180 raft_consensus.cc:2258] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.210361 24180 ts_tablet_manager.cc:1875] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.212432 23786 raft_consensus.cc:2454] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:41.213306 23993 leader_election.cc:302] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:41.214792 24642 raft_consensus.cc:2781] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:41.215375 24173 ts_tablet_manager.cc:1398] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.011s user 0.003s sys 0.002s
I20240502 08:16:41.215775 24173 tablet_bootstrap.cc:492] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:41.216120 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e134e90efb4a48bb82b53a80bda708b3" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:41.217479 24173 tablet_bootstrap.cc:654] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.217773 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e134e90efb4a48bb82b53a80bda708b3" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:41.218767 23996 leader_election.cc:341] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:41.219348 24642 raft_consensus.cc:686] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
W20240502 08:16:41.219517 23996 leader_election.cc:341] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:41.219733 24642 consensus_queue.cc:237] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.220351 24571 leader_election.cc:288] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:41.227159 24551 raft_consensus.cc:482] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:41.227435 24551 raft_consensus.cc:504] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.228329 24551 leader_election.cc:288] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:41.230448 23861 leader_election.cc:341] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: d5bab1b7c48248a7973c27d0a865b34b
W20240502 08:16:41.231130 23858 leader_election.cc:341] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: d5bab1b7c48248a7973c27d0a865b34b
I20240502 08:16:41.231298 23858 leader_election.cc:302] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:41.231784 24173 tablet_bootstrap.cc:492] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:41.231876 24551 raft_consensus.cc:2726] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:41.231984 24173 ts_tablet_manager.cc:1367] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.005s
I20240502 08:16:41.232694 24173 raft_consensus.cc:348] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.232971 24173 raft_consensus.cc:374] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.233105 24173 raft_consensus.cc:729] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.233351 24173 consensus_queue.cc:260] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.233925 24173 ts_tablet_manager.cc:1398] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:41.234341 24173 tablet_bootstrap.cc:492] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:41.236129 24173 tablet_bootstrap.cc:654] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:41.237936 23597 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 117644us
W20240502 08:16:41.238284 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet eb96ad2289754e428a1c109b2d59c53b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb96ad2289754e428a1c109b2d59c53b already in progress: creating tablet
W20240502 08:16:41.238605 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 24a4104f4cb64889ad66810788ad6b29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24a4104f4cb64889ad66810788ad6b29 already in progress: creating tablet
I20240502 08:16:41.238898 24596 raft_consensus.cc:482] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:41.239174 24596 raft_consensus.cc:504] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.240036 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 10adabdd31f848548ef6f2408ccc2050 (table default.Table_c45e7b653ae641b293b87c150748c864 [id=5ff5989d908243c9a58013285842dd46]) successfully deleted
I20240502 08:16:41.240235 24551 raft_consensus.cc:482] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:41.240303 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet c470d420c45843a69a63d9800b7cfc97 (table default.Table_95acc6e36a0b448eb3fd6d5df86e2de4 [id=1328b5d30ef24f98bf00c4429383ce92]) successfully deleted
I20240502 08:16:41.240456 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 21832f0151ec4cf48f51f9bd2cb2b5ec (table default.Table_75cadbecaad04383945a6e0b1e226d86 [id=4c539e8625094a278029356ac44da4c0]) successfully deleted
I20240502 08:16:41.240378 24551 raft_consensus.cc:504] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.241034 24551 leader_election.cc:288] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:41.241951 23861 leader_election.cc:341] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: f51b26043ef44b30846498cd1497607c
I20240502 08:16:41.241979 24596 leader_election.cc:288] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:41.242543 23723 leader_election.cc:341] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 0be4f8d5cc09483d8d1aa1e37679b544
W20240502 08:16:41.242897 23858 leader_election.cc:341] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: f51b26043ef44b30846498cd1497607c
I20240502 08:16:41.243052 23858 leader_election.cc:302] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:41.243803 24551 raft_consensus.cc:2726] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:41.242826 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be4f8d5cc09483d8d1aa1e37679b544" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:41.244500 23723 leader_election.cc:341] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:41.244645 23723 leader_election.cc:302] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:41.246099 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae9c817c7be42da8c689267c7eb5a06 (DEFAULT_TABLE table=default.Table_70eb03c32f654f229093f610d22f9972 [id=d9f38d2a946c498f86eddd3d3277464d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.246596 24596 raft_consensus.cc:2726] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:41.248900 23749 tablet_service.cc:1467] Processing CreateTablet for tablet a91e2b50321e4241847d97a558b419f2 (DEFAULT_TABLE table=default.Table_5ad32437311f45d3a50a7fe4b134adb9 [id=243d930b07314a519d68fa40bd636226]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.250557 24173 tablet_bootstrap.cc:492] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:41.250741 24173 ts_tablet_manager.cc:1367] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20240502 08:16:41.251580 24173 raft_consensus.cc:348] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.251793 24173 raft_consensus.cc:374] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.251859 24173 raft_consensus.cc:729] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.252030 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c22209bdcbfd4dfa9e643ab9c98dc005" } modify_external_catalogs: true
I20240502 08:16:41.252110 24173 consensus_queue.cc:260] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.252257 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c22209bdcbfd4dfa9e643ab9c98dc005" } modify_external_catalogs: true
I20240502 08:16:41.252655 24173 ts_tablet_manager.cc:1398] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:41.252955 24173 tablet_bootstrap.cc:492] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:41.254592 24173 tablet_bootstrap.cc:654] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:41.256868 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dead13bba95f43cb809e17f6d7fba028 consensus state has no leader
I20240502 08:16:41.257493 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d4e98fc3f0334516b7074a56d74d8332"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:41.258033 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4e98fc3f0334516b7074a56d74d8332 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:41.278010 23993 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 64735.4us
I20240502 08:16:41.280603 24175 tablet_bootstrap.cc:492] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:41.280786 24175 ts_tablet_manager.cc:1367] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.195s user 0.015s sys 0.007s
I20240502 08:16:41.282235 24174 tablet_bootstrap.cc:492] T 10adabdd31f848548ef6f2408ccc2050 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:41.282407 24174 ts_tablet_manager.cc:1367] T 10adabdd31f848548ef6f2408ccc2050 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.203s user 0.004s sys 0.000s
I20240502 08:16:41.283329 24174 raft_consensus.cc:348] T 10adabdd31f848548ef6f2408ccc2050 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.283535 24174 raft_consensus.cc:374] T 10adabdd31f848548ef6f2408ccc2050 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.283630 24174 raft_consensus.cc:729] T 10adabdd31f848548ef6f2408ccc2050 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.283872 24174 consensus_queue.cc:260] T 10adabdd31f848548ef6f2408ccc2050 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.284387 24174 ts_tablet_manager.cc:1398] T 10adabdd31f848548ef6f2408ccc2050 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:41.284723 24174 tablet_bootstrap.cc:492] T dcdd765895254a39b86b692b5d812c83 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:41.288538 24630 consensus_queue.cc:1035] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:41.289116 24175 raft_consensus.cc:348] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.289353 24175 raft_consensus.cc:374] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.289446 24175 raft_consensus.cc:729] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.289747 24175 consensus_queue.cc:260] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.290449 24175 ts_tablet_manager.cc:1398] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 08:16:41.290777 24175 tablet_bootstrap.cc:492] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:41.292354 24175 tablet_bootstrap.cc:654] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.301805 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae9c817c7be42da8c689267c7eb5a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.301890 24180 ts_tablet_manager.cc:1888] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.302021 24180 log.cc:1196] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/3cc30014a1ac45de937751eed36877a9
I20240502 08:16:41.302466 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a91e2b50321e4241847d97a558b419f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.240939 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet b11f504b93574e42a9bf30fde6d08207 (table default.Table_ec5e04e87fba492ab9a96d4034db240a [id=7900f407998344dfa30c1468428ef42a]) successfully deleted
W20240502 08:16:41.303248 23597 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 62330.9us
I20240502 08:16:41.307400 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet d5bab1b7c48248a7973c27d0a865b34b with delete_type TABLET_DATA_DELETED (replaced by tablet 7c374f42f353497fa1d745164e7fbf61) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:41.307933 23886 tablet_service.cc:1467] Processing CreateTablet for tablet b6d8410ba7cc4b03bb64a138b1f9bb14 (DEFAULT_TABLE table=default.Table_b9b22cc686024294ab1843b25cb74af7 [id=2c5d4e40edf045719a60cb12646c5868]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.308418 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d8410ba7cc4b03bb64a138b1f9bb14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.309146 24180 ts_tablet_manager.cc:1909] T 3cc30014a1ac45de937751eed36877a9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:41.309899 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 91a3455e5931488da3134d6b5c18aef2 (DEFAULT_TABLE table=default.Table_d3c66c36f4cd494585b1e4cb114b81a9 [id=c0408e826b4d4bcd819c828dca1bd9cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.310230 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a3455e5931488da3134d6b5c18aef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.312503 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 33c8131846dc41678827b86e87be18d3
I20240502 08:16:41.313647 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 6e7432f1df94431a83cf68356e3e78a3
I20240502 08:16:41.314555 23888 tablet_service.cc:1467] Processing CreateTablet for tablet a91e2b50321e4241847d97a558b419f2 (DEFAULT_TABLE table=default.Table_5ad32437311f45d3a50a7fe4b134adb9 [id=243d930b07314a519d68fa40bd636226]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.315145 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a91e2b50321e4241847d97a558b419f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.317951 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae9c817c7be42da8c689267c7eb5a06 (DEFAULT_TABLE table=default.Table_70eb03c32f654f229093f610d22f9972 [id=d9f38d2a946c498f86eddd3d3277464d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.318372 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae9c817c7be42da8c689267c7eb5a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.321349 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 981b7a7292884b1fb48af99db6e762c0 (DEFAULT_TABLE table=default.Table_e01d80203b2347cb9f8941844cdcc6fe [id=a681d30fb3e5469aad1bc27dfdd56ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.321843 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981b7a7292884b1fb48af99db6e762c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.325938 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee764398b364d029ae7f0f575a30d64 (DEFAULT_TABLE table=default.Table_bc1e9d346aa846f8abca7504185f7cdb [id=8d27878c6c8f4234908a2c78cb8bbd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.326350 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee764398b364d029ae7f0f575a30d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.326795 24180 tablet_replica.cc:330] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:41.327047 24180 raft_consensus.cc:2229] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.327270 24180 raft_consensus.cc:2258] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.329725 24179 ts_tablet_manager.cc:1888] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:16:41.329851 24179 log.cc:1196] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/08183523bf4a4d1c9020eb693aa07878
I20240502 08:16:41.334383 24180 ts_tablet_manager.cc:1875] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.335455 24039 tablet_service.cc:1467] Processing CreateTablet for tablet a91e2b50321e4241847d97a558b419f2 (DEFAULT_TABLE table=default.Table_5ad32437311f45d3a50a7fe4b134adb9 [id=243d930b07314a519d68fa40bd636226]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.335945 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a91e2b50321e4241847d97a558b419f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.336695 24174 tablet_bootstrap.cc:654] T dcdd765895254a39b86b692b5d812c83 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.337927 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae9c817c7be42da8c689267c7eb5a06 (DEFAULT_TABLE table=default.Table_70eb03c32f654f229093f610d22f9972 [id=d9f38d2a946c498f86eddd3d3277464d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.338424 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae9c817c7be42da8c689267c7eb5a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.338771 24173 tablet_bootstrap.cc:492] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:41.338944 24173 ts_tablet_manager.cc:1367] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.086s user 0.027s sys 0.044s
I20240502 08:16:41.339812 24173 raft_consensus.cc:348] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.340040 24173 raft_consensus.cc:374] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.340157 24173 raft_consensus.cc:729] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.340880 24179 ts_tablet_manager.cc:1909] T 08183523bf4a4d1c9020eb693aa07878 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:41.340426 24173 consensus_queue.cc:260] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.342041 24173 ts_tablet_manager.cc:1398] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:41.341967 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 981b7a7292884b1fb48af99db6e762c0 (DEFAULT_TABLE table=default.Table_e01d80203b2347cb9f8941844cdcc6fe [id=a681d30fb3e5469aad1bc27dfdd56ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.342379 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981b7a7292884b1fb48af99db6e762c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.342418 24173 tablet_bootstrap.cc:492] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:41.344221 24173 tablet_bootstrap.cc:654] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.345894 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee764398b364d029ae7f0f575a30d64 (DEFAULT_TABLE table=default.Table_bc1e9d346aa846f8abca7504185f7cdb [id=8d27878c6c8f4234908a2c78cb8bbd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.346241 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee764398b364d029ae7f0f575a30d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.349723 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c8131846dc41678827b86e87be18d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.349959 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e7432f1df94431a83cf68356e3e78a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.352252 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 3cc30014a1ac45de937751eed36877a9 (table default.Table_d8aa2cd6c42444c48c383bf13e3728fd [id=9cbc066db19142d3bf0fd50cbe8fc91a]) successfully deleted
I20240502 08:16:41.353806 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c8131846dc41678827b86e87be18d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.354064 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e7432f1df94431a83cf68356e3e78a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.354379 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 386) took 1022 ms. Trace:
I20240502 08:16:41.354537 24179 rpcz_store.cc:276] 0502 08:16:40.331824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:40.331878 (+ 54us) service_pool.cc:225] Handling call
0502 08:16:41.064863 (+732985us) ts_tablet_manager.cc:1120] Deleting tablet 08183523bf4a4d1c9020eb693aa07878
0502 08:16:41.064871 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:41.267615 (+202744us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:41.329609 (+ 61994us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:41.354331 (+ 24722us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":732613}
I20240502 08:16:41.354943 24179 tablet_replica.cc:330] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:41.355162 24179 raft_consensus.cc:2229] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.355331 24179 raft_consensus.cc:2258] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.355942 24179 ts_tablet_manager.cc:1875] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.357729 23429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2vVFYb/build/asan/bin/kudu with pid 23507
I20240502 08:16:41.353806 24174 tablet_bootstrap.cc:492] T dcdd765895254a39b86b692b5d812c83 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:41.358445 24174 ts_tablet_manager.cc:1367] T dcdd765895254a39b86b692b5d812c83 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.000s
I20240502 08:16:41.359689 24173 tablet_bootstrap.cc:492] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:41.359862 24173 ts_tablet_manager.cc:1367] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.001s
I20240502 08:16:41.360558 24173 raft_consensus.cc:348] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.360754 24173 raft_consensus.cc:374] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.360807 24173 raft_consensus.cc:729] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.361007 24173 consensus_queue.cc:260] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.361496 24173 ts_tablet_manager.cc:1398] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:41.361832 24173 tablet_bootstrap.cc:492] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:41.363320 24173 tablet_bootstrap.cc:654] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.359346 24174 raft_consensus.cc:348] T dcdd765895254a39b86b692b5d812c83 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.363503 24174 raft_consensus.cc:374] T dcdd765895254a39b86b692b5d812c83 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.363586 24174 raft_consensus.cc:729] T dcdd765895254a39b86b692b5d812c83 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.364033 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e7432f1df94431a83cf68356e3e78a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:41.364269 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c8131846dc41678827b86e87be18d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:41.364722 24174 consensus_queue.cc:260] T dcdd765895254a39b86b692b5d812c83 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.367400 24174 ts_tablet_manager.cc:1398] T dcdd765895254a39b86b692b5d812c83 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 08:16:41.367733 24174 tablet_bootstrap.cc:492] T 4302fec5c51e4c6c841710bab50b2997 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:41.371245 24175 tablet_bootstrap.cc:492] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:41.371412 24175 ts_tablet_manager.cc:1367] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.081s user 0.002s sys 0.001s
I20240502 08:16:41.372177 24175 raft_consensus.cc:348] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.372401 24175 raft_consensus.cc:374] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.372463 24175 raft_consensus.cc:729] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.372671 24175 consensus_queue.cc:260] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.373112 24175 ts_tablet_manager.cc:1398] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:41.373387 24175 tablet_bootstrap.cc:492] T 6788ffd2eb2248c3b84e474de56c37a9 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:41.374658 24173 tablet_bootstrap.cc:492] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:41.374811 24173 ts_tablet_manager.cc:1367] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.002s
I20240502 08:16:41.375110 24174 tablet_bootstrap.cc:654] T 4302fec5c51e4c6c841710bab50b2997 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.375530 24173 raft_consensus.cc:348] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.375723 24173 raft_consensus.cc:374] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.375797 24173 raft_consensus.cc:729] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.375983 24173 consensus_queue.cc:260] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.376500 24173 ts_tablet_manager.cc:1398] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:41.376775 24173 tablet_bootstrap.cc:492] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:41.378306 24173 tablet_bootstrap.cc:654] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.380753 24175 tablet_bootstrap.cc:654] T 6788ffd2eb2248c3b84e474de56c37a9 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.390594 24179 ts_tablet_manager.cc:1888] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.393595 24174 tablet_bootstrap.cc:492] T 4302fec5c51e4c6c841710bab50b2997 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:41.399607 24174 ts_tablet_manager.cc:1367] T 4302fec5c51e4c6c841710bab50b2997 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20240502 08:16:41.401316 24174 raft_consensus.cc:348] T 4302fec5c51e4c6c841710bab50b2997 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.405761 24174 raft_consensus.cc:374] T 4302fec5c51e4c6c841710bab50b2997 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.406526 24174 raft_consensus.cc:729] T 4302fec5c51e4c6c841710bab50b2997 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.398371 24180 ts_tablet_manager.cc:1888] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.411339 24180 log.cc:1196] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/6788ffd2eb2248c3b84e474de56c37a9
I20240502 08:16:41.411849 24180 ts_tablet_manager.cc:1909] T 6788ffd2eb2248c3b84e474de56c37a9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:41.413266 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 6788ffd2eb2248c3b84e474de56c37a9 (table default.Table_59462d2ccf514f0da864562120217931 [id=62273209104a4a8e9f24a9a77d2a90f0]) successfully deleted
W20240502 08:16:41.413954 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 33c8131846dc41678827b86e87be18d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33c8131846dc41678827b86e87be18d3 already in progress: creating tablet
W20240502 08:16:41.414320 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 6e7432f1df94431a83cf68356e3e78a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e7432f1df94431a83cf68356e3e78a3 already in progress: creating tablet
I20240502 08:16:41.407447 24174 consensus_queue.cc:260] T 4302fec5c51e4c6c841710bab50b2997 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.417235 24174 ts_tablet_manager.cc:1398] T 4302fec5c51e4c6c841710bab50b2997 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240502 08:16:41.417594 24174 tablet_bootstrap.cc:492] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:41.418828 24179 log.cc:1196] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/f36ceb0222ef473a828ca795f393174d
I20240502 08:16:41.419905 24179 ts_tablet_manager.cc:1909] T f36ceb0222ef473a828ca795f393174d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:41.424495 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 388) took 1047 ms. Trace:
I20240502 08:16:41.427908 24174 tablet_bootstrap.cc:654] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:41.441828 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 33c8131846dc41678827b86e87be18d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33c8131846dc41678827b86e87be18d3 already in progress: creating tablet
W20240502 08:16:41.442142 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 6e7432f1df94431a83cf68356e3e78a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e7432f1df94431a83cf68356e3e78a3 already in progress: creating tablet
I20240502 08:16:41.442724 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 08183523bf4a4d1c9020eb693aa07878 (table default.Table_c72b83e8da844cb6a570c14f50293f34 [id=0a2f8dbb62a64b0a96931ab5f8c17540]) successfully deleted
I20240502 08:16:41.443774 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 91a3455e5931488da3134d6b5c18aef2 (DEFAULT_TABLE table=default.Table_d3c66c36f4cd494585b1e4cb114b81a9 [id=c0408e826b4d4bcd819c828dca1bd9cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.444303 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a3455e5931488da3134d6b5c18aef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.449880 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 981b7a7292884b1fb48af99db6e762c0 (DEFAULT_TABLE table=default.Table_e01d80203b2347cb9f8941844cdcc6fe [id=a681d30fb3e5469aad1bc27dfdd56ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.450383 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981b7a7292884b1fb48af99db6e762c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.451663 24174 tablet_bootstrap.cc:492] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:41.453138 24174 ts_tablet_manager.cc:1367] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.000s
I20240502 08:16:41.453630 24179 rpcz_store.cc:276] 0502 08:16:40.377364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:40.377456 (+ 92us) service_pool.cc:225] Handling call
0502 08:16:41.354859 (+977403us) ts_tablet_manager.cc:1120] Deleting tablet f36ceb0222ef473a828ca795f393174d
0502 08:16:41.354871 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:41.371169 (+ 16298us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:41.390470 (+ 19301us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:41.424406 (+ 33936us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":977077}
I20240502 08:16:41.454581 24179 tablet_replica.cc:330] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:41.454862 24179 raft_consensus.cc:2229] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.455099 24179 raft_consensus.cc:2258] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.455771 24179 ts_tablet_manager.cc:1875] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.456727 24174 raft_consensus.cc:348] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.449388 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb2d0a4898544f182ab8070d29a7940 with delete_type TABLET_DATA_DELETED (replaced by tablet 1cd8338500f54604abcd6d42702c2eca) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:41.457409 24174 raft_consensus.cc:374] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.457494 24174 raft_consensus.cc:729] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.458920 24174 consensus_queue.cc:260] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.462263 24174 ts_tablet_manager.cc:1398] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:16:41.462563 24174 tablet_bootstrap.cc:492] T b6417aafbdcf416ea7bff56bce5e6ddb P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:41.466958 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet f36ceb0222ef473a828ca795f393174d (table default.Table_4b967ef2b2fc4e2caa4219afcae841a6 [id=c32fc96889554c32bdf1d4550633e91f]) successfully deleted
W20240502 08:16:41.471088 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 1ba1baa2279045b3adc58d187a12cbf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ba1baa2279045b3adc58d187a12cbf3 already in progress: creating tablet
I20240502 08:16:41.471648 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee764398b364d029ae7f0f575a30d64 (DEFAULT_TABLE table=default.Table_bc1e9d346aa846f8abca7504185f7cdb [id=8d27878c6c8f4234908a2c78cb8bbd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.472147 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee764398b364d029ae7f0f575a30d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.472374 24174 tablet_bootstrap.cc:654] T b6417aafbdcf416ea7bff56bce5e6ddb P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.488082 24175 tablet_bootstrap.cc:492] T 6788ffd2eb2248c3b84e474de56c37a9 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
W20240502 08:16:41.490564 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ed919909289410e867881baab841f53 consensus state has no leader
I20240502 08:16:41.492216 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_09e8bcad70344895b5831638daf0c93c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:41.493220 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09e8bcad70344895b5831638daf0c93c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:41.493597 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ac9fe60ab8574ba9b6c57d9ffea15fb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:41.494204 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac9fe60ab8574ba9b6c57d9ffea15fb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:41.494347 23608 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet dead13bba95f43cb809e17f6d7fba028 (table default.Table_978594faab984d60ba03d29d40328a6c [id=7409c7e1a3534932a787559a679e7098]). Expected version 3 got 0
I20240502 08:16:41.494637 23608 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet b6c52f1cae3a4ad38a9f95bfaf055e76 (table default.Table_35b808b0487f463dbcfa4a773a2a20f0 [id=922fb979b5894023a68820aece0b8c5e]). Expected version 1 got 0
I20240502 08:16:41.496037 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d7af10ea88e34ce9b9e4565cb73c02de" } new_table_name: "default.Table_fbada8fc944c43eb8921c9296435a394" modify_external_catalogs: true
I20240502 08:16:41.498622 23604 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet b6c52f1cae3a4ad38a9f95bfaf055e76 (table default.Table_35b808b0487f463dbcfa4a773a2a20f0 [id=922fb979b5894023a68820aece0b8c5e]). Expected version 1 got 0
I20240502 08:16:41.501045 24643 raft_consensus.cc:482] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:41.501199 24643 raft_consensus.cc:504] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.501960 24643 leader_election.cc:288] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:41.502712 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36aadfa1d264f23bcafa91744aed36d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:41.503710 23993 leader_election.cc:341] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:41.504316 23921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36aadfa1d264f23bcafa91744aed36d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:41.505028 23996 leader_election.cc:341] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:41.505143 23996 leader_election.cc:302] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:41.506110 24643 raft_consensus.cc:2726] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:41.508908 24179 ts_tablet_manager.cc:1888] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.509075 24179 log.cc:1196] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/a7c3800fb2784a1dba12d01c56c226e5
I20240502 08:16:41.509168 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_37ab1c2d8931468482c9d99dd6536fe5" } modify_external_catalogs: true
I20240502 08:16:41.509383 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_37ab1c2d8931468482c9d99dd6536fe5" } modify_external_catalogs: true
I20240502 08:16:41.509503 24173 tablet_bootstrap.cc:492] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:41.509536 24179 ts_tablet_manager.cc:1909] T a7c3800fb2784a1dba12d01c56c226e5 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:41.509793 24173 ts_tablet_manager.cc:1367] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.133s user 0.018s sys 0.030s
I20240502 08:16:41.510583 24173 raft_consensus.cc:348] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.510771 24179 tablet_replica.cc:330] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:41.510921 24173 raft_consensus.cc:374] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.511000 24179 raft_consensus.cc:2229] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.511055 24173 raft_consensus.cc:729] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.511184 24179 raft_consensus.cc:2258] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.511412 24173 consensus_queue.cc:260] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.511644 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet a7c3800fb2784a1dba12d01c56c226e5 (table default.Table_295f41167e6e481ea2443d5f8d02a998 [id=73559755c4594bcfa591d054964154ae]) successfully deleted
I20240502 08:16:41.511739 24179 ts_tablet_manager.cc:1875] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.512001 24173 ts_tablet_manager.cc:1398] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20240502 08:16:41.512168 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 78ddfc7b74644a0bbf8222d5f90e6658 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78ddfc7b74644a0bbf8222d5f90e6658 already in progress: creating tablet
I20240502 08:16:41.512360 24173 tablet_bootstrap.cc:492] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:41.513983 24173 tablet_bootstrap.cc:654] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.516918 24175 ts_tablet_manager.cc:1367] T 6788ffd2eb2248c3b84e474de56c37a9 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.144s user 0.010s sys 0.014s
I20240502 08:16:41.519408 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 4302fec5c51e4c6c841710bab50b2997 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.520670 24180 tablet_replica.cc:330] T 4302fec5c51e4c6c841710bab50b2997 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:41.520893 24180 raft_consensus.cc:2229] T 4302fec5c51e4c6c841710bab50b2997 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.521111 24180 raft_consensus.cc:2258] T 4302fec5c51e4c6c841710bab50b2997 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.521842 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 5d04312bd81f46ed8c49c42b1af1aec9
I20240502 08:16:41.521934 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet dcdd765895254a39b86b692b5d812c83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.522353 23607 catalog_manager.cc:6554] ReplaceTablet: tablet 5d04312bd81f46ed8c49c42b1af1aec9 deleted and replaced by tablet b45f3233dae94be1a1d0d8fb426d7f2c
I20240502 08:16:41.523013 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 3a74a6eb6d214d858903b3307e09e9e6
W20240502 08:16:41.523115 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c45aa0dd26524e0fba27e4eaf17d704e consensus state has no leader
I20240502 08:16:41.523862 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a74a6eb6d214d858903b3307e09e9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.523993 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 955e7bd2216646c6aef5acce8d5b0d77
I20240502 08:16:41.524278 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d04312bd81f46ed8c49c42b1af1aec9 with delete_type TABLET_DATA_DELETED (replaced by tablet b45f3233dae94be1a1d0d8fb426d7f2c) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.524470 23606 catalog_manager.cc:6554] ReplaceTablet: tablet 955e7bd2216646c6aef5acce8d5b0d77 deleted and replaced by tablet 4645143a00d440a7b6d56c5e30d249f1
I20240502 08:16:41.524710 23606 master_service.cc:946] ReplaceTablet: received request to replace tablet 0be4f8d5cc09483d8d1aa1e37679b544 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:41.524933 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 955e7bd2216646c6aef5acce8d5b0d77 with delete_type TABLET_DATA_DELETED (replaced by tablet 4645143a00d440a7b6d56c5e30d249f1) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.525485 24179 ts_tablet_manager.cc:1888] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.525661 24180 ts_tablet_manager.cc:1875] T 4302fec5c51e4c6c841710bab50b2997 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.527354 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 0be4f8d5cc09483d8d1aa1e37679b544 with delete_type TABLET_DATA_DELETED (replaced by tablet e686db13bb7541c69ae88f0fb7e27061) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.528105 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 10adabdd31f848548ef6f2408ccc2050 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.534447 23604 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet dead13bba95f43cb809e17f6d7fba028 (table default.Table_978594faab984d60ba03d29d40328a6c [id=7409c7e1a3534932a787559a679e7098]). Expected version 3 got 0
I20240502 08:16:41.534850 23604 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet b844458ae73f413cbf2113027a5af359 (table default.Table_0a4ff982972e4fdaae1583619c9d6b04 [id=fd93259a378b4ea582a6661a59104abb]). Expected version 1 got 0
I20240502 08:16:41.535203 23604 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 7a19310948e248d1a85479c41c61bda2 (table default.Table_785d5ddac96746379d1117bc8379566b [id=8521bfc13ef040a78cfb2cc4501b7f04]). Expected version 4 got 0
I20240502 08:16:41.535638 23604 catalog_manager.cc:5484] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67). New cstate: current_term: 1 leader_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:41.536367 23604 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet e134e90efb4a48bb82b53a80bda708b3 (table default.Table_cc80d1cedafa4e71906e5e9530fdd828 [id=47682c26a85842c7b8991bf3828a9336]). Expected version 1 got 0
I20240502 08:16:41.537387 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d04312bd81f46ed8c49c42b1af1aec9 with delete_type TABLET_DATA_DELETED (replaced by tablet b45f3233dae94be1a1d0d8fb426d7f2c) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.538322 23604 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet b6c52f1cae3a4ad38a9f95bfaf055e76 (table default.Table_35b808b0487f463dbcfa4a773a2a20f0 [id=922fb979b5894023a68820aece0b8c5e]). Expected version 1 got 0
I20240502 08:16:41.538497 23604 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 49b816ea1d474f2db94119441146ba78 (table default.Table_20df1b504848439c869ae3739ac54185 [id=a0606a4053d04dd3814f09419a4a70c2]). Expected version 1 got 0
I20240502 08:16:41.539922 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 955e7bd2216646c6aef5acce8d5b0d77 with delete_type TABLET_DATA_DELETED (replaced by tablet 4645143a00d440a7b6d56c5e30d249f1) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.540381 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a74a6eb6d214d858903b3307e09e9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.541961 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_dbb8428a8edc4d03ab61e66b62e1890d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:41.538961 23610 master_service.cc:946] ReplaceTablet: received request to replace tablet 2b98cc8728c042f4b7f6496fcc5ef913 from {username='slave'} at 127.0.0.1:36668
W20240502 08:16:41.542682 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbb8428a8edc4d03ab61e66b62e1890d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:41.544324 24173 tablet_bootstrap.cc:492] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:41.544494 24173 ts_tablet_manager.cc:1367] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.002s
I20240502 08:16:41.544596 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_f2608186d7f5434eb5f56e2ecdbdccfe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:41.545142 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2608186d7f5434eb5f56e2ecdbdccfe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:41.545339 24173 raft_consensus.cc:348] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.545534 24173 raft_consensus.cc:374] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.545671 24173 raft_consensus.cc:729] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.545925 24173 consensus_queue.cc:260] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.546069 24630 raft_consensus.cc:482] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:41.546263 24630 raft_consensus.cc:504] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.546526 24173 ts_tablet_manager.cc:1398] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:16:41.546854 24173 tablet_bootstrap.cc:492] T 8810b6f61db14fed8b12d36fed413dc8 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:41.547067 24630 leader_election.cc:288] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:41.548830 24173 tablet_bootstrap.cc:654] T 8810b6f61db14fed8b12d36fed413dc8 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:41.549225 23993 connection.cc:535] client connection to 127.22.225.125:36189 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:16:41.549328 23594 connection.cc:535] client connection to 127.22.225.125:36189 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:16:41.549499 23594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 8 similar messages]
I20240502 08:16:41.549535 23576 master_runner.cc:186] Error getting master registration for 127.22.225.125:36189: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107),
W20240502 08:16:41.549547 23720 connection.cc:535] client connection to 127.22.225.125:36189 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:16:41.549708 23576 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240502 08:16:41.556397 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 2b98cc8728c042f4b7f6496fcc5ef913
I20240502 08:16:41.540964 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6fd5def10796410c9fe46e557794e914"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:41.557049 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6fd5def10796410c9fe46e557794e914 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:41.541249 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 5d04312bd81f46ed8c49c42b1af1aec9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d04312bd81f46ed8c49c42b1af1aec9 already in progress: creating tablet
W20240502 08:16:41.558450 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 955e7bd2216646c6aef5acce8d5b0d77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 955e7bd2216646c6aef5acce8d5b0d77 already in progress: creating tablet
W20240502 08:16:41.558718 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 3a74a6eb6d214d858903b3307e09e9e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a74a6eb6d214d858903b3307e09e9e6 already in progress: creating tablet
I20240502 08:16:41.541536 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7f44f7f1aed6460cb1e73392da0987b8" } modify_external_catalogs: true
I20240502 08:16:41.559085 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7f44f7f1aed6460cb1e73392da0987b8" } modify_external_catalogs: true
I20240502 08:16:41.560389 23610 catalog_manager.cc:6554] ReplaceTablet: tablet 2b98cc8728c042f4b7f6496fcc5ef913 deleted and replaced by tablet 25c8a6aaabe2413d8ce88f54ce45a4c4
I20240502 08:16:41.560827 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b98cc8728c042f4b7f6496fcc5ef913 with delete_type TABLET_DATA_DELETED (replaced by tablet 25c8a6aaabe2413d8ce88f54ce45a4c4) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.556485 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 1305dbdac3994224a047df139cf8d518
W20240502 08:16:41.561465 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 2b98cc8728c042f4b7f6496fcc5ef913 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b98cc8728c042f4b7f6496fcc5ef913 already in progress: creating tablet
I20240502 08:16:41.562281 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c0c89ea894fe4f0ca7cabd062a857321"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:41.562758 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0c89ea894fe4f0ca7cabd062a857321 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:41.563328 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet 1305dbdac3994224a047df139cf8d518 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.563585 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b98cc8728c042f4b7f6496fcc5ef913 with delete_type TABLET_DATA_DELETED (replaced by tablet 25c8a6aaabe2413d8ce88f54ce45a4c4) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.563799 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 1305dbdac3994224a047df139cf8d518 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.541615 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6bdd34511d19462d91feac0f19089b68" } new_table_name: "default.Table_cf112903d1344a4484d86fb9785c0f17" modify_external_catalogs: true
W20240502 08:16:41.564496 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 1305dbdac3994224a047df139cf8d518 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1305dbdac3994224a047df139cf8d518 already in progress: creating tablet
I20240502 08:16:41.565384 23610 master_service.cc:946] ReplaceTablet: received request to replace tablet 4302fec5c51e4c6c841710bab50b2997 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:41.565990 24173 tablet_bootstrap.cc:492] T 8810b6f61db14fed8b12d36fed413dc8 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:41.566217 24173 ts_tablet_manager.cc:1367] T 8810b6f61db14fed8b12d36fed413dc8 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.005s
I20240502 08:16:41.567229 24173 raft_consensus.cc:348] T 8810b6f61db14fed8b12d36fed413dc8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.567577 24173 raft_consensus.cc:374] T 8810b6f61db14fed8b12d36fed413dc8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.567687 24173 raft_consensus.cc:729] T 8810b6f61db14fed8b12d36fed413dc8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.568076 24173 consensus_queue.cc:260] T 8810b6f61db14fed8b12d36fed413dc8 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.568805 24173 ts_tablet_manager.cc:1398] T 8810b6f61db14fed8b12d36fed413dc8 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:41.569177 24173 tablet_bootstrap.cc:492] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:41.570164 23594 consensus_peers.cc:485] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 -> Peer 84f3c07a31414c4f8fb61c44dc9f16e1 (127.22.225.125:36189): Couldn't send request to peer 84f3c07a31414c4f8fb61c44dc9f16e1. Status: Network error: Client connection negotiation failed: client connection to 127.22.225.125:36189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:41.570880 24173 tablet_bootstrap.cc:654] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.572825 24173 tablet_bootstrap.cc:492] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:41.539680 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e628d856ede54061bb6736c1f1b8b0c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:41.573048 24173 ts_tablet_manager.cc:1367] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
W20240502 08:16:41.573571 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e628d856ede54061bb6736c1f1b8b0c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:41.573798 24173 raft_consensus.cc:348] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.574082 24173 raft_consensus.cc:374] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.574188 24173 raft_consensus.cc:729] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.574451 24173 consensus_queue.cc:260] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.576740 24173 ts_tablet_manager.cc:1398] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:41.577047 24173 tablet_bootstrap.cc:492] T fd57b0357ca749c389bc61add20cd3e7 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:41.579077 24173 tablet_bootstrap.cc:654] T fd57b0357ca749c389bc61add20cd3e7 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.582340 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet b6c52f1cae3a4ad38a9f95bfaf055e76 (table default.Table_35b808b0487f463dbcfa4a773a2a20f0 [id=922fb979b5894023a68820aece0b8c5e]). Expected version 1 got 0
I20240502 08:16:41.584149 24180 ts_tablet_manager.cc:1888] T 4302fec5c51e4c6c841710bab50b2997 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.584384 24180 log.cc:1196] T 4302fec5c51e4c6c841710bab50b2997 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/4302fec5c51e4c6c841710bab50b2997
I20240502 08:16:41.584987 24180 ts_tablet_manager.cc:1909] T 4302fec5c51e4c6c841710bab50b2997 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:41.586058 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f7b5f018d84680ac5533bf99102d31 with delete_type TABLET_DATA_DELETED (replaced by tablet c17e1be6ef294769a01981f4d361be21) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.586366 24181 tablet_replica.cc:330] T d8f7b5f018d84680ac5533bf99102d31 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:41.586566 24181 raft_consensus.cc:2229] T d8f7b5f018d84680ac5533bf99102d31 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.586766 24181 raft_consensus.cc:2258] T d8f7b5f018d84680ac5533bf99102d31 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.587862 24181 ts_tablet_manager.cc:1875] T d8f7b5f018d84680ac5533bf99102d31 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.587989 24058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dead13bba95f43cb809e17f6d7fba028" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:41.588230 24058 raft_consensus.cc:2454] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:41.588668 23723 leader_election.cc:302] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:41.589005 24630 raft_consensus.cc:2781] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:41.589092 24630 raft_consensus.cc:482] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:41.589177 24630 raft_consensus.cc:3037] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:41.589411 24180 tablet_replica.cc:330] T dcdd765895254a39b86b692b5d812c83 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:41.589677 24180 raft_consensus.cc:2229] T dcdd765895254a39b86b692b5d812c83 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.589697 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet f36aadfa1d264f23bcafa91744aed36d with delete_type TABLET_DATA_DELETED (replaced by tablet 2085ced52fff46ebaa98324cbb2d8c11) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.589856 24180 raft_consensus.cc:2258] T dcdd765895254a39b86b692b5d812c83 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:41.591873 24667 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:16:41.551145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.125:36189 (local address 127.22.225.67:59219)
0502 08:16:41.591278 (+ 40133us) negotiation.cc:107] Waiting for socket to connect
0502 08:16:41.591513 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.125:36189: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":40038,"threads_started":1}
I20240502 08:16:41.591915 24630 raft_consensus.cc:504] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.592934 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 4302fec5c51e4c6c841710bab50b2997 (table default.Table_006d60eaae72439f829df411c32c1514 [id=e3f148f833bc4d99a3ab2ec8d5258507]) successfully deleted
W20240502 08:16:41.593055 24666 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:16:41.550321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.125:36189 (local address 127.22.225.65:57115)
0502 08:16:41.590759 (+ 40438us) negotiation.cc:107] Waiting for socket to connect
0502 08:16:41.592773 (+ 2014us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.125:36189: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":40331,"threads_started":1}
I20240502 08:16:41.593703 24630 leader_election.cc:288] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:41.594080 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 4302fec5c51e4c6c841710bab50b2997 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.594179 24174 tablet_bootstrap.cc:492] T b6417aafbdcf416ea7bff56bce5e6ddb P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:41.594326 24174 ts_tablet_manager.cc:1367] T b6417aafbdcf416ea7bff56bce5e6ddb P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.132s user 0.003s sys 0.000s
I20240502 08:16:41.595369 24174 raft_consensus.cc:348] T b6417aafbdcf416ea7bff56bce5e6ddb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.595654 24174 raft_consensus.cc:374] T b6417aafbdcf416ea7bff56bce5e6ddb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.595775 24174 raft_consensus.cc:729] T b6417aafbdcf416ea7bff56bce5e6ddb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.596074 24174 consensus_queue.cc:260] T b6417aafbdcf416ea7bff56bce5e6ddb P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.596940 24058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dead13bba95f43cb809e17f6d7fba028" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:41.597231 24058 raft_consensus.cc:3037] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:41.599617 24181 ts_tablet_manager.cc:1888] T d8f7b5f018d84680ac5533bf99102d31 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.599838 24181 log.cc:1196] T d8f7b5f018d84680ac5533bf99102d31 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/d8f7b5f018d84680ac5533bf99102d31
I20240502 08:16:41.600301 24058 raft_consensus.cc:2454] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:41.600471 24181 ts_tablet_manager.cc:1909] T d8f7b5f018d84680ac5533bf99102d31 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:41.600816 23723 leader_election.cc:302] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:41.600992 24174 ts_tablet_manager.cc:1398] T b6417aafbdcf416ea7bff56bce5e6ddb P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20240502 08:16:41.601135 24630 raft_consensus.cc:2781] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:41.601297 24630 raft_consensus.cc:686] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:41.601373 24174 tablet_bootstrap.cc:492] T 1065732d7333413d984ed56275fd61a9 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:41.601725 24630 consensus_queue.cc:237] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:41.602800 23597 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 4c395a29c7a1485eab39ca107dd91fca (table default.Table_ebc89613d91747e1b59b01780555b212 [id=272acbb9a6ca4f2dbcbfea6aacbe9cfc], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4c395a29c7a1485eab39ca107dd91fca consensus state has no leader
I20240502 08:16:41.603731 24174 tablet_bootstrap.cc:654] T 1065732d7333413d984ed56275fd61a9 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.604513 24643 raft_consensus.cc:482] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:41.604696 24643 raft_consensus.cc:504] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.605484 24643 leader_election.cc:288] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:41.608148 24181 tablet_replica.cc:330] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:41.608333 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet d8f7b5f018d84680ac5533bf99102d31 (table default.Table_498b372c15a04b70b87bd67b543501cf [id=38e955d6bd644d24a7c45172ab60282b]) successfully deleted
I20240502 08:16:41.608515 24181 raft_consensus.cc:2229] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.608678 24181 raft_consensus.cc:2258] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.609418 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d04312bd81f46ed8c49c42b1af1aec9 with delete_type TABLET_DATA_DELETED (replaced by tablet b45f3233dae94be1a1d0d8fb426d7f2c) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:41.609979 24179 log.cc:1196] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/b742c51f6204449a804889815604d5cb
I20240502 08:16:41.610493 24181 ts_tablet_manager.cc:1875] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.610832 23886 tablet_service.cc:1514] Processing DeleteTablet for tablet ce758c4a9258427987e8e83f579a2aa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:41.610954 23858 connection.cc:535] client connection to 127.22.225.125:36189 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:16:41.611053 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cc30014a1ac45de937751eed36877a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:41.612145 24179 ts_tablet_manager.cc:1909] T b742c51f6204449a804889815604d5cb P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:41.613236 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a74a6eb6d214d858903b3307e09e9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:41.613523 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b98cc8728c042f4b7f6496fcc5ef913 with delete_type TABLET_DATA_DELETED (replaced by tablet 25c8a6aaabe2413d8ce88f54ce45a4c4) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:41.613757 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 1305dbdac3994224a047df139cf8d518 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:41.614055 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dead13bba95f43cb809e17f6d7fba028" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:41.614514 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dead13bba95f43cb809e17f6d7fba028" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:41.613903 23921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4302fec5c51e4c6c841710bab50b2997" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:41.614966 23888 tablet_service.cc:1514] Processing DeleteTablet for tablet 955e7bd2216646c6aef5acce8d5b0d77 with delete_type TABLET_DATA_DELETED (replaced by tablet 4645143a00d440a7b6d56c5e30d249f1) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:41.615484 23996 leader_election.cc:341] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:41.616318 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 398) took 1031 ms. Trace:
I20240502 08:16:41.616442 24179 rpcz_store.cc:276] 0502 08:16:40.582208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:40.582263 (+ 55us) service_pool.cc:225] Handling call
0502 08:16:41.510713 (+928450us) ts_tablet_manager.cc:1120] Deleting tablet b742c51f6204449a804889815604d5cb
0502 08:16:41.510720 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:41.514790 (+ 4070us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:41.524550 (+ 9760us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:41.613176 (+ 88626us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":928134}
I20240502 08:16:41.616959 24179 tablet_replica.cc:330] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:41.617204 24179 raft_consensus.cc:2229] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.617395 24179 raft_consensus.cc:2258] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.618202 24179 ts_tablet_manager.cc:1875] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.626070 24174 tablet_bootstrap.cc:492] T 1065732d7333413d984ed56275fd61a9 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:41.626343 24174 ts_tablet_manager.cc:1367] T 1065732d7333413d984ed56275fd61a9 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.009s
I20240502 08:16:41.627491 24174 raft_consensus.cc:348] T 1065732d7333413d984ed56275fd61a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.627837 24174 raft_consensus.cc:374] T 1065732d7333413d984ed56275fd61a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.627949 24174 raft_consensus.cc:729] T 1065732d7333413d984ed56275fd61a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.628242 24174 consensus_queue.cc:260] T 1065732d7333413d984ed56275fd61a9 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.629014 24174 ts_tablet_manager.cc:1398] T 1065732d7333413d984ed56275fd61a9 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240502 08:16:41.629419 24174 tablet_bootstrap.cc:492] T d8f7b5f018d84680ac5533bf99102d31 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:41.632202 24179 ts_tablet_manager.cc:1888] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.633205 24181 ts_tablet_manager.cc:1888] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.633422 24181 log.cc:1196] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/f36aadfa1d264f23bcafa91744aed36d
I20240502 08:16:41.633906 24181 ts_tablet_manager.cc:1909] T f36aadfa1d264f23bcafa91744aed36d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:41.635321 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet f36aadfa1d264f23bcafa91744aed36d (table default.Table_897d3dc97c8241d58a1a9a5329a1f892 [id=e18870bff56747b192b4698f1ca3a8ba]) successfully deleted
I20240502 08:16:41.635519 24174 tablet_bootstrap.cc:654] T d8f7b5f018d84680ac5533bf99102d31 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:41.635764 23993 leader_election.cc:341] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 4302fec5c51e4c6c841710bab50b2997
I20240502 08:16:41.637147 23993 leader_election.cc:302] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:41.636950 24181 tablet_replica.cc:330] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:41.637939 24181 raft_consensus.cc:2229] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.638664 24180 ts_tablet_manager.cc:1875] T dcdd765895254a39b86b692b5d812c83 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.638990 24643 raft_consensus.cc:2684] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Received pre-election callback for term 1 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20240502 08:16:41.639314 24181 raft_consensus.cc:2258] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.640267 24181 ts_tablet_manager.cc:1875] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:41.646790 23723 leader_election.cc:341] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:41.647270 23723 leader_election.cc:341] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:41.647331 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet b742c51f6204449a804889815604d5cb (table default.Table_9507475756c3401fa75fc34d4a4b7814 [id=f42bb435dc484af7990e8cf4671e1c85]) successfully deleted
I20240502 08:16:41.647445 24179 log.cc:1196] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/f51b26043ef44b30846498cd1497607c
I20240502 08:16:41.647918 24179 ts_tablet_manager.cc:1909] T f51b26043ef44b30846498cd1497607c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:41.648905 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 399) took 1023 ms. Trace:
I20240502 08:16:41.649065 24179 rpcz_store.cc:276] 0502 08:16:40.625764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:40.625817 (+ 53us) service_pool.cc:225] Handling call
0502 08:16:41.616846 (+991029us) ts_tablet_manager.cc:1120] Deleting tablet f51b26043ef44b30846498cd1497607c
0502 08:16:41.616861 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:41.625407 (+ 8546us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:41.632051 (+ 6644us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:41.648859 (+ 16808us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":990829}
I20240502 08:16:41.649547 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet f51b26043ef44b30846498cd1497607c (table default.Table_da5e259b0d0c47cf94ab30e3e98d1540 [id=956913447d9b4f55920aa4382692d608]) successfully deleted
W20240502 08:16:41.649889 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet fbef606ce7ac499d9bd84b49d804621c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbef606ce7ac499d9bd84b49d804621c already in progress: creating tablet
I20240502 08:16:41.650223 24179 tablet_replica.cc:330] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
W20240502 08:16:41.650208 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet f36aadfa1d264f23bcafa91744aed36d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f36aadfa1d264f23bcafa91744aed36d already in progress: creating tablet
I20240502 08:16:41.650331 24181 ts_tablet_manager.cc:1888] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.650451 24181 log.cc:1196] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/4302fec5c51e4c6c841710bab50b2997
I20240502 08:16:41.650485 24179 raft_consensus.cc:2229] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.650689 24179 raft_consensus.cc:2258] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.650871 24181 ts_tablet_manager.cc:1909] T 4302fec5c51e4c6c841710bab50b2997 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:41.651350 24179 ts_tablet_manager.cc:1875] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.652004 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 4302fec5c51e4c6c841710bab50b2997 (table default.Table_006d60eaae72439f829df411c32c1514 [id=e3f148f833bc4d99a3ab2ec8d5258507]) successfully deleted
I20240502 08:16:41.640261 24175 raft_consensus.cc:348] T 6788ffd2eb2248c3b84e474de56c37a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.653903 24175 raft_consensus.cc:374] T 6788ffd2eb2248c3b84e474de56c37a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.654024 24175 raft_consensus.cc:729] T 6788ffd2eb2248c3b84e474de56c37a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.654121 24489 consensus_queue.cc:1035] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:41.654260 24175 consensus_queue.cc:260] T 6788ffd2eb2248c3b84e474de56c37a9 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.654824 24175 ts_tablet_manager.cc:1398] T 6788ffd2eb2248c3b84e474de56c37a9 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.137s user 0.001s sys 0.001s
I20240502 08:16:41.655105 24175 tablet_bootstrap.cc:492] T c470d420c45843a69a63d9800b7cfc97 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:41.656708 24175 tablet_bootstrap.cc:654] T c470d420c45843a69a63d9800b7cfc97 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.669476 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b4d8ce6957a042f486a41196af1caaaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:41.670078 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4d8ce6957a042f486a41196af1caaaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:41.671238 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5fa34ef38ad04a649c1fb10b958e99e1" } new_table_name: "default.Table_dfd4c11d83904abea317a3f01cf3c01a" modify_external_catalogs: true
I20240502 08:16:41.671253 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b76e1d838fa444179e6dbaa30058595a" } new_table_name: "default.Table_dcbb721cb3904016a2866ef34f6bf557" modify_external_catalogs: true
I20240502 08:16:41.683813 24175 tablet_bootstrap.cc:492] T c470d420c45843a69a63d9800b7cfc97 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:41.684036 24175 ts_tablet_manager.cc:1367] T c470d420c45843a69a63d9800b7cfc97 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20240502 08:16:41.684793 24175 raft_consensus.cc:348] T c470d420c45843a69a63d9800b7cfc97 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.685042 24175 raft_consensus.cc:374] T c470d420c45843a69a63d9800b7cfc97 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.685122 24175 raft_consensus.cc:729] T c470d420c45843a69a63d9800b7cfc97 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.685359 24175 consensus_queue.cc:260] T c470d420c45843a69a63d9800b7cfc97 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.686794 24596 consensus_queue.cc:1035] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:41.688409 24174 tablet_bootstrap.cc:492] T d8f7b5f018d84680ac5533bf99102d31 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:41.688637 24174 ts_tablet_manager.cc:1367] T d8f7b5f018d84680ac5533bf99102d31 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.059s user 0.019s sys 0.012s
I20240502 08:16:41.689391 24174 raft_consensus.cc:348] T d8f7b5f018d84680ac5533bf99102d31 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.689648 24174 raft_consensus.cc:374] T d8f7b5f018d84680ac5533bf99102d31 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.689751 24174 raft_consensus.cc:729] T d8f7b5f018d84680ac5533bf99102d31 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.690021 24174 consensus_queue.cc:260] T d8f7b5f018d84680ac5533bf99102d31 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.690546 24174 ts_tablet_manager.cc:1398] T d8f7b5f018d84680ac5533bf99102d31 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:41.695356 24175 ts_tablet_manager.cc:1398] T c470d420c45843a69a63d9800b7cfc97 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 08:16:41.695744 24175 tablet_bootstrap.cc:492] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:41.698114 24180 ts_tablet_manager.cc:1888] T dcdd765895254a39b86b692b5d812c83 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.698266 24180 log.cc:1196] T dcdd765895254a39b86b692b5d812c83 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/dcdd765895254a39b86b692b5d812c83
I20240502 08:16:41.698659 24180 ts_tablet_manager.cc:1909] T dcdd765895254a39b86b692b5d812c83 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:41.699343 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet b6417aafbdcf416ea7bff56bce5e6ddb with delete_type TABLET_DATA_DELETED (replaced by tablet 96b8f21c64e343d88d84e9acd0bc69b4) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.700455 24173 tablet_bootstrap.cc:492] T fd57b0357ca749c389bc61add20cd3e7 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:41.700765 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9741459a96e649fb8151cc28b642523e" } modify_external_catalogs: true
I20240502 08:16:41.700961 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9741459a96e649fb8151cc28b642523e" } modify_external_catalogs: true
I20240502 08:16:41.701010 24173 ts_tablet_manager.cc:1367] T fd57b0357ca749c389bc61add20cd3e7 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.124s user 0.036s sys 0.033s
I20240502 08:16:41.701467 24175 tablet_bootstrap.cc:654] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.701865 24173 raft_consensus.cc:348] T fd57b0357ca749c389bc61add20cd3e7 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.702098 24173 raft_consensus.cc:374] T fd57b0357ca749c389bc61add20cd3e7 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.702172 24173 raft_consensus.cc:729] T fd57b0357ca749c389bc61add20cd3e7 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.702409 24173 consensus_queue.cc:260] T fd57b0357ca749c389bc61add20cd3e7 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.703408 24180 tablet_replica.cc:330] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:41.703521 24173 ts_tablet_manager.cc:1398] T fd57b0357ca749c389bc61add20cd3e7 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:41.703600 24180 raft_consensus.cc:2229] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.703769 24180 raft_consensus.cc:2258] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.703866 24173 tablet_bootstrap.cc:492] T 01675f637a5a4645949abfd87b3d09a2 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:41.704250 24180 ts_tablet_manager.cc:1875] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.704840 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet dcdd765895254a39b86b692b5d812c83 (table default.Table_dd3c838f4a014e178351d135f5d4567d [id=6be0b9c539dd40059dab4bae00b7b180]) successfully deleted
I20240502 08:16:41.705277 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet f797d3c6bdc4493398954fbbf9943971
I20240502 08:16:41.705613 23888 tablet_service.cc:1514] Processing DeleteTablet for tablet 6788ffd2eb2248c3b84e474de56c37a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:41.707319 24174 tablet_bootstrap.cc:492] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:41.707643 24179 ts_tablet_manager.cc:1888] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.707808 24179 log.cc:1196] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/43e870c3a0e340d1bad1c1fa0380a762
I20240502 08:16:41.708225 24179 ts_tablet_manager.cc:1909] T 43e870c3a0e340d1bad1c1fa0380a762 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
W20240502 08:16:41.708544 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 3a74a6eb6d214d858903b3307e09e9e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a74a6eb6d214d858903b3307e09e9e6 already in progress: creating tablet
W20240502 08:16:41.708823 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 5d04312bd81f46ed8c49c42b1af1aec9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d04312bd81f46ed8c49c42b1af1aec9 already in progress: creating tablet
I20240502 08:16:41.709522 24179 tablet_replica.cc:330] T 8633864e248f416cb651da3b0c2aa936 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:41.709708 24179 raft_consensus.cc:2229] T 8633864e248f416cb651da3b0c2aa936 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
W20240502 08:16:41.709806 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 955e7bd2216646c6aef5acce8d5b0d77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 955e7bd2216646c6aef5acce8d5b0d77 already in progress: creating tablet
I20240502 08:16:41.709885 24179 raft_consensus.cc:2258] T 8633864e248f416cb651da3b0c2aa936 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.710104 24173 tablet_bootstrap.cc:654] T 01675f637a5a4645949abfd87b3d09a2 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.710371 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet f797d3c6bdc4493398954fbbf9943971 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.710408 24179 ts_tablet_manager.cc:1875] T 8633864e248f416cb651da3b0c2aa936 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.711304 24174 tablet_bootstrap.cc:654] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.713121 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 43e870c3a0e340d1bad1c1fa0380a762 (table default.Table_299ae144ccc6402abe33d07ef693e5b4 [id=7e0c798bc77a4770a4bbe8658f181ffa]) successfully deleted
W20240502 08:16:41.713826 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet c00f29fb76b44d6ea4f76963d2d0751c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c00f29fb76b44d6ea4f76963d2d0751c already in progress: creating tablet
W20240502 08:16:41.714061 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet c17e1be6ef294769a01981f4d361be21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c17e1be6ef294769a01981f4d361be21 already in progress: creating tablet
I20240502 08:16:41.715703 23601 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 23ce83c8001d4b2988b88c0ed9bf948c (table default.Table_190eae2a8c64440f8d19842f8b6ac254 [id=54454efbe9394170822f237dc8e6115e]). Expected version 1 got 0
I20240502 08:16:41.716938 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1600e0e09856405ea0d90ac82aa47fda" } modify_external_catalogs: true
I20240502 08:16:41.717150 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1600e0e09856405ea0d90ac82aa47fda" } modify_external_catalogs: true
I20240502 08:16:41.718327 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c31d9edf126f4ae6a1c06bc92f31a5b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:41.718945 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c31d9edf126f4ae6a1c06bc92f31a5b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:41.721180 23888 tablet_service.cc:1514] Processing DeleteTablet for tablet c470d420c45843a69a63d9800b7cfc97 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:41.721282 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet f797d3c6bdc4493398954fbbf9943971 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.721228 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet f797d3c6bdc4493398954fbbf9943971 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:41.722543 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet f797d3c6bdc4493398954fbbf9943971 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f797d3c6bdc4493398954fbbf9943971 already in progress: creating tablet
I20240502 08:16:41.724491 24175 tablet_bootstrap.cc:492] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:41.724706 24175 ts_tablet_manager.cc:1367] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.004s
I20240502 08:16:41.725505 24174 tablet_bootstrap.cc:492] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:41.725715 24174 ts_tablet_manager.cc:1367] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.001s
I20240502 08:16:41.725942 24175 raft_consensus.cc:348] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.726327 24175 raft_consensus.cc:374] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.726429 24175 raft_consensus.cc:729] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.726475 24174 raft_consensus.cc:348] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.726759 24174 raft_consensus.cc:374] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.726826 24174 raft_consensus.cc:729] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.726722 24175 consensus_queue.cc:260] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.727322 24551 raft_consensus.cc:482] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:41.727465 24551 raft_consensus.cc:504] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.728195 24551 leader_election.cc:288] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:41.729250 23858 leader_election.cc:341] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 6bb2d0a4898544f182ab8070d29a7940
W20240502 08:16:41.729512 23861 leader_election.cc:341] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 6bb2d0a4898544f182ab8070d29a7940
I20240502 08:16:41.729687 23861 leader_election.cc:302] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:41.730228 24551 raft_consensus.cc:2726] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:41.730901 24175 ts_tablet_manager.cc:1398] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:16:41.731303 24175 tablet_bootstrap.cc:492] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:41.727054 24174 consensus_queue.cc:260] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.734026 24489 raft_consensus.cc:482] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:41.734184 24489 raft_consensus.cc:504] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.734826 24489 leader_election.cc:288] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:41.736416 23921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c470d420c45843a69a63d9800b7cfc97" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:41.736624 23921 raft_consensus.cc:2454] T c470d420c45843a69a63d9800b7cfc97 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
W20240502 08:16:41.737329 23723 leader_election.cc:341] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: c470d420c45843a69a63d9800b7cfc97
I20240502 08:16:41.738024 24174 ts_tablet_manager.cc:1398] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 08:16:41.738529 24175 tablet_bootstrap.cc:654] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.740702 24180 ts_tablet_manager.cc:1888] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.740830 24180 log.cc:1196] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/0be4f8d5cc09483d8d1aa1e37679b544
I20240502 08:16:41.741215 24180 ts_tablet_manager.cc:1909] T 0be4f8d5cc09483d8d1aa1e37679b544 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:41.742163 24180 tablet_replica.cc:330] T 10adabdd31f848548ef6f2408ccc2050 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:41.742342 24180 raft_consensus.cc:2229] T 10adabdd31f848548ef6f2408ccc2050 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.742311 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 0be4f8d5cc09483d8d1aa1e37679b544 (table default.Table_178527d31d7044fc98fa41886d651049 [id=6dc275aed42b42118a70099d3eae3c21]) successfully deleted
I20240502 08:16:41.742487 24180 raft_consensus.cc:2258] T 10adabdd31f848548ef6f2408ccc2050 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.744367 24174 tablet_bootstrap.cc:492] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:41.746142 24174 tablet_bootstrap.cc:654] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:41.746546 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a5a5a410b6314818b10927e35032f95d consensus state has no leader
I20240502 08:16:41.746847 23723 leader_election.cc:302] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:41.747383 24489 raft_consensus.cc:2781] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:41.747530 24489 raft_consensus.cc:482] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:41.747643 24489 raft_consensus.cc:3037] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:41.749222 24174 tablet_bootstrap.cc:492] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:41.749388 24174 ts_tablet_manager.cc:1367] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:16:41.749958 24180 ts_tablet_manager.cc:1875] T 10adabdd31f848548ef6f2408ccc2050 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.750270 24174 raft_consensus.cc:348] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.750473 24174 raft_consensus.cc:374] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.750560 24174 raft_consensus.cc:729] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.750730 24179 ts_tablet_manager.cc:1888] T 8633864e248f416cb651da3b0c2aa936 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.750865 24179 log.cc:1196] T 8633864e248f416cb651da3b0c2aa936 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/8633864e248f416cb651da3b0c2aa936
I20240502 08:16:41.751273 24179 ts_tablet_manager.cc:1909] T 8633864e248f416cb651da3b0c2aa936 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:41.752787 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_586d3e878d614a3da2f2988f6ad484d3" } new_table_name: "default.Table_7f8b1dce548042b7886ff9f5e150c3f1" modify_external_catalogs: true
I20240502 08:16:41.753199 24179 tablet_replica.cc:330] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:41.753412 24179 raft_consensus.cc:2229] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.754863 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 8633864e248f416cb651da3b0c2aa936 (table default.Table_8ba240c391da43978f93a27dce48e557 [id=1b5536950db244d1978b154da0be33e0]) successfully deleted
I20240502 08:16:41.750643 24489 raft_consensus.cc:504] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.756949 24643 raft_consensus.cc:482] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:41.757050 24489 leader_election.cc:288] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:41.757105 24643 raft_consensus.cc:504] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.750772 24174 consensus_queue.cc:260] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.757809 24643 leader_election.cc:288] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:41.758549 24179 raft_consensus.cc:2258] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:41.758533 23723 leader_election.cc:341] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: c470d420c45843a69a63d9800b7cfc97
I20240502 08:16:41.758728 24174 ts_tablet_manager.cc:1398] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 08:16:41.759562 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_890e6db207d54accbbd91ba88c78095e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:41.759759 24174 tablet_bootstrap.cc:492] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:41.759956 24179 ts_tablet_manager.cc:1875] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:41.760049 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_890e6db207d54accbbd91ba88c78095e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:41.761106 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c470d420c45843a69a63d9800b7cfc97" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:41.761219 24174 tablet_bootstrap.cc:654] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.762730 24180 ts_tablet_manager.cc:1888] T 10adabdd31f848548ef6f2408ccc2050 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.762873 24180 log.cc:1196] T 10adabdd31f848548ef6f2408ccc2050 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/10adabdd31f848548ef6f2408ccc2050
I20240502 08:16:41.763206 24180 ts_tablet_manager.cc:1909] T 10adabdd31f848548ef6f2408ccc2050 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:41.763820 24173 tablet_bootstrap.cc:492] T 01675f637a5a4645949abfd87b3d09a2 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:41.763972 24173 ts_tablet_manager.cc:1367] T 01675f637a5a4645949abfd87b3d09a2 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.060s user 0.014s sys 0.008s
I20240502 08:16:41.764163 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 10adabdd31f848548ef6f2408ccc2050 (table default.Table_c45e7b653ae641b293b87c150748c864 [id=5ff5989d908243c9a58013285842dd46]) successfully deleted
W20240502 08:16:41.764964 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 1305dbdac3994224a047df139cf8d518 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1305dbdac3994224a047df139cf8d518 already in progress: creating tablet
I20240502 08:16:41.765103 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b816ea1d474f2db94119441146ba78" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:41.765764 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 2b98cc8728c042f4b7f6496fcc5ef913 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b98cc8728c042f4b7f6496fcc5ef913 already in progress: creating tablet
I20240502 08:16:41.766140 24180 tablet_replica.cc:330] T b6417aafbdcf416ea7bff56bce5e6ddb P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:41.765553 24173 raft_consensus.cc:348] T 01675f637a5a4645949abfd87b3d09a2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.766467 24173 raft_consensus.cc:374] T 01675f637a5a4645949abfd87b3d09a2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.766469 24180 raft_consensus.cc:2229] T b6417aafbdcf416ea7bff56bce5e6ddb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.766636 24173 raft_consensus.cc:729] T 01675f637a5a4645949abfd87b3d09a2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.766757 24180 raft_consensus.cc:2258] T b6417aafbdcf416ea7bff56bce5e6ddb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.766940 24173 consensus_queue.cc:260] T 01675f637a5a4645949abfd87b3d09a2 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.767750 24173 ts_tablet_manager.cc:1398] T 01675f637a5a4645949abfd87b3d09a2 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:41.768081 24173 tablet_bootstrap.cc:492] T 9700ea9dc58442e3ab1b7835a8c1e22c P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:41.769078 24174 tablet_bootstrap.cc:492] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:41.769320 24174 ts_tablet_manager.cc:1367] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20240502 08:16:41.770064 24173 tablet_bootstrap.cc:654] T 9700ea9dc58442e3ab1b7835a8c1e22c P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:41.770502 23993 leader_election.cc:341] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:41.770542 24174 raft_consensus.cc:348] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.771020 23923 raft_consensus.cc:3037] T c470d420c45843a69a63d9800b7cfc97 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:41.771286 24175 tablet_bootstrap.cc:492] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:41.771729 24175 ts_tablet_manager.cc:1367] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20240502 08:16:41.772455 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b816ea1d474f2db94119441146ba78" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:41.772938 23996 leader_election.cc:341] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:41.773069 23996 leader_election.cc:302] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:41.773001 24175 raft_consensus.cc:348] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.773293 24643 raft_consensus.cc:2726] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:41.771548 24174 raft_consensus.cc:374] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.773687 24175 raft_consensus.cc:374] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.773795 24175 raft_consensus.cc:729] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.774073 24174 raft_consensus.cc:729] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.774375 24175 consensus_queue.cc:260] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.775256 24175 ts_tablet_manager.cc:1398] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:41.775573 24174 consensus_queue.cc:260] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.776232 24174 ts_tablet_manager.cc:1398] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 08:16:41.777726 24174 tablet_bootstrap.cc:492] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:41.777868 24180 ts_tablet_manager.cc:1875] T b6417aafbdcf416ea7bff56bce5e6ddb P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.779374 24174 tablet_bootstrap.cc:654] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.783903 24180 ts_tablet_manager.cc:1888] T b6417aafbdcf416ea7bff56bce5e6ddb P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.784026 24180 log.cc:1196] T b6417aafbdcf416ea7bff56bce5e6ddb P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/b6417aafbdcf416ea7bff56bce5e6ddb
I20240502 08:16:41.784353 24180 ts_tablet_manager.cc:1909] T b6417aafbdcf416ea7bff56bce5e6ddb P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:41.784626 24173 tablet_bootstrap.cc:492] T 9700ea9dc58442e3ab1b7835a8c1e22c P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:41.784783 24173 ts_tablet_manager.cc:1367] T 9700ea9dc58442e3ab1b7835a8c1e22c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20240502 08:16:41.785475 24173 raft_consensus.cc:348] T 9700ea9dc58442e3ab1b7835a8c1e22c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.785722 24173 raft_consensus.cc:374] T 9700ea9dc58442e3ab1b7835a8c1e22c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.785781 24173 raft_consensus.cc:729] T 9700ea9dc58442e3ab1b7835a8c1e22c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.785992 24173 consensus_queue.cc:260] T 9700ea9dc58442e3ab1b7835a8c1e22c P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.786530 24173 ts_tablet_manager.cc:1398] T 9700ea9dc58442e3ab1b7835a8c1e22c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:41.786826 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet b6417aafbdcf416ea7bff56bce5e6ddb (table default.Table_9d372763eb03435f8b7074a802d86ffe [id=9ff8ea75a9bc4de194a70fe2088a94bc]) successfully deleted
I20240502 08:16:41.786940 24173 tablet_bootstrap.cc:492] T 5453cc0b93ac4cbb90a0588bb2100d60 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:41.787550 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet f797d3c6bdc4493398954fbbf9943971 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f797d3c6bdc4493398954fbbf9943971 already in progress: creating tablet
I20240502 08:16:41.788552 24173 tablet_bootstrap.cc:654] T 5453cc0b93ac4cbb90a0588bb2100d60 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.794412 23923 raft_consensus.cc:2454] T c470d420c45843a69a63d9800b7cfc97 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:41.795818 23723 leader_election.cc:302] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
W20240502 08:16:41.797019 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e9171ebf74424c9aae5e1c8a5118f768 consensus state has no leader
I20240502 08:16:41.797030 24596 raft_consensus.cc:2781] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:41.797181 24596 raft_consensus.cc:686] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:41.797474 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7db65f49fb724e4cae21bcc31d769c17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:41.797521 24596 consensus_queue.cc:237] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:41.798043 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7db65f49fb724e4cae21bcc31d769c17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:41.799476 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b97b5ba571754aac8434d0b5ceb37bd7 has no consensus state
I20240502 08:16:41.799845 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c0c89ea894fe4f0ca7cabd062a857321" } new_table_name: "default.Table_bf53572d3cc544e1aa5674239aaa62f9" modify_external_catalogs: true
I20240502 08:16:41.800310 24174 tablet_bootstrap.cc:492] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:41.800446 24174 ts_tablet_manager.cc:1367] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.004s
I20240502 08:16:41.801029 23602 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet b45f3233dae94be1a1d0d8fb426d7f2c
I20240502 08:16:41.801234 24174 raft_consensus.cc:348] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.801317 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d035b24dfb644b7aa77b160ee28d36e5" } modify_external_catalogs: true
I20240502 08:16:41.801414 24174 raft_consensus.cc:374] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.801482 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d035b24dfb644b7aa77b160ee28d36e5" } modify_external_catalogs: true
I20240502 08:16:41.801503 24174 raft_consensus.cc:729] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.801733 24174 consensus_queue.cc:260] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.802287 24174 ts_tablet_manager.cc:1398] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:16:41.802212 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_593fb38ca79d4c73aaf5f73fda9862ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:41.802600 24174 tablet_bootstrap.cc:492] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
W20240502 08:16:41.802685 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_593fb38ca79d4c73aaf5f73fda9862ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:41.804221 24174 tablet_bootstrap.cc:654] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.804620 23606 catalog_manager.cc:5484] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:41.805097 23606 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet dead13bba95f43cb809e17f6d7fba028 (table default.Table_978594faab984d60ba03d29d40328a6c [id=7409c7e1a3534932a787559a679e7098]). Expected version 3 got 0
I20240502 08:16:41.805262 24175 tablet_bootstrap.cc:492] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:41.805410 23606 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 23ce83c8001d4b2988b88c0ed9bf948c (table default.Table_190eae2a8c64440f8d19842f8b6ac254 [id=54454efbe9394170822f237dc8e6115e]). Expected version 1 got 0
I20240502 08:16:41.806221 23605 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet e134e90efb4a48bb82b53a80bda708b3 (table default.Table_cc80d1cedafa4e71906e5e9530fdd828 [id=47682c26a85842c7b8991bf3828a9336]). Expected version 1 got 0
I20240502 08:16:41.806861 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_af7c06dd8b3f4085b62eaef9ff9b57cb" } modify_external_catalogs: true
I20240502 08:16:41.807020 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_af7c06dd8b3f4085b62eaef9ff9b57cb" } modify_external_catalogs: true
W20240502 08:16:41.808504 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 23ba091d8efb44858bf90a104c0f442f consensus state has no leader
I20240502 08:16:41.808753 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 02e6a1ba23dd4f82bc721a60c0526871 (DEFAULT_TABLE table=default.Table_91aaef5fd943406aa3548482115afc65 [id=4a26d823fb2e417f9211095db2b1e003]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.808916 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_441797423e20442785b0b07f4161668f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:41.809146 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02e6a1ba23dd4f82bc721a60c0526871. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:41.809343 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_441797423e20442785b0b07f4161668f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:41.809381 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 23ba091d8efb44858bf90a104c0f442f (DEFAULT_TABLE table=default.Table_586d3e878d614a3da2f2988f6ad484d3 [id=39e09ba050be418293957c566ee53c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.809751 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ba091d8efb44858bf90a104c0f442f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.810467 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 02e6a1ba23dd4f82bc721a60c0526871 (DEFAULT_TABLE table=default.Table_91aaef5fd943406aa3548482115afc65 [id=4a26d823fb2e417f9211095db2b1e003]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.810865 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02e6a1ba23dd4f82bc721a60c0526871. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.811830 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 255f69f56d344ce7805eca0bfdf8a063 (DEFAULT_TABLE table=default.Table_8bbe9f5123cc4dacb9fd2749d84f9988 [id=6c492ddfa9aa4d53b14c0cb7c0e116f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.812204 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 2ebf55c8024c483f9ef0afe6945a02ff (DEFAULT_TABLE table=default.Table_cf8b38c1f7e343bc9d21651af74e76c8 [id=00551fb59aa04e23a8c7cb96f413164d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.812646 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 255f69f56d344ce7805eca0bfdf8a063 (DEFAULT_TABLE table=default.Table_8bbe9f5123cc4dacb9fd2749d84f9988 [id=6c492ddfa9aa4d53b14c0cb7c0e116f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.812980 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 255f69f56d344ce7805eca0bfdf8a063. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.813695 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 2ebf55c8024c483f9ef0afe6945a02ff (DEFAULT_TABLE table=default.Table_cf8b38c1f7e343bc9d21651af74e76c8 [id=00551fb59aa04e23a8c7cb96f413164d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.813952 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 14075c65ed7a4aa78166943aa32f904f (DEFAULT_TABLE table=default.Table_2ecca22d4d934ddbacb4eb1826759e48 [id=3970bee0bcfc4cd6b0ab71113e1cf423]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.813963 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 02e6a1ba23dd4f82bc721a60c0526871 (DEFAULT_TABLE table=default.Table_91aaef5fd943406aa3548482115afc65 [id=4a26d823fb2e417f9211095db2b1e003]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.814173 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ebf55c8024c483f9ef0afe6945a02ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.814457 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 255f69f56d344ce7805eca0bfdf8a063 (DEFAULT_TABLE table=default.Table_8bbe9f5123cc4dacb9fd2749d84f9988 [id=6c492ddfa9aa4d53b14c0cb7c0e116f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.814877 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 255f69f56d344ce7805eca0bfdf8a063. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.814985 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 255f69f56d344ce7805eca0bfdf8a063. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.815979 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 14075c65ed7a4aa78166943aa32f904f (DEFAULT_TABLE table=default.Table_2ecca22d4d934ddbacb4eb1826759e48 [id=3970bee0bcfc4cd6b0ab71113e1cf423]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.816393 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 19e4cd31e7774546a54704ccd405ffe6 (DEFAULT_TABLE table=default.Table_7202b499dd484f6196154d3bc98435fb [id=ae75c0f04f8e45e399aab1e0c2fa9bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.816792 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 2ebf55c8024c483f9ef0afe6945a02ff (DEFAULT_TABLE table=default.Table_cf8b38c1f7e343bc9d21651af74e76c8 [id=00551fb59aa04e23a8c7cb96f413164d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.817206 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 23ba091d8efb44858bf90a104c0f442f (DEFAULT_TABLE table=default.Table_586d3e878d614a3da2f2988f6ad484d3 [id=39e09ba050be418293957c566ee53c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.817700 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 23ba091d8efb44858bf90a104c0f442f (DEFAULT_TABLE table=default.Table_586d3e878d614a3da2f2988f6ad484d3 [id=39e09ba050be418293957c566ee53c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.817797 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 14075c65ed7a4aa78166943aa32f904f (DEFAULT_TABLE table=default.Table_2ecca22d4d934ddbacb4eb1826759e48 [id=3970bee0bcfc4cd6b0ab71113e1cf423]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.817994 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ba091d8efb44858bf90a104c0f442f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.818222 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 19e4cd31e7774546a54704ccd405ffe6 (DEFAULT_TABLE table=default.Table_7202b499dd484f6196154d3bc98435fb [id=ae75c0f04f8e45e399aab1e0c2fa9bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.818581 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19e4cd31e7774546a54704ccd405ffe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.821800 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14075c65ed7a4aa78166943aa32f904f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.821817 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14075c65ed7a4aa78166943aa32f904f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.824738 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02e6a1ba23dd4f82bc721a60c0526871. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.824786 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14075c65ed7a4aa78166943aa32f904f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.825759 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ba091d8efb44858bf90a104c0f442f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.826207 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ebf55c8024c483f9ef0afe6945a02ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.827447 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet 8810b6f61db14fed8b12d36fed413dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.827731 24181 tablet_replica.cc:330] T 8810b6f61db14fed8b12d36fed413dc8 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:41.827962 24181 raft_consensus.cc:2229] T 8810b6f61db14fed8b12d36fed413dc8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.828150 24181 raft_consensus.cc:2258] T 8810b6f61db14fed8b12d36fed413dc8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.829694 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet e0af5f4daf3149738647a08219efb245 with delete_type TABLET_DATA_DELETED (replaced by tablet b8b1fa17dad143f0a7f1751e651eb81a) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.830778 24181 ts_tablet_manager.cc:1875] T 8810b6f61db14fed8b12d36fed413dc8 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.839069 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ebf55c8024c483f9ef0afe6945a02ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.839298 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19e4cd31e7774546a54704ccd405ffe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.846935 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 045b0882a40f4eaba94935d9bc90da0c (DEFAULT_TABLE table=default.Table_b3289c94e9274ea591f2b4e00a58a84d [id=abb72d2b04794e4aab7acee7a3a0925b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.847575 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 045b0882a40f4eaba94935d9bc90da0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.856400 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 045b0882a40f4eaba94935d9bc90da0c (DEFAULT_TABLE table=default.Table_b3289c94e9274ea591f2b4e00a58a84d [id=abb72d2b04794e4aab7acee7a3a0925b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.856925 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 045b0882a40f4eaba94935d9bc90da0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.864431 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 592ab8523b8d479d9caead321d5888c3
I20240502 08:16:41.864549 23603 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet e134e90efb4a48bb82b53a80bda708b3 (table default.Table_cc80d1cedafa4e71906e5e9530fdd828 [id=47682c26a85842c7b8991bf3828a9336]). Expected version 1 got 0
I20240502 08:16:41.865293 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 577b149f52e44d5d9a9f6b9d85a6864d
I20240502 08:16:41.867483 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f7b5f018d84680ac5533bf99102d31 with delete_type TABLET_DATA_DELETED (replaced by tablet c17e1be6ef294769a01981f4d361be21) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.867766 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet 592ab8523b8d479d9caead321d5888c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.867952 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 577b149f52e44d5d9a9f6b9d85a6864d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:41.868418 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d48d1f30fb34ee6a9a68908a3442d43 has no consensus state
I20240502 08:16:41.869202 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 72ee52bcc61b4816acda5b8140b63504
I20240502 08:16:41.869524 24180 tablet_replica.cc:330] T d8f7b5f018d84680ac5533bf99102d31 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:41.869762 24180 raft_consensus.cc:2229] T d8f7b5f018d84680ac5533bf99102d31 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.869938 24180 raft_consensus.cc:2258] T d8f7b5f018d84680ac5533bf99102d31 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.871039 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ee52bcc61b4816acda5b8140b63504 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.871039 24180 ts_tablet_manager.cc:1875] T d8f7b5f018d84680ac5533bf99102d31 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.872225 23888 tablet_service.cc:1514] Processing DeleteTablet for tablet 1065732d7333413d984ed56275fd61a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:41.872452 23886 tablet_service.cc:1514] Processing DeleteTablet for tablet 577b149f52e44d5d9a9f6b9d85a6864d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:41.872648 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 592ab8523b8d479d9caead321d5888c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:41.874287 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet b6417aafbdcf416ea7bff56bce5e6ddb with delete_type TABLET_DATA_DELETED (replaced by tablet 96b8f21c64e343d88d84e9acd0bc69b4) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.875893 24179 ts_tablet_manager.cc:1888] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.876035 24179 log.cc:1196] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/d5bab1b7c48248a7973c27d0a865b34b
I20240502 08:16:41.876559 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet 1065732d7333413d984ed56275fd61a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:35 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.876684 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 1be95e0576c143e5a94c7aaf8d3e0625 (DEFAULT_TABLE table=default.Table_6ce45bf2f8e64d759b44ee43f3df434d [id=c817273ee3b84c21b43137b220ba13b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.878002 24175 tablet_bootstrap.cc:654] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.877945 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1b8fb2a811c54b659fb14d3bb815bc75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:41.878506 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b8fb2a811c54b659fb14d3bb815bc75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:41.878796 23757 tablet_service.cc:1467] Processing CreateTablet for tablet b97b5ba571754aac8434d0b5ceb37bd7 (DEFAULT_TABLE table=default.Table_dfd4c11d83904abea317a3f01cf3c01a [id=8c2ca203bd9d4b01a1e9cab9546d0676]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.879173 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b97b5ba571754aac8434d0b5ceb37bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.879276 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ee52bcc61b4816acda5b8140b63504 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:41.880506 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c95dfda9d3b4e2ca9397b77ad71c011 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.882557 23758 tablet_service.cc:1467] Processing CreateTablet for tablet a17fb4f2f02e43dc9f7cfdb7c73f7774 (DEFAULT_TABLE table=default.Table_480b6f612ca64d2caaa95f87d6ce8464 [id=a1004106109d47e7a1f9c6337480d3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.882901 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a17fb4f2f02e43dc9f7cfdb7c73f7774. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.885035 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 1be95e0576c143e5a94c7aaf8d3e0625 (DEFAULT_TABLE table=default.Table_6ce45bf2f8e64d759b44ee43f3df434d [id=c817273ee3b84c21b43137b220ba13b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.885185 23899 tablet_service.cc:1467] Processing CreateTablet for tablet a17fb4f2f02e43dc9f7cfdb7c73f7774 (DEFAULT_TABLE table=default.Table_480b6f612ca64d2caaa95f87d6ce8464 [id=a1004106109d47e7a1f9c6337480d3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.885908 23894 tablet_service.cc:1467] Processing CreateTablet for tablet b97b5ba571754aac8434d0b5ceb37bd7 (DEFAULT_TABLE table=default.Table_dfd4c11d83904abea317a3f01cf3c01a [id=8c2ca203bd9d4b01a1e9cab9546d0676]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.886397 24179 ts_tablet_manager.cc:1909] T d5bab1b7c48248a7973c27d0a865b34b P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:41.886490 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 2085ced52fff46ebaa98324cbb2d8c11 (DEFAULT_TABLE table=default.Table_897d3dc97c8241d58a1a9a5329a1f892 [id=e18870bff56747b192b4698f1ca3a8ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.886967 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet d5bab1b7c48248a7973c27d0a865b34b (table default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]) successfully deleted
W20240502 08:16:41.887318 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 6e7432f1df94431a83cf68356e3e78a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e7432f1df94431a83cf68356e3e78a3 already in progress: creating tablet
I20240502 08:16:41.887369 24179 tablet_replica.cc:330] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
W20240502 08:16:41.887537 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 33c8131846dc41678827b86e87be18d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33c8131846dc41678827b86e87be18d3 already in progress: creating tablet
I20240502 08:16:41.887614 24179 raft_consensus.cc:2229] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.887789 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2085ced52fff46ebaa98324cbb2d8c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.887842 24179 raft_consensus.cc:2258] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.888518 24179 ts_tablet_manager.cc:1875] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.888849 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be95e0576c143e5a94c7aaf8d3e0625. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.889849 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be95e0576c143e5a94c7aaf8d3e0625. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.891781 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 2085ced52fff46ebaa98324cbb2d8c11 (DEFAULT_TABLE table=default.Table_897d3dc97c8241d58a1a9a5329a1f892 [id=e18870bff56747b192b4698f1ca3a8ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.892194 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2085ced52fff46ebaa98324cbb2d8c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.892458 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b97b5ba571754aac8434d0b5ceb37bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.892686 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a17fb4f2f02e43dc9f7cfdb7c73f7774. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.896984 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ee52bcc61b4816acda5b8140b63504 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.896586 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet 577b149f52e44d5d9a9f6b9d85a6864d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.900589 23607 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 23ce83c8001d4b2988b88c0ed9bf948c (table default.Table_190eae2a8c64440f8d19842f8b6ac254 [id=54454efbe9394170822f237dc8e6115e]). Expected version 1 got 0
I20240502 08:16:41.900674 24180 ts_tablet_manager.cc:1888] T d8f7b5f018d84680ac5533bf99102d31 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.900837 24180 log.cc:1196] T d8f7b5f018d84680ac5533bf99102d31 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/d8f7b5f018d84680ac5533bf99102d31
I20240502 08:16:41.900777 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 2085ced52fff46ebaa98324cbb2d8c11 (DEFAULT_TABLE table=default.Table_897d3dc97c8241d58a1a9a5329a1f892 [id=e18870bff56747b192b4698f1ca3a8ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.901222 24180 ts_tablet_manager.cc:1909] T d8f7b5f018d84680ac5533bf99102d31 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:41.901197 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet 592ab8523b8d479d9caead321d5888c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.901765 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2085ced52fff46ebaa98324cbb2d8c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.901810 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet b6417aafbdcf416ea7bff56bce5e6ddb from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:41.902171 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8cb458b151584d4a8916e34bf519d16b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:41.902401 24028 tablet_service.cc:1467] Processing CreateTablet for tablet a17fb4f2f02e43dc9f7cfdb7c73f7774 (DEFAULT_TABLE table=default.Table_480b6f612ca64d2caaa95f87d6ce8464 [id=a1004106109d47e7a1f9c6337480d3c0]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:41.902719 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cb458b151584d4a8916e34bf519d16b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:41.901530 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 19e4cd31e7774546a54704ccd405ffe6 (DEFAULT_TABLE table=default.Table_7202b499dd484f6196154d3bc98435fb [id=ae75c0f04f8e45e399aab1e0c2fa9bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.903051 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a17fb4f2f02e43dc9f7cfdb7c73f7774. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.903663 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet fd57b0357ca749c389bc61add20cd3e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.903827 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 1be95e0576c143e5a94c7aaf8d3e0625 (DEFAULT_TABLE table=default.Table_6ce45bf2f8e64d759b44ee43f3df434d [id=c817273ee3b84c21b43137b220ba13b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.903849 24551 raft_consensus.cc:482] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:41.903997 24551 raft_consensus.cc:504] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.904134 24643 raft_consensus.cc:482] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:41.904253 24643 raft_consensus.cc:504] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.904717 24551 leader_election.cc:288] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:41.904850 24551 raft_consensus.cc:482] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:41.904903 24643 leader_election.cc:288] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:41.904994 24551 raft_consensus.cc:504] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.905092 24031 tablet_service.cc:1467] Processing CreateTablet for tablet b97b5ba571754aac8434d0b5ceb37bd7 (DEFAULT_TABLE table=default.Table_dfd4c11d83904abea317a3f01cf3c01a [id=8c2ca203bd9d4b01a1e9cab9546d0676]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.906256 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c95dfda9d3b4e2ca9397b77ad71c011" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:41.906678 24551 leader_election.cc:288] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:41.907059 23858 leader_election.cc:341] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: ce758c4a9258427987e8e83f579a2aa2
W20240502 08:16:41.907342 23858 leader_election.cc:341] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 3cc30014a1ac45de937751eed36877a9
W20240502 08:16:41.907740 23723 consensus_peers.cc:485] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:41.908314 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 01675f637a5a4645949abfd87b3d09a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.908725 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 6e35efdc0434429aa093fec7770782ef (DEFAULT_TABLE table=default.Table_b1e5c38f8b45471698ed791873ecdb99 [id=e769c65cb8424c8497d6d171b25e4bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.909242 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e35efdc0434429aa093fec7770782ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.913668 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet 9700ea9dc58442e3ab1b7835a8c1e22c with delete_type TABLET_DATA_DELETED (replaced by tablet 5d04312bd81f46ed8c49c42b1af1aec9) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:41.913863 24643 raft_consensus.cc:482] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:41.914026 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 6e35efdc0434429aa093fec7770782ef (DEFAULT_TABLE table=default.Table_b1e5c38f8b45471698ed791873ecdb99 [id=e769c65cb8424c8497d6d171b25e4bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.917100 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6b486df3232a4864b82575bd742573c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:41.917668 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b486df3232a4864b82575bd742573c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:41.917642 23763 tablet_service.cc:1467] Processing CreateTablet for tablet d4b03d759a354a39aea44fd5d4eeb089 (DEFAULT_TABLE table=default.Table_93bd02b663b8448aa250379544e73dea [id=b58292888cd64cdebee649c401ad8f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.918197 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b03d759a354a39aea44fd5d4eeb089. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:41.921669 23993 leader_election.cc:341] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:41.922724 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2d9b44d9d9d34aa184855169eb278e94" } modify_external_catalogs: true
I20240502 08:16:41.922912 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2d9b44d9d9d34aa184855169eb278e94" } modify_external_catalogs: true
I20240502 08:16:41.925285 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet d8f7b5f018d84680ac5533bf99102d31 (table default.Table_498b372c15a04b70b87bd67b543501cf [id=38e955d6bd644d24a7c45172ab60282b]) successfully deleted
W20240502 08:16:41.925972 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 592ab8523b8d479d9caead321d5888c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 592ab8523b8d479d9caead321d5888c3 already in progress: creating tablet
I20240502 08:16:41.926082 23603 master_service.cc:946] ReplaceTablet: received request to replace tablet c08228fcd1304528b9446003851f6c70 from {username='slave'} at 127.0.0.1:36668
W20240502 08:16:41.926213 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 577b149f52e44d5d9a9f6b9d85a6864d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 577b149f52e44d5d9a9f6b9d85a6864d already in progress: creating tablet
W20240502 08:16:41.926443 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 72ee52bcc61b4816acda5b8140b63504 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72ee52bcc61b4816acda5b8140b63504 already in progress: creating tablet
I20240502 08:16:41.928802 24180 tablet_replica.cc:330] T 1065732d7333413d984ed56275fd61a9 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
W20240502 08:16:41.929030 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet b6417aafbdcf416ea7bff56bce5e6ddb because the tablet was not found. No further retry: Not found: Tablet not found: b6417aafbdcf416ea7bff56bce5e6ddb
I20240502 08:16:41.929045 24180 raft_consensus.cc:2229] T 1065732d7333413d984ed56275fd61a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.929234 24180 raft_consensus.cc:2258] T 1065732d7333413d984ed56275fd61a9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.932341 23603 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet b844458ae73f413cbf2113027a5af359 (table default.Table_0a4ff982972e4fdaae1583619c9d6b04 [id=fd93259a378b4ea582a6661a59104abb]). Expected version 1 got 0
I20240502 08:16:41.932504 23603 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 49b816ea1d474f2db94119441146ba78 (table default.Table_20df1b504848439c869ae3739ac54185 [id=a0606a4053d04dd3814f09419a4a70c2]). Expected version 1 got 0
I20240502 08:16:41.934041 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet c470d420c45843a69a63d9800b7cfc97 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.933998 24181 ts_tablet_manager.cc:1888] T 8810b6f61db14fed8b12d36fed413dc8 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:41.935285 23892 tablet_service.cc:1467] Processing CreateTablet for tablet d4b03d759a354a39aea44fd5d4eeb089 (DEFAULT_TABLE table=default.Table_93bd02b663b8448aa250379544e73dea [id=b58292888cd64cdebee649c401ad8f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.937404 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 21832f0151ec4cf48f51f9bd2cb2b5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:41.938534 24058 raft_consensus.cc:1261] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:41.939167 24596 consensus_queue.cc:1035] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:41.939102 24643 raft_consensus.cc:504] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.939548 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 6e35efdc0434429aa093fec7770782ef (DEFAULT_TABLE table=default.Table_b1e5c38f8b45471698ed791873ecdb99 [id=e769c65cb8424c8497d6d171b25e4bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.939543 24028 tablet_service.cc:1467] Processing CreateTablet for tablet d4b03d759a354a39aea44fd5d4eeb089 (DEFAULT_TABLE table=default.Table_93bd02b663b8448aa250379544e73dea [id=b58292888cd64cdebee649c401ad8f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.940264 24643 leader_election.cc:288] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:41.940713 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 045b0882a40f4eaba94935d9bc90da0c (DEFAULT_TABLE table=default.Table_b3289c94e9274ea591f2b4e00a58a84d [id=abb72d2b04794e4aab7acee7a3a0925b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.940837 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19e4cd31e7774546a54704ccd405ffe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.941213 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e35efdc0434429aa093fec7770782ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.942474 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0af5f4daf3149738647a08219efb245" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:41.943125 23861 leader_election.cc:341] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: ce758c4a9258427987e8e83f579a2aa2
I20240502 08:16:41.943310 23861 leader_election.cc:302] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:41.943573 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c95dfda9d3b4e2ca9397b77ad71c011" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:41.943924 23861 leader_election.cc:341] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 3cc30014a1ac45de937751eed36877a9
I20240502 08:16:41.944078 23861 leader_election.cc:302] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:41.944406 24551 raft_consensus.cc:2726] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:41.944620 24551 raft_consensus.cc:2726] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:41.944775 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0af5f4daf3149738647a08219efb245" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:41.945293 23996 leader_election.cc:341] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:41.945307 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be95e0576c143e5a94c7aaf8d3e0625. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:41.945811 23996 leader_election.cc:341] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:41.945938 23996 leader_election.cc:302] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:41.947731 24596 raft_consensus.cc:482] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:41.947906 24596 raft_consensus.cc:504] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.949244 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6ecc8a33231743169b89cf2bd44a49e6" } new_table_name: "default.Table_2df50e9ecea9421b9bfebd892ebe62bd" modify_external_catalogs: true
I20240502 08:16:41.953773 24596 leader_election.cc:288] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:41.954182 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21832f0151ec4cf48f51f9bd2cb2b5ec" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:41.954856 23723 leader_election.cc:341] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:41.955904 23605 master_service.cc:946] ReplaceTablet: received request to replace tablet 090bec30f5664421b9bfe3f2810afd14 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:41.956137 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b97b5ba571754aac8434d0b5ceb37bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.957278 24175 tablet_bootstrap.cc:492] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:41.957597 24643 raft_consensus.cc:2726] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:41.957882 24643 raft_consensus.cc:482] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:41.957895 24175 ts_tablet_manager.cc:1367] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.182s user 0.005s sys 0.001s
W20240502 08:16:41.958424 23723 leader_election.cc:341] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 21832f0151ec4cf48f51f9bd2cb2b5ec
I20240502 08:16:41.958586 23723 leader_election.cc:302] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:41.958846 24180 ts_tablet_manager.cc:1875] T 1065732d7333413d984ed56275fd61a9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:41.959259 24175 raft_consensus.cc:348] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.959542 24175 raft_consensus.cc:374] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.959687 24175 raft_consensus.cc:729] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.959892 24489 raft_consensus.cc:2726] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:41.960387 24175 consensus_queue.cc:260] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:41.961019 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b3352fb8a27f46c78a9919a2de2510c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:41.961612 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3352fb8a27f46c78a9919a2de2510c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:41.961508 23993 leader_election.cc:341] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:41.964107 24671 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } } }
I20240502 08:16:41.964334 24671 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:41.961763 23993 leader_election.cc:302] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:41.965185 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5fb99cde55ed4b33a109255d52e75384"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:41.965802 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fb99cde55ed4b33a109255d52e75384 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:41.967003 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b03d759a354a39aea44fd5d4eeb089. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.971797 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 045b0882a40f4eaba94935d9bc90da0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.973865 24643 raft_consensus.cc:504] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:41.974618 24643 leader_election.cc:288] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:41.974817 24643 raft_consensus.cc:2726] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:41.975023 24181 log.cc:1196] T 8810b6f61db14fed8b12d36fed413dc8 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/8810b6f61db14fed8b12d36fed413dc8
I20240502 08:16:41.975080 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2d7b6f4e442ae95b2d1002d571b6a" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:41.975505 24181 ts_tablet_manager.cc:1909] T 8810b6f61db14fed8b12d36fed413dc8 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
W20240502 08:16:41.975723 23996 leader_election.cc:341] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:41.976791 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 8810b6f61db14fed8b12d36fed413dc8 (table default.Table_ce1b3e3032d84f72b573bf1049c30237 [id=4c18f80dd28145b399b53670c182821c]) successfully deleted
I20240502 08:16:41.977636 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2d7b6f4e442ae95b2d1002d571b6a" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:41.978372 23993 leader_election.cc:341] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:41.978551 23993 leader_election.cc:302] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:41.978837 24643 raft_consensus.cc:2726] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:41.978811 24175 ts_tablet_manager.cc:1398] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20240502 08:16:41.980124 24175 tablet_bootstrap.cc:492] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:41.984599 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b03d759a354a39aea44fd5d4eeb089. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.984881 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e35efdc0434429aa093fec7770782ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.987404 24175 tablet_bootstrap.cc:654] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.998451 24181 tablet_replica.cc:330] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:41.998769 24181 raft_consensus.cc:2229] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:41.999050 24181 raft_consensus.cc:2258] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:41.999886 24181 ts_tablet_manager.cc:1875] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:42.002703 24175 tablet_bootstrap.cc:492] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:42.005463 24175 ts_tablet_manager.cc:1367] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20240502 08:16:42.008975 24175 raft_consensus.cc:348] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.017882 24175 raft_consensus.cc:374] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.028856 24175 raft_consensus.cc:729] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.030252 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet ee022e8e86f8477a8fa1e58d52644b03 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:42.031683 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet b2eac3bde41c4415bdb97fc53a5b395f
I20240502 08:16:42.033746 24174 tablet_bootstrap.cc:492] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:42.033924 24174 ts_tablet_manager.cc:1367] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.231s user 0.005s sys 0.017s
I20240502 08:16:42.033996 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet b2eac3bde41c4415bdb97fc53a5b395f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.034729 24174 raft_consensus.cc:348] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:42.034986 24174 raft_consensus.cc:374] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.035070 24174 raft_consensus.cc:729] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.035336 24174 consensus_queue.cc:260] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:42.038244 24174 ts_tablet_manager.cc:1398] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:42.038226 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet b2eac3bde41c4415bdb97fc53a5b395f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:42.042332 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fa986dbf19ba412dbe12164c322d7558"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:42.042419 24175 consensus_queue.cc:260] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.042536 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cf8b38c1f7e343bc9d21651af74e76c8" } new_table_name: "default.Table_50749af5743f447ebb3195b85f663cf5" modify_external_catalogs: true
W20240502 08:16:42.042860 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa986dbf19ba412dbe12164c322d7558 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.042941 24175 ts_tablet_manager.cc:1398] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20240502 08:16:42.043715 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_09e8bcad70344895b5831638daf0c93c" } modify_external_catalogs: true
I20240502 08:16:42.043890 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_09e8bcad70344895b5831638daf0c93c" } modify_external_catalogs: true
I20240502 08:16:42.044791 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 01675f637a5a4645949abfd87b3d09a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:42.045046 24175 tablet_bootstrap.cc:492] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:42.045270 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f2608186d7f5434eb5f56e2ecdbdccfe" } new_table_name: "default.Table_59270ebb70304dec9598e44f5d3bf506" modify_external_catalogs: true
I20240502 08:16:42.046489 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_acb191a2bab44ddd84855f88d5b7e073"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.046969 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acb191a2bab44ddd84855f88d5b7e073 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.047047 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet 6788ffd2eb2248c3b84e474de56c37a9 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:42.048405 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_978594faab984d60ba03d29d40328a6c" } new_table_name: "default.Table_ef79f64b4af44b88a006a32b437c3207" modify_external_catalogs: true
I20240502 08:16:42.049320 24174 tablet_bootstrap.cc:492] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:42.050098 24181 ts_tablet_manager.cc:1888] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:42.050225 24181 log.cc:1196] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/e0af5f4daf3149738647a08219efb245
I20240502 08:16:42.050483 24175 tablet_bootstrap.cc:654] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.050582 24181 ts_tablet_manager.cc:1909] T e0af5f4daf3149738647a08219efb245 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:42.050899 24174 tablet_bootstrap.cc:654] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.051373 24179 ts_tablet_manager.cc:1888] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:42.051497 24179 log.cc:1196] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/6bb2d0a4898544f182ab8070d29a7940
I20240502 08:16:42.051838 24179 ts_tablet_manager.cc:1909] T 6bb2d0a4898544f182ab8070d29a7940 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:42.052809 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 6bb2d0a4898544f182ab8070d29a7940 (table default.Table_482a1b0b7a064e02805ac3b86fc62f99 [id=3c1d614616374739aeb928e797479df4]) successfully deleted
I20240502 08:16:42.053977 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet b11f504b93574e42a9bf30fde6d08207 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:42.054188 24179 tablet_replica.cc:330] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:42.054387 24179 raft_consensus.cc:2229] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:42.054550 24179 raft_consensus.cc:2258] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:42.055092 24179 ts_tablet_manager.cc:1875] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:42.056756 23602 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet dead13bba95f43cb809e17f6d7fba028 (table default.Table_978594faab984d60ba03d29d40328a6c [id=7409c7e1a3534932a787559a679e7098]). Expected version 3 got 0
I20240502 08:16:42.057473 24181 tablet_replica.cc:330] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:42.057736 24181 raft_consensus.cc:2229] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:42.057935 24181 raft_consensus.cc:2258] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:42.058552 24181 ts_tablet_manager.cc:1875] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:42.058782 23606 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet b6c52f1cae3a4ad38a9f95bfaf055e76 (table default.Table_35b808b0487f463dbcfa4a773a2a20f0 [id=922fb979b5894023a68820aece0b8c5e]). Expected version 1 got 0
I20240502 08:16:42.061009 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet e0af5f4daf3149738647a08219efb245 (table default.Table_d4bd37f8c6e54ecf82cd247846aa06d0 [id=57722a7341c541b19efa213bcf48eb53]) successfully deleted
I20240502 08:16:42.067938 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9c8c85b0e8084a3698ebc3aae915c41c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.068514 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c8c85b0e8084a3698ebc3aae915c41c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.072077 24181 ts_tablet_manager.cc:1888] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:42.072507 24181 log.cc:1196] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/5c95dfda9d3b4e2ca9397b77ad71c011
I20240502 08:16:42.072957 24181 ts_tablet_manager.cc:1909] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:42.074393 24181 tablet_replica.cc:330] T fd57b0357ca749c389bc61add20cd3e7 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:42.074695 24181 raft_consensus.cc:2229] T fd57b0357ca749c389bc61add20cd3e7 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:42.074949 24181 raft_consensus.cc:2258] T fd57b0357ca749c389bc61add20cd3e7 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:42.075098 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 5c95dfda9d3b4e2ca9397b77ad71c011 (table default.Table_c201704a319a4e078054899061f76615 [id=71234c2c912b4a79b7b08d87cab6fad0]) successfully deleted
W20240502 08:16:42.075385 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 72ee52bcc61b4816acda5b8140b63504 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72ee52bcc61b4816acda5b8140b63504 already in progress: creating tablet
W20240502 08:16:42.075615 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 577b149f52e44d5d9a9f6b9d85a6864d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 577b149f52e44d5d9a9f6b9d85a6864d already in progress: creating tablet
I20240502 08:16:42.075793 24179 ts_tablet_manager.cc:1888] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240502 08:16:42.075876 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 592ab8523b8d479d9caead321d5888c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 592ab8523b8d479d9caead321d5888c3 already in progress: creating tablet
I20240502 08:16:42.076102 24181 ts_tablet_manager.cc:1875] T fd57b0357ca749c389bc61add20cd3e7 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:42.077769 23606 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 642744194e22461f8316a78689fd99ff (table default.Table_2df50e9ecea9421b9bfebd892ebe62bd [id=18ab2c0716464d82a0e33bd99bd0735d]). Expected version 1 got 0
I20240502 08:16:42.082439 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d3c66c36f4cd494585b1e4cb114b81a9" } modify_external_catalogs: true
I20240502 08:16:42.082638 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d3c66c36f4cd494585b1e4cb114b81a9" } modify_external_catalogs: true
W20240502 08:16:42.084197 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2ebf55c8024c483f9ef0afe6945a02ff consensus state has no leader
I20240502 08:16:42.086108 23605 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet dead13bba95f43cb809e17f6d7fba028 (table default.Table_978594faab984d60ba03d29d40328a6c [id=7409c7e1a3534932a787559a679e7098]). Expected version 3 got 0
I20240502 08:16:42.086302 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d10cdf44509d4b4aabd9fb5c4a035f4b" } modify_external_catalogs: true
I20240502 08:16:42.086587 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d10cdf44509d4b4aabd9fb5c4a035f4b" } modify_external_catalogs: true
I20240502 08:16:42.087924 24181 ts_tablet_manager.cc:1888] T fd57b0357ca749c389bc61add20cd3e7 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:42.090097 24180 ts_tablet_manager.cc:1888] T 1065732d7333413d984ed56275fd61a9 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:42.092135 24180 log.cc:1196] T 1065732d7333413d984ed56275fd61a9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/1065732d7333413d984ed56275fd61a9
I20240502 08:16:42.092635 24180 ts_tablet_manager.cc:1909] T 1065732d7333413d984ed56275fd61a9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:42.094269 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 1065732d7333413d984ed56275fd61a9 (table default.Table_c422ad5c5f164908bfae0d2977635bb3 [id=3584daf64dec4be7a42a2eee267b4861]) successfully deleted
I20240502 08:16:42.094765 24180 tablet_replica.cc:330] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:42.095110 24180 raft_consensus.cc:2229] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:42.095468 24180 pending_rounds.cc:62] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:16:42.095625 24180 pending_rounds.cc:68] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156436165591040 op_type: NO_OP noop_request { }
I20240502 08:16:42.095889 24180 raft_consensus.cc:2866] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:42.096076 24180 raft_consensus.cc:2258] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:42.097218 24180 ts_tablet_manager.cc:1875] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:42.097775 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet b2eac3bde41c4415bdb97fc53a5b395f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:42.101959 24181 log.cc:1196] T fd57b0357ca749c389bc61add20cd3e7 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/fd57b0357ca749c389bc61add20cd3e7
I20240502 08:16:42.102475 24181 ts_tablet_manager.cc:1909] T fd57b0357ca749c389bc61add20cd3e7 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:42.102968 23603 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet fe888eb87575438b98568a74ca3e9f6d
I20240502 08:16:42.103389 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c466cab9104c4631a59e1d5e481c5fe0" } new_table_name: "default.Table_82fcfc40653a468598257e70678d34ea" modify_external_catalogs: true
I20240502 08:16:42.103559 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet fd57b0357ca749c389bc61add20cd3e7 (table default.Table_f52a4fa0e0c04f19a517f946b59a850f [id=fd141979a86c45bf9648396d086f80d4]) successfully deleted
I20240502 08:16:42.103886 24181 tablet_replica.cc:330] T 01675f637a5a4645949abfd87b3d09a2 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:42.104146 24181 raft_consensus.cc:2229] T 01675f637a5a4645949abfd87b3d09a2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:42.104293 24181 raft_consensus.cc:2258] T 01675f637a5a4645949abfd87b3d09a2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:42.104313 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_dbb8428a8edc4d03ab61e66b62e1890d" } modify_external_catalogs: true
I20240502 08:16:42.104553 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_dbb8428a8edc4d03ab61e66b62e1890d" } modify_external_catalogs: true
I20240502 08:16:42.105217 24181 ts_tablet_manager.cc:1875] T 01675f637a5a4645949abfd87b3d09a2 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:42.105257 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58 has no consensus state
I20240502 08:16:42.105752 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5d88e7a8816a45ccb7a49f559050dad2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.106338 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d88e7a8816a45ccb7a49f559050dad2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.112483 24181 ts_tablet_manager.cc:1888] T 01675f637a5a4645949abfd87b3d09a2 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:42.112685 24181 log.cc:1196] T 01675f637a5a4645949abfd87b3d09a2 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/01675f637a5a4645949abfd87b3d09a2
I20240502 08:16:42.113082 24181 ts_tablet_manager.cc:1909] T 01675f637a5a4645949abfd87b3d09a2 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:42.114189 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 01675f637a5a4645949abfd87b3d09a2 (table default.Table_c5d06caccb424872b965a751c171a0a6 [id=4411ea4d183f4694b16b6255567ab62b]) successfully deleted
I20240502 08:16:42.116690 24179 log.cc:1196] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/ce758c4a9258427987e8e83f579a2aa2
I20240502 08:16:42.104961 24175 tablet_bootstrap.cc:492] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:42.118839 24181 tablet_replica.cc:330] T 9700ea9dc58442e3ab1b7835a8c1e22c P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:42.119112 24181 raft_consensus.cc:2229] T 9700ea9dc58442e3ab1b7835a8c1e22c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:42.119333 24181 raft_consensus.cc:2258] T 9700ea9dc58442e3ab1b7835a8c1e22c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:42.122541 24175 ts_tablet_manager.cc:1367] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.075s user 0.000s sys 0.008s
I20240502 08:16:42.123654 24175 raft_consensus.cc:348] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.123970 24175 raft_consensus.cc:374] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.124809 24175 raft_consensus.cc:729] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.124917 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 768167b977c14b53a7537879e349d354
I20240502 08:16:42.125356 24175 consensus_queue.cc:260] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.126224 24181 ts_tablet_manager.cc:1875] T 9700ea9dc58442e3ab1b7835a8c1e22c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:42.126547 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet 768167b977c14b53a7537879e349d354 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:42.127030 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet 768167b977c14b53a7537879e349d354 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:42.130964 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e2792c45255c42adaeefb7aef4e25715 consensus state has no leader
I20240502 08:16:42.133819 24179 ts_tablet_manager.cc:1909] T ce758c4a9258427987e8e83f579a2aa2 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:42.134652 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet 768167b977c14b53a7537879e349d354 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:42.138144 24706 tablet.cc:1722] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:42.140547 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet ce758c4a9258427987e8e83f579a2aa2 (table default.Table_cb0ea06f5e0b47e5978fc9a5ed8880ce [id=9965057f98614e8284d007156af6fdd0]) successfully deleted
I20240502 08:16:42.140974 24702 tablet.cc:1722] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:42.141156 24179 tablet_replica.cc:330] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:42.141258 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet 4302fec5c51e4c6c841710bab50b2997 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:42.141552 24175 ts_tablet_manager.cc:1398] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240502 08:16:42.141971 24175 tablet_bootstrap.cc:492] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:42.143050 24704 tablet.cc:1722] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:42.144570 24175 tablet_bootstrap.cc:654] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.146754 24703 tablet.cc:1722] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:42.148309 24173 tablet_bootstrap.cc:492] T 5453cc0b93ac4cbb90a0588bb2100d60 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:42.148444 24179 raft_consensus.cc:2229] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:42.149394 24173 ts_tablet_manager.cc:1367] T 5453cc0b93ac4cbb90a0588bb2100d60 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.363s user 0.018s sys 0.016s
I20240502 08:16:42.149529 24705 tablet.cc:1722] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:42.150522 24180 ts_tablet_manager.cc:1888] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:42.149473 24179 raft_consensus.cc:2258] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:42.152052 23608 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet dead13bba95f43cb809e17f6d7fba028 (table default.Table_ef79f64b4af44b88a006a32b437c3207 [id=7409c7e1a3534932a787559a679e7098]). Expected version 4 got 0
I20240502 08:16:42.157531 24179 ts_tablet_manager.cc:1875] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:42.162954 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 0a2ff8af162d40cc87d99ec4fe91229d
I20240502 08:16:42.164338 23608 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet dead13bba95f43cb809e17f6d7fba028 (table default.Table_ef79f64b4af44b88a006a32b437c3207 [id=7409c7e1a3534932a787559a679e7098]). Expected version 4 got 3
I20240502 08:16:42.166940 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 91a3455e5931488da3134d6b5c18aef2
I20240502 08:16:42.169348 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a2ff8af162d40cc87d99ec4fe91229d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:42.171756 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a3455e5931488da3134d6b5c18aef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:42.174909 24175 tablet_bootstrap.cc:492] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:42.175168 24175 ts_tablet_manager.cc:1367] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.003s
I20240502 08:16:42.176054 24175 raft_consensus.cc:348] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.176411 24175 raft_consensus.cc:374] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.176556 24175 raft_consensus.cc:729] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.179682 23892 tablet_service.cc:1467] Processing CreateTablet for tablet f3b16f002d064b618760df1e77edaf17 (DEFAULT_TABLE table=default.Table_593fb38ca79d4c73aaf5f73fda9862ba [id=2882e539f12e456bb6f263ca7efc64b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.180181 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b16f002d064b618760df1e77edaf17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.183733 24175 consensus_queue.cc:260] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.184384 24175 ts_tablet_manager.cc:1398] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 08:16:42.184783 24175 tablet_bootstrap.cc:492] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:42.186010 23610 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_2df50e9ecea9421b9bfebd892ebe62bd [id=18ab2c0716464d82a0e33bd99bd0735d] alter complete (version 1)
I20240502 08:16:42.186964 24175 tablet_bootstrap.cc:654] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.187642 24706 tablet.cc:1722] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240502 08:16:42.194408 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 10adabdd31f848548ef6f2408ccc2050 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:42.194907 23610 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet dead13bba95f43cb809e17f6d7fba028 (table default.Table_ef79f64b4af44b88a006a32b437c3207 [id=7409c7e1a3534932a787559a679e7098]). Expected version 4 got 3
I20240502 08:16:42.170194 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a2ff8af162d40cc87d99ec4fe91229d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:42.171463 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a3455e5931488da3134d6b5c18aef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:42.178546 23757 tablet_service.cc:1467] Processing CreateTablet for tablet f3b16f002d064b618760df1e77edaf17 (DEFAULT_TABLE table=default.Table_593fb38ca79d4c73aaf5f73fda9862ba [id=2882e539f12e456bb6f263ca7efc64b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.199038 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b16f002d064b618760df1e77edaf17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.198292 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2c611d7af6428db289a7e2fbd586d4 (DEFAULT_TABLE table=default.Table_890e6db207d54accbbd91ba88c78095e [id=5242c465ca08472c9d38859ed79f95df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.201938 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2c611d7af6428db289a7e2fbd586d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.202219 24704 tablet.cc:1722] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240502 08:16:42.208680 24695 raft_consensus.cc:482] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:42.208851 24695 raft_consensus.cc:504] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.209718 24696 raft_consensus.cc:482] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:42.209738 24695 leader_election.cc:288] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:42.209674 24173 raft_consensus.cc:348] T 5453cc0b93ac4cbb90a0588bb2100d60 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.209928 24173 raft_consensus.cc:374] T 5453cc0b93ac4cbb90a0588bb2100d60 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.209846 24696 raft_consensus.cc:504] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:42.210032 24173 raft_consensus.cc:729] T 5453cc0b93ac4cbb90a0588bb2100d60 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.210523 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dead13bba95f43cb809e17f6d7fba028" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:42.210742 24696 leader_election.cc:288] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:42.210880 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dead13bba95f43cb809e17f6d7fba028" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:42.210956 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a3455e5931488da3134d6b5c18aef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.210991 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a2ff8af162d40cc87d99ec4fe91229d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.211609 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a19310948e248d1a85479c41c61bda2" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:42.210284 24173 consensus_queue.cc:260] T 5453cc0b93ac4cbb90a0588bb2100d60 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.212540 24173 ts_tablet_manager.cc:1398] T 5453cc0b93ac4cbb90a0588bb2100d60 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.062s user 0.002s sys 0.000s
W20240502 08:16:42.212610 23993 leader_election.cc:341] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:42.212837 24173 tablet_bootstrap.cc:492] T c5d28098dcbc4761a34ee65769d6fd7c P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:42.214632 24173 tablet_bootstrap.cc:654] T c5d28098dcbc4761a34ee65769d6fd7c P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.215641 24029 tablet_service.cc:1467] Processing CreateTablet for tablet f3b16f002d064b618760df1e77edaf17 (DEFAULT_TABLE table=default.Table_593fb38ca79d4c73aaf5f73fda9862ba [id=2882e539f12e456bb6f263ca7efc64b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.215914 24181 ts_tablet_manager.cc:1888] T 9700ea9dc58442e3ab1b7835a8c1e22c P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:42.216038 24181 log.cc:1196] T 9700ea9dc58442e3ab1b7835a8c1e22c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/9700ea9dc58442e3ab1b7835a8c1e22c
I20240502 08:16:42.216444 24181 ts_tablet_manager.cc:1909] T 9700ea9dc58442e3ab1b7835a8c1e22c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:42.217235 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b16f002d064b618760df1e77edaf17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.218107 23858 leader_election.cc:302] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:42.218612 24695 raft_consensus.cc:2726] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:42.222465 24696 consensus_queue.cc:1035] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:42.231792 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 5453cc0b93ac4cbb90a0588bb2100d60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.233251 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 9700ea9dc58442e3ab1b7835a8c1e22c (table default.Table_9741459a96e649fb8151cc28b642523e [id=2b9169f285754bfda0f4f8dc7f5cb653]) successfully deleted
I20240502 08:16:42.233950 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1f2e6b876bcf4bc39b3935a9a85e1219"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.235100 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f2e6b876bcf4bc39b3935a9a85e1219 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.238731 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2c611d7af6428db289a7e2fbd586d4 (DEFAULT_TABLE table=default.Table_890e6db207d54accbbd91ba88c78095e [id=5242c465ca08472c9d38859ed79f95df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.239058 24173 tablet_bootstrap.cc:492] T c5d28098dcbc4761a34ee65769d6fd7c P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:42.239244 24173 ts_tablet_manager.cc:1367] T c5d28098dcbc4761a34ee65769d6fd7c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.003s
I20240502 08:16:42.239284 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2c611d7af6428db289a7e2fbd586d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.240034 24173 raft_consensus.cc:348] T c5d28098dcbc4761a34ee65769d6fd7c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.240242 24173 raft_consensus.cc:374] T c5d28098dcbc4761a34ee65769d6fd7c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.240334 24173 raft_consensus.cc:729] T c5d28098dcbc4761a34ee65769d6fd7c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.240585 24173 consensus_queue.cc:260] T c5d28098dcbc4761a34ee65769d6fd7c P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.241086 24173 ts_tablet_manager.cc:1398] T c5d28098dcbc4761a34ee65769d6fd7c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:42.241400 24173 tablet_bootstrap.cc:492] T 0dff0d73a933427fb1f2686bac33eba1 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:42.244565 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e01d80203b2347cb9f8941844cdcc6fe" } new_table_name: "default.Table_94d88b6d81024599a0005647fa02c965" modify_external_catalogs: true
I20240502 08:16:42.245446 24179 ts_tablet_manager.cc:1888] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:42.245620 24179 log.cc:1196] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/3cc30014a1ac45de937751eed36877a9
I20240502 08:16:42.246084 24179 ts_tablet_manager.cc:1909] T 3cc30014a1ac45de937751eed36877a9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:42.246783 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 3cc30014a1ac45de937751eed36877a9 (table default.Table_d8aa2cd6c42444c48c383bf13e3728fd [id=9cbc066db19142d3bf0fd50cbe8fc91a]) successfully deleted
I20240502 08:16:42.216861 24180 log.cc:1196] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/c470d420c45843a69a63d9800b7cfc97
I20240502 08:16:42.247783 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_441797423e20442785b0b07f4161668f" } modify_external_catalogs: true
I20240502 08:16:42.251077 24173 tablet_bootstrap.cc:654] T 0dff0d73a933427fb1f2686bac33eba1 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.252434 24174 tablet_bootstrap.cc:492] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:42.252411 24181 tablet_replica.cc:330] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:42.253679 24179 tablet_replica.cc:330] T 6788ffd2eb2248c3b84e474de56c37a9 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:42.249287 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_59acaa50be384a71964771a569304fad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:42.253829 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_441797423e20442785b0b07f4161668f" } modify_external_catalogs: true
I20240502 08:16:42.249630 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0611d28b3eeb4bd8b06d3b02e17190b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:42.249717 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_662b6129b4274c9e851b05a110135289"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:42.254509 23605 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_ef79f64b4af44b88a006a32b437c3207 [id=7409c7e1a3534932a787559a679e7098] alter complete (version 4)
W20240502 08:16:42.254732 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_662b6129b4274c9e851b05a110135289 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:42.255282 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59acaa50be384a71964771a569304fad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:42.254354 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0611d28b3eeb4bd8b06d3b02e17190b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.255764 24179 raft_consensus.cc:2229] T 6788ffd2eb2248c3b84e474de56c37a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:42.255952 24179 raft_consensus.cc:2258] T 6788ffd2eb2248c3b84e474de56c37a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:42.256489 24179 ts_tablet_manager.cc:1875] T 6788ffd2eb2248c3b84e474de56c37a9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:42.251598 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet b2eac3bde41c4415bdb97fc53a5b395f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2eac3bde41c4415bdb97fc53a5b395f already in progress: creating tablet
W20240502 08:16:42.258292 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 3a74a6eb6d214d858903b3307e09e9e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a74a6eb6d214d858903b3307e09e9e6 already in progress: creating tablet
I20240502 08:16:42.258339 24630 raft_consensus.cc:482] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:16:42.258527 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 2b98cc8728c042f4b7f6496fcc5ef913 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b98cc8728c042f4b7f6496fcc5ef913 already in progress: creating tablet
I20240502 08:16:42.258474 24630 raft_consensus.cc:504] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
W20240502 08:16:42.258720 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 1305dbdac3994224a047df139cf8d518 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1305dbdac3994224a047df139cf8d518 already in progress: creating tablet
W20240502 08:16:42.258936 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 955e7bd2216646c6aef5acce8d5b0d77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 955e7bd2216646c6aef5acce8d5b0d77 already in progress: creating tablet
I20240502 08:16:42.259174 24630 leader_election.cc:288] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:42.259207 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet c5c377e037894728ac084b6500ad20de from {username='slave'} at 127.0.0.1:36668
W20240502 08:16:42.260633 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 981b7a7292884b1fb48af99db6e762c0 consensus state has no leader
I20240502 08:16:42.270121 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b816ea1d474f2db94119441146ba78" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:42.270756 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b816ea1d474f2db94119441146ba78" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:42.270994 24057 raft_consensus.cc:2454] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:42.271453 23723 leader_election.cc:302] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:42.271960 24630 raft_consensus.cc:2781] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:42.272073 24630 raft_consensus.cc:482] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:42.272153 24630 raft_consensus.cc:3037] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:42.273290 24179 ts_tablet_manager.cc:1888] T 6788ffd2eb2248c3b84e474de56c37a9 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240502 08:16:42.273315 23723 leader_election.cc:341] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:42.273429 24179 log.cc:1196] T 6788ffd2eb2248c3b84e474de56c37a9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/6788ffd2eb2248c3b84e474de56c37a9
I20240502 08:16:42.275102 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_823d8553cb624d3ab7a8df24374651ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.275753 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_823d8553cb624d3ab7a8df24374651ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.276976 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d28098dcbc4761a34ee65769d6fd7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.278407 24174 ts_tablet_manager.cc:1367] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.229s user 0.003s sys 0.002s
I20240502 08:16:42.279006 24630 raft_consensus.cc:504] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:42.279240 24174 raft_consensus.cc:348] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:42.279395 24174 raft_consensus.cc:374] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.279660 24630 leader_election.cc:288] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:42.280160 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b816ea1d474f2db94119441146ba78" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:42.280397 24057 raft_consensus.cc:3037] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:42.283301 24057 raft_consensus.cc:2454] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:42.283951 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet c5c377e037894728ac084b6500ad20de
I20240502 08:16:42.284571 23602 catalog_manager.cc:6554] ReplaceTablet: tablet c5c377e037894728ac084b6500ad20de deleted and replaced by tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6
I20240502 08:16:42.285928 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a52d298f258f4463a634e594b1254755"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.286629 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a52d298f258f4463a634e594b1254755 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.287312 23723 leader_election.cc:302] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:42.289013 24630 raft_consensus.cc:2781] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:42.289160 24630 raft_consensus.cc:686] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:42.289196 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_66741640bb3e41808da39a2de1103d0d" } new_table_name: "default.Table_29ca35f287d14925b43e3d6898fd1b12" modify_external_catalogs: true
I20240502 08:16:42.289458 24630 consensus_queue.cc:237] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:42.289204 23607 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 6a32a02a374942398dfbac414afe2292
I20240502 08:16:42.292171 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet c5c377e037894728ac084b6500ad20de with delete_type TABLET_DATA_DELETED (replaced by tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:42.292584 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet c5c377e037894728ac084b6500ad20de with delete_type TABLET_DATA_DELETED (replaced by tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.294937 24181 raft_consensus.cc:2229] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:42.295264 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a19310948e248d1a85479c41c61bda2" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:42.295656 24181 raft_consensus.cc:2258] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:42.295926 23996 leader_election.cc:341] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:42.296092 23996 leader_election.cc:302] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:42.296414 24696 raft_consensus.cc:2726] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:42.296597 24181 ts_tablet_manager.cc:1875] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:42.296831 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b816ea1d474f2db94119441146ba78" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:42.297466 23723 leader_election.cc:341] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:42.297618 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet c5c377e037894728ac084b6500ad20de with delete_type TABLET_DATA_DELETED (replaced by tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:42.297852 24179 ts_tablet_manager.cc:1909] T 6788ffd2eb2248c3b84e474de56c37a9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:42.298780 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 6788ffd2eb2248c3b84e474de56c37a9 (table default.Table_59462d2ccf514f0da864562120217931 [id=62273209104a4a8e9f24a9a77d2a90f0]) successfully deleted
I20240502 08:16:42.302172 24179 tablet_replica.cc:330] T c470d420c45843a69a63d9800b7cfc97 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:42.302438 24179 raft_consensus.cc:2229] T c470d420c45843a69a63d9800b7cfc97 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:42.303064 24179 raft_consensus.cc:2258] T c470d420c45843a69a63d9800b7cfc97 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:42.303697 24179 ts_tablet_manager.cc:1875] T c470d420c45843a69a63d9800b7cfc97 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:42.305706 24180 ts_tablet_manager.cc:1909] T c470d420c45843a69a63d9800b7cfc97 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:42.307286 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet c470d420c45843a69a63d9800b7cfc97 (table default.Table_95acc6e36a0b448eb3fd6d5df86e2de4 [id=1328b5d30ef24f98bf00c4429383ce92]) successfully deleted
I20240502 08:16:42.307713 24180 tablet_replica.cc:330] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:42.307978 24180 raft_consensus.cc:2229] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:42.308247 24180 raft_consensus.cc:2258] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:42.310782 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9feed0e1d82c472aaf0e9b92cd9173d5" } modify_external_catalogs: true
I20240502 08:16:42.311031 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9feed0e1d82c472aaf0e9b92cd9173d5" } modify_external_catalogs: true
I20240502 08:16:42.315482 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2c611d7af6428db289a7e2fbd586d4 (DEFAULT_TABLE table=default.Table_890e6db207d54accbbd91ba88c78095e [id=5242c465ca08472c9d38859ed79f95df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.315588 23757 tablet_service.cc:1467] Processing CreateTablet for tablet ed428ef1be8a49c19e4b6b48025b9cac (DEFAULT_TABLE table=default.Table_c31d9edf126f4ae6a1c06bc92f31a5b9 [id=43202b0bc3234eec8809f7a88275f464]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.316022 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2c611d7af6428db289a7e2fbd586d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.316061 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed428ef1be8a49c19e4b6b48025b9cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.316452 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d26ea2d33fbc414e84ffcc7193fa3a52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:42.316615 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ce5f27ca66f44a44af682629af53654f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.317046 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d26ea2d33fbc414e84ffcc7193fa3a52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:42.317250 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce5f27ca66f44a44af682629af53654f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.318838 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bdbe807258a5454795eaff8c27bb17dd" } new_table_name: "default.Table_efda0c8c5d1d4e138081bfe146a06fa4" modify_external_catalogs: true
I20240502 08:16:42.319432 23894 tablet_service.cc:1467] Processing CreateTablet for tablet ed428ef1be8a49c19e4b6b48025b9cac (DEFAULT_TABLE table=default.Table_c31d9edf126f4ae6a1c06bc92f31a5b9 [id=43202b0bc3234eec8809f7a88275f464]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.319897 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed428ef1be8a49c19e4b6b48025b9cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.324957 24024 tablet_service.cc:1467] Processing CreateTablet for tablet ed428ef1be8a49c19e4b6b48025b9cac (DEFAULT_TABLE table=default.Table_c31d9edf126f4ae6a1c06bc92f31a5b9 [id=43202b0bc3234eec8809f7a88275f464]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.325434 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed428ef1be8a49c19e4b6b48025b9cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.326599 23888 tablet_service.cc:1467] Processing CreateTablet for tablet b054783710824dbc8f7bc3ea38f605b0 (DEFAULT_TABLE table=default.Table_b4d8ce6957a042f486a41196af1caaaa [id=81907f5504a84a4f8d80666b3cec9d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.327059 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b054783710824dbc8f7bc3ea38f605b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.327332 24028 tablet_service.cc:1467] Processing CreateTablet for tablet b054783710824dbc8f7bc3ea38f605b0 (DEFAULT_TABLE table=default.Table_b4d8ce6957a042f486a41196af1caaaa [id=81907f5504a84a4f8d80666b3cec9d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.327713 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b054783710824dbc8f7bc3ea38f605b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.330175 23766 tablet_service.cc:1467] Processing CreateTablet for tablet b054783710824dbc8f7bc3ea38f605b0 (DEFAULT_TABLE table=default.Table_b4d8ce6957a042f486a41196af1caaaa [id=81907f5504a84a4f8d80666b3cec9d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.330544 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b054783710824dbc8f7bc3ea38f605b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.331271 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b3289c94e9274ea591f2b4e00a58a84d" } new_table_name: "default.Table_9b5b8dc599374049ac7d7fa21ac83a27" modify_external_catalogs: true
I20240502 08:16:42.332626 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7db65f49fb724e4cae21bcc31d769c17" } modify_external_catalogs: true
I20240502 08:16:42.332794 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7db65f49fb724e4cae21bcc31d769c17" } modify_external_catalogs: true
I20240502 08:16:42.333776 24695 raft_consensus.cc:482] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:42.333896 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_18ad0085c7a0415c854a64868bf0735c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:42.333962 24695 raft_consensus.cc:504] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:42.334357 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18ad0085c7a0415c854a64868bf0735c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.335597 23429 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2vVFYb/build/asan/bin/kudu
/tmp/dist-test-task2vVFYb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.125:36189
--webserver_interface=127.22.225.125
--webserver_port=41449
--builtin_ntp_servers=127.22.225.84:40725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.126:43265,127.22.225.125:36189,127.22.225.124:41745
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240502 08:16:42.349051 24695 leader_election.cc:288] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:42.350147 23610 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet e134e90efb4a48bb82b53a80bda708b3 (table default.Table_cc80d1cedafa4e71906e5e9530fdd828 [id=47682c26a85842c7b8991bf3828a9336]). Expected version 1 got 0
W20240502 08:16:42.350983 23858 leader_election.cc:341] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: b11f504b93574e42a9bf30fde6d08207
I20240502 08:16:42.352301 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58 (DEFAULT_TABLE table=default.Table_59270ebb70304dec9598e44f5d3bf506 [id=cc291e346cfe48f4b90e7240c8f25064]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.352838 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.352968 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2ff8af162d40cc87d99ec4fe91229d (DEFAULT_TABLE table=default.Table_dbb8428a8edc4d03ab61e66b62e1890d [id=a7733c75e74f45938ef50ddcf297d897]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.353300 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a2ff8af162d40cc87d99ec4fe91229d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.356086 24173 tablet_bootstrap.cc:492] T 0dff0d73a933427fb1f2686bac33eba1 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:42.356295 24173 ts_tablet_manager.cc:1367] T 0dff0d73a933427fb1f2686bac33eba1 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.115s user 0.016s sys 0.012s
I20240502 08:16:42.357266 24173 raft_consensus.cc:348] T 0dff0d73a933427fb1f2686bac33eba1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.357488 24173 raft_consensus.cc:374] T 0dff0d73a933427fb1f2686bac33eba1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.361111 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 29f82ad5860843f1adfbb40cdb21a4a3 (DEFAULT_TABLE table=default.Table_6fd5def10796410c9fe46e557794e914 [id=353606ecad684d6b8c48a3df100ebcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.361536 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f82ad5860843f1adfbb40cdb21a4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.362176 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58 (DEFAULT_TABLE table=default.Table_59270ebb70304dec9598e44f5d3bf506 [id=cc291e346cfe48f4b90e7240c8f25064]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.362322 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 29f82ad5860843f1adfbb40cdb21a4a3 (DEFAULT_TABLE table=default.Table_6fd5def10796410c9fe46e557794e914 [id=353606ecad684d6b8c48a3df100ebcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.362803 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 7d48d1f30fb34ee6a9a68908a3442d43 (DEFAULT_TABLE table=default.Table_bf53572d3cc544e1aa5674239aaa62f9 [id=1ed3c6918a9a43b69cc7dca9805de1c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.363200 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.363579 24643 raft_consensus.cc:482] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:42.363719 24643 raft_consensus.cc:504] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:42.364423 24643 leader_election.cc:288] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:42.365348 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b844458ae73f413cbf2113027a5af359" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:42.365607 23786 raft_consensus.cc:2454] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:42.365726 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f82ad5860843f1adfbb40cdb21a4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.366061 23993 leader_election.cc:302] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:42.366400 24643 raft_consensus.cc:2781] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:42.366510 24643 raft_consensus.cc:482] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:42.366593 24643 raft_consensus.cc:3037] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:42.369658 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 7d48d1f30fb34ee6a9a68908a3442d43 (DEFAULT_TABLE table=default.Table_bf53572d3cc544e1aa5674239aaa62f9 [id=1ed3c6918a9a43b69cc7dca9805de1c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.370082 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d48d1f30fb34ee6a9a68908a3442d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.377735 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d48d1f30fb34ee6a9a68908a3442d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.369599 24643 raft_consensus.cc:504] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:42.378368 24643 leader_election.cc:288] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
W20240502 08:16:42.379345 23861 leader_election.cc:341] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: b11f504b93574e42a9bf30fde6d08207
I20240502 08:16:42.379784 23861 leader_election.cc:302] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:42.379086 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b844458ae73f413cbf2113027a5af359" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:42.381721 23786 raft_consensus.cc:3037] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:42.384339 23786 raft_consensus.cc:2454] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:42.384866 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2b5aaf0ddb384715934e7eb81369ecac" } new_table_name: "default.Table_0c640f9efa084ae7953f217ce7601a4f" modify_external_catalogs: true
I20240502 08:16:42.385166 24173 raft_consensus.cc:729] T 0dff0d73a933427fb1f2686bac33eba1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.385586 23605 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet e134e90efb4a48bb82b53a80bda708b3 (table default.Table_cc80d1cedafa4e71906e5e9530fdd828 [id=47682c26a85842c7b8991bf3828a9336]). Expected version 1 got 0
I20240502 08:16:42.385900 23605 catalog_manager.cc:5484] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:42.386317 23605 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 49b816ea1d474f2db94119441146ba78 (table default.Table_20df1b504848439c869ae3739ac54185 [id=a0606a4053d04dd3814f09419a4a70c2]). Expected version 1 got 0
I20240502 08:16:42.386965 24173 consensus_queue.cc:260] T 0dff0d73a933427fb1f2686bac33eba1 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.387563 24173 ts_tablet_manager.cc:1398] T 0dff0d73a933427fb1f2686bac33eba1 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.031s user 0.000s sys 0.003s
I20240502 08:16:42.387929 24173 tablet_bootstrap.cc:492] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:42.380553 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 7d48d1f30fb34ee6a9a68908a3442d43 (DEFAULT_TABLE table=default.Table_bf53572d3cc544e1aa5674239aaa62f9 [id=1ed3c6918a9a43b69cc7dca9805de1c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.389922 24551 raft_consensus.cc:2726] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:42.390292 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b844458ae73f413cbf2113027a5af359" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:42.385208 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36f2d7b6f4e442ae95b2d1002d571b6a consensus state has no leader
I20240502 08:16:42.390975 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_20f1683b9d814b74a26458bb2c1c769b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.391520 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20f1683b9d814b74a26458bb2c1c769b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:42.391893 23996 leader_election.cc:341] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:42.392215 24695 raft_consensus.cc:482] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:42.392267 24173 tablet_bootstrap.cc:654] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.392263 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 29f82ad5860843f1adfbb40cdb21a4a3 (DEFAULT_TABLE table=default.Table_6fd5def10796410c9fe46e557794e914 [id=353606ecad684d6b8c48a3df100ebcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.392997 24695 raft_consensus.cc:504] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.394393 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e134e90efb4a48bb82b53a80bda708b3" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:42.395740 23993 leader_election.cc:302] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:42.396656 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b844458ae73f413cbf2113027a5af359" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:42.396953 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58 (DEFAULT_TABLE table=default.Table_59270ebb70304dec9598e44f5d3bf506 [id=cc291e346cfe48f4b90e7240c8f25064]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:42.397495 23996 leader_election.cc:341] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:42.397872 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d48d1f30fb34ee6a9a68908a3442d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.398429 24643 raft_consensus.cc:2781] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:42.398555 24643 raft_consensus.cc:686] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:42.398826 24643 consensus_queue.cc:237] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:42.399070 24695 leader_election.cc:288] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:42.399329 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:42.404966 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 045b0882a40f4eaba94935d9bc90da0c consensus state has no leader
I20240502 08:16:42.405810 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f82ad5860843f1adfbb40cdb21a4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.406325 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet d16110e1b1664802a98182c73f1e41d3
I20240502 08:16:42.408618 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 2c2ed5eee2b0412ab3f385df556e1e7a
I20240502 08:16:42.409750 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2ed5eee2b0412ab3f385df556e1e7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:42.409873 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_190eae2a8c64440f8d19842f8b6ac254" } modify_external_catalogs: true
I20240502 08:16:42.410032 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet d16110e1b1664802a98182c73f1e41d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:42.410068 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_190eae2a8c64440f8d19842f8b6ac254" } modify_external_catalogs: true
I20240502 08:16:42.413465 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet d16110e1b1664802a98182c73f1e41d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.415557 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b141c2b67ffe43ff97d32c9fff1d1fe0" } modify_external_catalogs: true
I20240502 08:16:42.415761 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b141c2b67ffe43ff97d32c9fff1d1fe0" } modify_external_catalogs: true
I20240502 08:16:42.418078 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dff0d73a933427fb1f2686bac33eba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.421466 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet d16110e1b1664802a98182c73f1e41d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:42.425138 24695 raft_consensus.cc:482] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:42.425266 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2ed5eee2b0412ab3f385df556e1e7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.425321 24695 raft_consensus.cc:504] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.428254 24695 leader_election.cc:288] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:42.431149 23925 raft_consensus.cc:3037] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:42.432413 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2ed5eee2b0412ab3f385df556e1e7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:42.433303 23861 leader_election.cc:341] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 1065732d7333413d984ed56275fd61a9
W20240502 08:16:42.433516 23858 leader_election.cc:341] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 1065732d7333413d984ed56275fd61a9
I20240502 08:16:42.434257 23858 leader_election.cc:302] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:42.435384 24695 raft_consensus.cc:2726] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:42.279479 24174 raft_consensus.cc:729] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.443410 24174 consensus_queue.cc:260] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:42.444052 24174 ts_tablet_manager.cc:1398] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.166s user 0.001s sys 0.001s
I20240502 08:16:42.444381 24174 tablet_bootstrap.cc:492] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:42.449307 24180 ts_tablet_manager.cc:1875] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:42.454114 23923 raft_consensus.cc:3037] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:42.455202 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2ff8af162d40cc87d99ec4fe91229d (DEFAULT_TABLE table=default.Table_dbb8428a8edc4d03ab61e66b62e1890d [id=a7733c75e74f45938ef50ddcf297d897]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.455884 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a2ff8af162d40cc87d99ec4fe91229d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.460397 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e134e90efb4a48bb82b53a80bda708b3" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:42.463936 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ebc89613d91747e1b59b01780555b212" } new_table_name: "default.Table_493b1b22b2e44d92b2eb1f9775a171bd" modify_external_catalogs: true
I20240502 08:16:42.465312 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet 45f02efa659e457ab57bc32ed3f66bb9 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:42.471841 23858 leader_election.cc:302] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:42.472579 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2ff8af162d40cc87d99ec4fe91229d (DEFAULT_TABLE table=default.Table_dbb8428a8edc4d03ab61e66b62e1890d [id=a7733c75e74f45938ef50ddcf297d897]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:42.477619 23993 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 81912.2us
I20240502 08:16:42.479812 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 57df4bde56a44809ad43d563baeefecb (DEFAULT_TABLE table=default.Table_ac9fe60ab8574ba9b6c57d9ffea15fb2 [id=4865a49282a348f1ae2ccf8f292c0672]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.480285 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57df4bde56a44809ad43d563baeefecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.480486 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 57df4bde56a44809ad43d563baeefecb (DEFAULT_TABLE table=default.Table_ac9fe60ab8574ba9b6c57d9ffea15fb2 [id=4865a49282a348f1ae2ccf8f292c0672]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.483744 24179 ts_tablet_manager.cc:1888] T c470d420c45843a69a63d9800b7cfc97 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:42.483963 24179 log.cc:1196] T c470d420c45843a69a63d9800b7cfc97 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/c470d420c45843a69a63d9800b7cfc97
I20240502 08:16:42.484702 24179 ts_tablet_manager.cc:1909] T c470d420c45843a69a63d9800b7cfc97 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:42.485662 24695 raft_consensus.cc:482] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:42.490370 24174 tablet_bootstrap.cc:654] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.494416 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_91aaef5fd943406aa3548482115afc65" } modify_external_catalogs: true
I20240502 08:16:42.494627 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_91aaef5fd943406aa3548482115afc65" } modify_external_catalogs: true
I20240502 08:16:42.494731 23606 catalog_manager.cc:5484] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67). New cstate: current_term: 1 leader_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:42.495337 23606 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet b844458ae73f413cbf2113027a5af359 (table default.Table_0a4ff982972e4fdaae1583619c9d6b04 [id=fd93259a378b4ea582a6661a59104abb]). Expected version 1 got 0
I20240502 08:16:42.495972 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 3449d7d1e8044b6cbf20766a5597e5ab (DEFAULT_TABLE table=default.Table_d4e98fc3f0334516b7074a56d74d8332 [id=a9b5841b2f054c77a70a61b8f8692434]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.496201 24551 raft_consensus.cc:2726] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:42.496362 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_dd2bda6955ae4ccab8e639637c37df7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.496976 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd2bda6955ae4ccab8e639637c37df7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.501117 24643 consensus_queue.cc:1035] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:42.501747 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e52b07c276964899970877ed4cbbcdee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.502288 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e52b07c276964899970877ed4cbbcdee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.504078 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 23ce83c8001d4b2988b88c0ed9bf948c
I20240502 08:16:42.504776 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 91f1ab141585441db46d3027833932b0
I20240502 08:16:42.505842 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 3449d7d1e8044b6cbf20766a5597e5ab (DEFAULT_TABLE table=default.Table_d4e98fc3f0334516b7074a56d74d8332 [id=a9b5841b2f054c77a70a61b8f8692434]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.506227 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3449d7d1e8044b6cbf20766a5597e5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.510283 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet 23ce83c8001d4b2988b88c0ed9bf948c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.510504 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f1ab141585441db46d3027833932b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.510593 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_347c7bcdb5674f45b6f1842416643c3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.511235 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_347c7bcdb5674f45b6f1842416643c3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.517666 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a2ff8af162d40cc87d99ec4fe91229d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.519137 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 23ce83c8001d4b2988b88c0ed9bf948c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:42.519546 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 3449d7d1e8044b6cbf20766a5597e5ab (DEFAULT_TABLE table=default.Table_d4e98fc3f0334516b7074a56d74d8332 [id=a9b5841b2f054c77a70a61b8f8692434]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.519730 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f1ab141585441db46d3027833932b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:42.520016 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3449d7d1e8044b6cbf20766a5597e5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.518250 24695 raft_consensus.cc:504] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.528779 24695 leader_election.cc:288] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:42.530489 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ace31be347a84e8096ec59fe11f3f0d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:42.532361 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 23ce83c8001d4b2988b88c0ed9bf948c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:42.533416 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f1ab141585441db46d3027833932b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:42.534593 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 25c8a6aaabe2413d8ce88f54ce45a4c4 (DEFAULT_TABLE table=default.Table_5fe628d3da624233941f440e3743da88 [id=ca161ce6780b446fa5743d3974cfa27b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.535120 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25c8a6aaabe2413d8ce88f54ce45a4c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.494063 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 57df4bde56a44809ad43d563baeefecb (DEFAULT_TABLE table=default.Table_ac9fe60ab8574ba9b6c57d9ffea15fb2 [id=4865a49282a348f1ae2ccf8f292c0672]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:42.542004 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ace31be347a84e8096ec59fe11f3f0d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.544693 23603 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 49b816ea1d474f2db94119441146ba78 (table default.Table_20df1b504848439c869ae3739ac54185 [id=a0606a4053d04dd3814f09419a4a70c2]). Expected version 1 got 0
I20240502 08:16:42.545158 23603 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet b6c52f1cae3a4ad38a9f95bfaf055e76 (table default.Table_35b808b0487f463dbcfa4a773a2a20f0 [id=922fb979b5894023a68820aece0b8c5e]). Expected version 1 got 0
I20240502 08:16:42.545423 23603 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 1ce5ccfd29724dc49afd0892b99f3169 (table default.Table_efda0c8c5d1d4e138081bfe146a06fa4 [id=3a9d7e8fe2c84cb4942fee9053a012b0]). Expected version 2 got 1
I20240502 08:16:42.546401 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57df4bde56a44809ad43d563baeefecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.547161 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3449d7d1e8044b6cbf20766a5597e5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.549186 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet f36aadfa1d264f23bcafa91744aed36d with delete_type TABLET_DATA_DELETED (replaced by tablet 2085ced52fff46ebaa98324cbb2d8c11) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:42.549850 23858 leader_election.cc:341] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 4302fec5c51e4c6c841710bab50b2997
I20240502 08:16:42.551266 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_06aacf94362b40b9afb9c267bdcef6a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.552080 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06aacf94362b40b9afb9c267bdcef6a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:42.552877 23861 leader_election.cc:341] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 4302fec5c51e4c6c841710bab50b2997
I20240502 08:16:42.553443 23858 leader_election.cc:302] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:42.553958 24695 raft_consensus.cc:2726] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:42.554059 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet c470d420c45843a69a63d9800b7cfc97 (table default.Table_95acc6e36a0b448eb3fd6d5df86e2de4 [id=1328b5d30ef24f98bf00c4429383ce92]) successfully deleted
W20240502 08:16:42.555356 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c395a29c7a1485eab39ca107dd91fca consensus state has no leader
W20240502 08:16:42.557286 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet f797d3c6bdc4493398954fbbf9943971 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f797d3c6bdc4493398954fbbf9943971 already in progress: creating tablet
I20240502 08:16:42.553368 24179 tablet_replica.cc:330] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:42.563622 24174 tablet_bootstrap.cc:492] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:42.568596 24174 ts_tablet_manager.cc:1367] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.124s user 0.004s sys 0.000s
I20240502 08:16:42.569676 24174 raft_consensus.cc:348] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:42.571688 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2ed5eee2b0412ab3f385df556e1e7a (DEFAULT_TABLE table=default.Table_7db65f49fb724e4cae21bcc31d769c17 [id=1cbc193cf0d14f64964397f6e55c6d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.580961 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2ed5eee2b0412ab3f385df556e1e7a (DEFAULT_TABLE table=default.Table_7db65f49fb724e4cae21bcc31d769c17 [id=1cbc193cf0d14f64964397f6e55c6d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.581995 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 25c8a6aaabe2413d8ce88f54ce45a4c4 (DEFAULT_TABLE table=default.Table_5fe628d3da624233941f440e3743da88 [id=ca161ce6780b446fa5743d3974cfa27b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.590381 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 02e6a1ba23dd4f82bc721a60c0526871
I20240502 08:16:42.593076 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_78883cd551c74a68b0e2a4dfe40e6955"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.596269 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78883cd551c74a68b0e2a4dfe40e6955 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.599771 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 523603a31b5b49e0a18a5a64f1357d64 (DEFAULT_TABLE table=default.Table_e628d856ede54061bb6736c1f1b8b0c1 [id=7607ea1299bc4493b9481f666bd74198]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.603065 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57df4bde56a44809ad43d563baeefecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.603838 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet 02e6a1ba23dd4f82bc721a60c0526871 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:42.604720 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2ed5eee2b0412ab3f385df556e1e7a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:42.605603 23596 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet b844458ae73f413cbf2113027a5af359 (table default.Table_0a4ff982972e4fdaae1583619c9d6b04 [id=fd93259a378b4ea582a6661a59104abb], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b844458ae73f413cbf2113027a5af359 consensus state has no leader
I20240502 08:16:42.607877 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_295f41167e6e481ea2443d5f8d02a998" } new_table_name: "default.Table_d132dfd0ffff4fd68be038259362c284" modify_external_catalogs: true
I20240502 08:16:42.617477 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_50749af5743f447ebb3195b85f663cf5" } new_table_name: "default.Table_6cde1d82c6b1472a843c94cccec9ee6c" modify_external_catalogs: true
I20240502 08:16:42.621305 24179 raft_consensus.cc:2229] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:42.621537 24179 raft_consensus.cc:2258] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:42.624130 24705 tablet.cc:1722] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:42.624348 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 523603a31b5b49e0a18a5a64f1357d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.624271 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 4645143a00d440a7b6d56c5e30d249f1 (DEFAULT_TABLE table=default.Table_b3278c5d6d0b4593ac33bc9977f5c3b1 [id=e86031550cec46a793785c0b3859fa92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.624732 24702 tablet.cc:1722] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:42.624992 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2ed5eee2b0412ab3f385df556e1e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.624886 24174 raft_consensus.cc:374] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.625169 24174 raft_consensus.cc:729] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.624749 24704 tablet.cc:1722] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:42.626084 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 02e6a1ba23dd4f82bc721a60c0526871 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.626886 24181 ts_tablet_manager.cc:1888] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:16:42.629781 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4645143a00d440a7b6d56c5e30d249f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.629868 24179 ts_tablet_manager.cc:1875] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:42.629968 24174 consensus_queue.cc:260] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:42.632900 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 25c8a6aaabe2413d8ce88f54ce45a4c4 (DEFAULT_TABLE table=default.Table_5fe628d3da624233941f440e3743da88 [id=ca161ce6780b446fa5743d3974cfa27b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.633307 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 4645143a00d440a7b6d56c5e30d249f1 (DEFAULT_TABLE table=default.Table_b3278c5d6d0b4593ac33bc9977f5c3b1 [id=e86031550cec46a793785c0b3859fa92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.639374 24180 ts_tablet_manager.cc:1888] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:42.640166 24174 ts_tablet_manager.cc:1398] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.070s user 0.003s sys 0.000s
I20240502 08:16:42.640869 24708 tablet.cc:1722] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:42.646211 24489 raft_consensus.cc:482] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:42.646956 24706 tablet.cc:1722] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:42.647537 24173 tablet_bootstrap.cc:492] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:42.650216 24173 ts_tablet_manager.cc:1367] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.260s user 0.020s sys 0.046s
I20240502 08:16:42.650683 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25c8a6aaabe2413d8ce88f54ce45a4c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.648007 24181 log.cc:1196] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/768167b977c14b53a7537879e349d354
I20240502 08:16:42.658296 23603 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet e134e90efb4a48bb82b53a80bda708b3 (table default.Table_cc80d1cedafa4e71906e5e9530fdd828 [id=47682c26a85842c7b8991bf3828a9336]). Expected version 1 got 0
I20240502 08:16:42.658607 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 45f02efa659e457ab57bc32ed3f66bb9
I20240502 08:16:42.659090 23602 catalog_manager.cc:6554] ReplaceTablet: tablet 45f02efa659e457ab57bc32ed3f66bb9 deleted and replaced by tablet df06e0edb57f45e5ac71b568c14769cd
I20240502 08:16:42.659337 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_82fcfc40653a468598257e70678d34ea" } new_table_name: "default.Table_4c264de039c8469284d9921dc3bdf875" modify_external_catalogs: true
I20240502 08:16:42.659583 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_136c710cf7d94381a2afea174adb4e56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.660137 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_136c710cf7d94381a2afea174adb4e56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.660852 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fb665eaa632a40188077e3127c5a3e9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.661377 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb665eaa632a40188077e3127c5a3e9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.663028 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9bb2be9b511a49eca4228b582c5ac739" } modify_external_catalogs: true
I20240502 08:16:42.663205 23601 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet dead13bba95f43cb809e17f6d7fba028 (table default.Table_ef79f64b4af44b88a006a32b437c3207 [id=7409c7e1a3534932a787559a679e7098]). Expected version 4 got 0
I20240502 08:16:42.663249 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9bb2be9b511a49eca4228b582c5ac739" } modify_external_catalogs: true
I20240502 08:16:42.663952 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8bae47845bfb4c78804a850866569860"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.664422 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8bae47845bfb4c78804a850866569860 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.666625 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_35b808b0487f463dbcfa4a773a2a20f0" } modify_external_catalogs: true
I20240502 08:16:42.666797 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_35b808b0487f463dbcfa4a773a2a20f0" } modify_external_catalogs: true
I20240502 08:16:42.666870 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25c8a6aaabe2413d8ce88f54ce45a4c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.667495 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2ed5eee2b0412ab3f385df556e1e7a (DEFAULT_TABLE table=default.Table_7db65f49fb724e4cae21bcc31d769c17 [id=1cbc193cf0d14f64964397f6e55c6d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.659060 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 523603a31b5b49e0a18a5a64f1357d64 (DEFAULT_TABLE table=default.Table_e628d856ede54061bb6736c1f1b8b0c1 [id=7607ea1299bc4493b9481f666bd74198]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.672164 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f02efa659e457ab57bc32ed3f66bb9 with delete_type TABLET_DATA_DELETED (replaced by tablet df06e0edb57f45e5ac71b568c14769cd) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.673077 24181 ts_tablet_manager.cc:1909] T 768167b977c14b53a7537879e349d354 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:42.676576 24173 raft_consensus.cc:348] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.676939 24173 raft_consensus.cc:374] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.677093 24173 raft_consensus.cc:729] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.677428 24173 consensus_queue.cc:260] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.678704 24173 ts_tablet_manager.cc:1398] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240502 08:16:42.682165 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 768167b977c14b53a7537879e349d354 (table default.Table_d10cdf44509d4b4aabd9fb5c4a035f4b [id=ab1669d2a12f4187809987ed41fa0379]) successfully deleted
I20240502 08:16:42.682854 24173 tablet_bootstrap.cc:492] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:42.683506 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 4645143a00d440a7b6d56c5e30d249f1 (DEFAULT_TABLE table=default.Table_b3278c5d6d0b4593ac33bc9977f5c3b1 [id=e86031550cec46a793785c0b3859fa92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.684126 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 523603a31b5b49e0a18a5a64f1357d64 (DEFAULT_TABLE table=default.Table_e628d856ede54061bb6736c1f1b8b0c1 [id=7607ea1299bc4493b9481f666bd74198]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.686882 23610 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 7a19310948e248d1a85479c41c61bda2 (table default.Table_785d5ddac96746379d1117bc8379566b [id=8521bfc13ef040a78cfb2cc4501b7f04]). Expected version 4 got 0
W20240502 08:16:42.686961 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 91a3455e5931488da3134d6b5c18aef2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91a3455e5931488da3134d6b5c18aef2 already in progress: creating tablet
W20240502 08:16:42.687340 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 0a2ff8af162d40cc87d99ec4fe91229d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a2ff8af162d40cc87d99ec4fe91229d already in progress: creating tablet
I20240502 08:16:42.687830 24181 tablet_replica.cc:330] T 5453cc0b93ac4cbb90a0588bb2100d60 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:42.688622 24181 raft_consensus.cc:2229] T 5453cc0b93ac4cbb90a0588bb2100d60 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:42.689321 24181 raft_consensus.cc:2258] T 5453cc0b93ac4cbb90a0588bb2100d60 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:42.690119 24173 tablet_bootstrap.cc:654] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.694103 24695 raft_consensus.cc:482] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:42.694272 24695 raft_consensus.cc:504] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.695041 24695 leader_election.cc:288] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:42.699909 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2ed5eee2b0412ab3f385df556e1e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.700294 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 523603a31b5b49e0a18a5a64f1357d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.700769 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4645143a00d440a7b6d56c5e30d249f1. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:42.718264 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e2792c45255c42adaeefb7aef4e25715 consensus state has no leader
I20240502 08:16:42.718765 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_70eb03c32f654f229093f610d22f9972" } new_table_name: "default.Table_d718a36474964fc09d2813f1210c5ce2" modify_external_catalogs: true
I20240502 08:16:42.720970 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 02e6a1ba23dd4f82bc721a60c0526871 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:42.729890 23886 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f02efa659e457ab57bc32ed3f66bb9 with delete_type TABLET_DATA_DELETED (replaced by tablet df06e0edb57f45e5ac71b568c14769cd) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:42.730749 24705 tablet.cc:1722] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240502 08:16:42.731623 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b4cedcada1f94d4296418229ee7512a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.732199 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4cedcada1f94d4296418229ee7512a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.742622 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d4bd37f8c6e54ecf82cd247846aa06d0" } modify_external_catalogs: true
I20240502 08:16:42.742830 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d4bd37f8c6e54ecf82cd247846aa06d0" } modify_external_catalogs: true
I20240502 08:16:42.752130 24181 ts_tablet_manager.cc:1875] T 5453cc0b93ac4cbb90a0588bb2100d60 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:42.753463 23861 leader_election.cc:341] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 10adabdd31f848548ef6f2408ccc2050
I20240502 08:16:42.753850 24643 raft_consensus.cc:482] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:42.754065 24643 raft_consensus.cc:504] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.755421 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 0f306a288b90413b9d8ea8b76d23363a
I20240502 08:16:42.757920 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f306a288b90413b9d8ea8b76d23363a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:42.759613 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fac7df16fc41d7a3297055a7258270" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:42.761042 24643 leader_election.cc:288] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
W20240502 08:16:42.761492 23996 leader_election.cc:341] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:42.761510 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f306a288b90413b9d8ea8b76d23363a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.761941 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f306a288b90413b9d8ea8b76d23363a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:42.762032 24174 tablet_bootstrap.cc:492] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:42.762068 24489 raft_consensus.cc:504] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
W20240502 08:16:42.761941 23858 leader_election.cc:341] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 10adabdd31f848548ef6f2408ccc2050
I20240502 08:16:42.762461 23858 leader_election.cc:302] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:42.762881 24489 leader_election.cc:288] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:42.762903 24695 raft_consensus.cc:2726] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:42.763152 24180 log.cc:1196] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/21832f0151ec4cf48f51f9bd2cb2b5ec
I20240502 08:16:42.763739 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4645143a00d440a7b6d56c5e30d249f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.766232 24174 tablet_bootstrap.cc:654] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.768388 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01675f637a5a4645949abfd87b3d09a2" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:42.768952 23723 leader_election.cc:341] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:42.769894 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 523603a31b5b49e0a18a5a64f1357d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.769913 24180 ts_tablet_manager.cc:1909] T 21832f0151ec4cf48f51f9bd2cb2b5ec P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:42.771689 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 21832f0151ec4cf48f51f9bd2cb2b5ec (table default.Table_75cadbecaad04383945a6e0b1e226d86 [id=4c539e8625094a278029356ac44da4c0]) successfully deleted
I20240502 08:16:42.775195 24180 tablet_replica.cc:330] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:42.775480 24180 raft_consensus.cc:2229] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:42.776193 24180 raft_consensus.cc:2258] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:42.778616 23723 leader_election.cc:341] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 01675f637a5a4645949abfd87b3d09a2
I20240502 08:16:42.778795 23723 leader_election.cc:302] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
W20240502 08:16:42.779052 23723 raft_consensus.cc:2670] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:16:42.781291 24702 tablet.cc:1722] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:42.781514 24706 tablet.cc:1722] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:42.784920 24174 tablet_bootstrap.cc:492] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:42.785086 24174 ts_tablet_manager.cc:1367] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.144s user 0.009s sys 0.001s
I20240502 08:16:42.785981 24174 raft_consensus.cc:348] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:42.786203 24174 raft_consensus.cc:374] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.786290 24174 raft_consensus.cc:729] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
W20240502 08:16:42.786442 24175 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.054s sys 0.111s
W20240502 08:16:42.786546 24175 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.598s user 0.054s sys 0.111s
I20240502 08:16:42.786527 24174 consensus_queue.cc:260] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:42.790205 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fac7df16fc41d7a3297055a7258270" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:42.790580 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f02efa659e457ab57bc32ed3f66bb9 with delete_type TABLET_DATA_DELETED (replaced by tablet df06e0edb57f45e5ac71b568c14769cd) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:42.792788 23993 leader_election.cc:341] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:42.792968 23993 leader_election.cc:302] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:42.793431 24643 raft_consensus.cc:2726] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:42.782265 24180 ts_tablet_manager.cc:1875] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:42.795732 24173 tablet_bootstrap.cc:492] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:42.795994 24173 ts_tablet_manager.cc:1367] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.113s user 0.010s sys 0.009s
I20240502 08:16:42.797042 24173 raft_consensus.cc:348] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.797616 24174 ts_tablet_manager.cc:1398] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240502 08:16:42.797894 24173 raft_consensus.cc:374] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.798051 24173 raft_consensus.cc:729] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.798323 24173 consensus_queue.cc:260] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.798929 24173 ts_tablet_manager.cc:1398] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:16:42.801339 24179 ts_tablet_manager.cc:1888] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:42.801508 24179 log.cc:1196] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/1065732d7333413d984ed56275fd61a9
I20240502 08:16:42.801891 24179 ts_tablet_manager.cc:1909] T 1065732d7333413d984ed56275fd61a9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:42.802305 24181 ts_tablet_manager.cc:1888] T 5453cc0b93ac4cbb90a0588bb2100d60 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:42.802444 24181 log.cc:1196] T 5453cc0b93ac4cbb90a0588bb2100d60 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/5453cc0b93ac4cbb90a0588bb2100d60
I20240502 08:16:42.802932 24181 ts_tablet_manager.cc:1909] T 5453cc0b93ac4cbb90a0588bb2100d60 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:42.803290 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 1065732d7333413d984ed56275fd61a9 (table default.Table_c422ad5c5f164908bfae0d2977635bb3 [id=3584daf64dec4be7a42a2eee267b4861]) successfully deleted
W20240502 08:16:42.803596 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 577b149f52e44d5d9a9f6b9d85a6864d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 577b149f52e44d5d9a9f6b9d85a6864d already in progress: creating tablet
I20240502 08:16:42.803741 24179 tablet_replica.cc:330] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
W20240502 08:16:42.803822 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 592ab8523b8d479d9caead321d5888c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 592ab8523b8d479d9caead321d5888c3 already in progress: creating tablet
I20240502 08:16:42.803930 24179 raft_consensus.cc:2229] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
W20240502 08:16:42.804057 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 72ee52bcc61b4816acda5b8140b63504 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72ee52bcc61b4816acda5b8140b63504 already in progress: creating tablet
I20240502 08:16:42.804116 24179 raft_consensus.cc:2258] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:42.804260 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet b2eac3bde41c4415bdb97fc53a5b395f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2eac3bde41c4415bdb97fc53a5b395f already in progress: creating tablet
I20240502 08:16:42.804703 24179 ts_tablet_manager.cc:1875] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:42.804741 24173 tablet_bootstrap.cc:492] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:42.806432 24181 tablet_replica.cc:330] T c5d28098dcbc4761a34ee65769d6fd7c P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:42.806483 24175 tablet_bootstrap.cc:492] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:42.806628 24175 ts_tablet_manager.cc:1367] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.622s user 0.057s sys 0.111s
I20240502 08:16:42.806674 24181 raft_consensus.cc:2229] T c5d28098dcbc4761a34ee65769d6fd7c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:42.807482 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 5453cc0b93ac4cbb90a0588bb2100d60 (table default.Table_a92f0fc8bc664448a6acb94dec9daa14 [id=8c859fafcd4b49f0b4e9fc9d8961c9ad]) successfully deleted
I20240502 08:16:42.807682 24181 raft_consensus.cc:2258] T c5d28098dcbc4761a34ee65769d6fd7c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:42.808207 24173 tablet_bootstrap.cc:654] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.808607 24181 ts_tablet_manager.cc:1875] T c5d28098dcbc4761a34ee65769d6fd7c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:42.810395 24705 tablet.cc:1722] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:42.811802 24174 tablet_bootstrap.cc:492] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:42.814544 24174 tablet_bootstrap.cc:654] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.817102 24705 tablet.cc:1722] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:42.820349 24175 raft_consensus.cc:348] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.820595 24175 raft_consensus.cc:374] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.820698 24175 raft_consensus.cc:729] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.820906 24175 consensus_queue.cc:260] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.821476 24175 ts_tablet_manager.cc:1398] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240502 08:16:42.821799 24175 tablet_bootstrap.cc:492] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:42.828495 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7929569b25404ea08efa4875a3c241a3" } modify_external_catalogs: true
I20240502 08:16:42.828723 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7929569b25404ea08efa4875a3c241a3" } modify_external_catalogs: true
I20240502 08:16:42.830545 24175 tablet_bootstrap.cc:654] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.832629 24175 tablet_bootstrap.cc:492] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:42.832823 24175 ts_tablet_manager.cc:1367] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240502 08:16:42.833650 24174 tablet_bootstrap.cc:492] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:42.833632 24175 raft_consensus.cc:348] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:42.833956 24175 raft_consensus.cc:374] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.834046 24175 raft_consensus.cc:729] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.834350 24175 consensus_queue.cc:260] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:42.835005 24175 ts_tablet_manager.cc:1398] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:16:42.835323 24175 tablet_bootstrap.cc:492] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:42.836889 24175 tablet_bootstrap.cc:654] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.829545 24705 tablet.cc:1722] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:42.848778 24181 ts_tablet_manager.cc:1888] T c5d28098dcbc4761a34ee65769d6fd7c P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:42.848938 24181 log.cc:1196] T c5d28098dcbc4761a34ee65769d6fd7c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/c5d28098dcbc4761a34ee65769d6fd7c
I20240502 08:16:42.849319 24181 ts_tablet_manager.cc:1909] T c5d28098dcbc4761a34ee65769d6fd7c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:42.849920 24175 tablet_bootstrap.cc:492] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:42.850113 24175 ts_tablet_manager.cc:1367] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.006s
I20240502 08:16:42.850980 24175 raft_consensus.cc:348] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.851187 24175 raft_consensus.cc:374] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.851305 24175 raft_consensus.cc:729] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.851372 24174 ts_tablet_manager.cc:1367] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.003s
I20240502 08:16:42.852739 24175 consensus_queue.cc:260] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.853262 23601 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet e134e90efb4a48bb82b53a80bda708b3 (table default.Table_cc80d1cedafa4e71906e5e9530fdd828 [id=47682c26a85842c7b8991bf3828a9336]). Expected version 1 got 0
I20240502 08:16:42.854223 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_59acaa50be384a71964771a569304fad" } new_table_name: "default.Table_417bf79bff5e4e01ab05528227bf9081" modify_external_catalogs: true
W20240502 08:16:42.857141 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4b291bdb81a4efbac7f6691e5577bbe consensus state has no leader
W20240502 08:16:42.857477 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2ebf55c8024c483f9ef0afe6945a02ff consensus state has no leader
I20240502 08:16:42.857658 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c67035cbec4e42109cb3d466e0a6ad18" } modify_external_catalogs: true
I20240502 08:16:42.857952 23610 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_efda0c8c5d1d4e138081bfe146a06fa4 [id=3a9d7e8fe2c84cb4942fee9053a012b0] alter complete (version 2)
I20240502 08:16:42.858381 23610 master_service.cc:946] ReplaceTablet: received request to replace tablet e47459b5f5bb4ecb96f76240fefa1f5d from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:42.859635 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet b6c52f1cae3a4ad38a9f95bfaf055e76
I20240502 08:16:42.861038 23607 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_0c640f9efa084ae7953f217ce7601a4f [id=ba62a673079d424992daa0f6b0c5d6b9] alter complete (version 3)
I20240502 08:16:42.861871 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet b6c52f1cae3a4ad38a9f95bfaf055e76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.862762 24173 tablet_bootstrap.cc:492] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:42.857754 23606 master_service.cc:946] ReplaceTablet: received request to replace tablet 1da01e2c2f054ccfb53383e70e16da84 from {username='slave'} at 127.0.0.1:36668
W20240502 08:16:42.864465 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ae9c817c7be42da8c689267c7eb5a06 consensus state has no leader
I20240502 08:16:42.857903 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c67035cbec4e42109cb3d466e0a6ad18" } modify_external_catalogs: true
I20240502 08:16:42.864765 24173 ts_tablet_manager.cc:1367] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.008s
I20240502 08:16:42.865664 24175 ts_tablet_manager.cc:1398] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:16:42.866053 24175 tablet_bootstrap.cc:492] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:42.866914 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet c5d28098dcbc4761a34ee65769d6fd7c (table default.Table_81d96be059774b5bbaf8c8d21b2935ce [id=dd9c1fd15bf446fd87bd3647cb7f041d]) successfully deleted
I20240502 08:16:42.866978 24173 raft_consensus.cc:348] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.867425 24173 raft_consensus.cc:374] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.867532 24173 raft_consensus.cc:729] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
W20240502 08:16:42.868506 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet c5c377e037894728ac084b6500ad20de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5c377e037894728ac084b6500ad20de already in progress: creating tablet
I20240502 08:16:42.869066 24173 consensus_queue.cc:260] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.869665 24173 ts_tablet_manager.cc:1398] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240502 08:16:42.869813 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fac7df16fc41d7a3297055a7258270 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.870011 24173 tablet_bootstrap.cc:492] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:42.870816 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet d16110e1b1664802a98182c73f1e41d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d16110e1b1664802a98182c73f1e41d3 already in progress: creating tablet
I20240502 08:16:42.871559 24181 tablet_replica.cc:330] T 0dff0d73a933427fb1f2686bac33eba1 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:42.871788 24181 raft_consensus.cc:2229] T 0dff0d73a933427fb1f2686bac33eba1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:42.872026 24181 raft_consensus.cc:2258] T 0dff0d73a933427fb1f2686bac33eba1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:42.872893 24181 ts_tablet_manager.cc:1875] T 0dff0d73a933427fb1f2686bac33eba1 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:42.873456 24179 ts_tablet_manager.cc:1888] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:42.874119 24179 log.cc:1196] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/b11f504b93574e42a9bf30fde6d08207
I20240502 08:16:42.876513 24173 tablet_bootstrap.cc:654] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.877400 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c41a9c564219469d9d8b285cd4c64711"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:42.877871 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet b6c52f1cae3a4ad38a9f95bfaf055e76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:42.878067 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c41a9c564219469d9d8b285cd4c64711 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.878186 24179 ts_tablet_manager.cc:1909] T b11f504b93574e42a9bf30fde6d08207 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:42.879050 24179 tablet_replica.cc:330] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:42.879418 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c002eb9168244875bfe05cafb44aae75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:42.879688 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet b11f504b93574e42a9bf30fde6d08207 (table default.Table_ec5e04e87fba492ab9a96d4034db240a [id=7900f407998344dfa30c1468428ef42a]) successfully deleted
W20240502 08:16:42.880029 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c002eb9168244875bfe05cafb44aae75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.881394 24179 raft_consensus.cc:2229] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:42.881327 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_bf29da2c486e479c8e6706cddeea1731"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:42.881788 24179 raft_consensus.cc:2258] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:42.881906 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf29da2c486e479c8e6706cddeea1731 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.882692 24179 ts_tablet_manager.cc:1875] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:42.886101 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet b8b1fa17dad143f0a7f1751e651eb81a
I20240502 08:16:42.887187 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b1fa17dad143f0a7f1751e651eb81a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.891393 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet d6a19bb8705c45c0bfc537d3a6bf8ae2
I20240502 08:16:42.891618 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e47459b5f5bb4ecb96f76240fefa1f5d
I20240502 08:16:42.892079 23610 catalog_manager.cc:6554] ReplaceTablet: tablet e47459b5f5bb4ecb96f76240fefa1f5d deleted and replaced by tablet 4917c1f0decf43c2aca74c67d8b45341
W20240502 08:16:42.892439 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 418ab502915c4e4ea9998dd2d0180c48 has no consensus state
I20240502 08:16:42.893440 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.893702 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet e47459b5f5bb4ecb96f76240fefa1f5d with delete_type TABLET_DATA_DELETED (replaced by tablet 4917c1f0decf43c2aca74c67d8b45341) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.895860 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1d04c3885cac4598a01f1669c65be7c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.896528 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d04c3885cac4598a01f1669c65be7c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.899667 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet 171406810543435e832a47ad5b3c80a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e5de793e74247dd9e73b7b1d58027d9) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.901510 24175 tablet_bootstrap.cc:654] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.904332 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_20e4d00ccdf240fd922450cb02438505"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.904925 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20e4d00ccdf240fd922450cb02438505 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.905959 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a47e8e08a0ef4797aaf83cfa8adf0019"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.906601 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a47e8e08a0ef4797aaf83cfa8adf0019 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.907702 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b1fa17dad143f0a7f1751e651eb81a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:42.908151 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:42.908741 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet e47459b5f5bb4ecb96f76240fefa1f5d with delete_type TABLET_DATA_DELETED (replaced by tablet 4917c1f0decf43c2aca74c67d8b45341) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:42.910290 24179 ts_tablet_manager.cc:1888] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:16:42.910795 24179 log.cc:1196] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/768167b977c14b53a7537879e349d354
I20240502 08:16:42.912153 24179 ts_tablet_manager.cc:1909] T 768167b977c14b53a7537879e349d354 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:42.912632 24179 tablet_replica.cc:330] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:42.912786 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet b6c52f1cae3a4ad38a9f95bfaf055e76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:42.912822 24179 raft_consensus.cc:2229] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:42.913015 24179 raft_consensus.cc:2258] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:42.913558 24179 ts_tablet_manager.cc:1875] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:42.915679 24175 tablet_bootstrap.cc:492] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:42.915874 24175 ts_tablet_manager.cc:1367] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.004s
I20240502 08:16:42.916831 24175 raft_consensus.cc:348] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.917045 24175 raft_consensus.cc:374] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.917119 24175 raft_consensus.cc:729] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.917368 24175 consensus_queue.cc:260] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.917937 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet e47459b5f5bb4ecb96f76240fefa1f5d with delete_type TABLET_DATA_DELETED (replaced by tablet 4917c1f0decf43c2aca74c67d8b45341) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:42.918025 24175 ts_tablet_manager.cc:1398] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:16:42.918164 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:42.918308 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b1fa17dad143f0a7f1751e651eb81a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:42.918279 24175 tablet_bootstrap.cc:492] T fd57b0357ca749c389bc61add20cd3e7 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:42.919904 24175 tablet_bootstrap.cc:654] T fd57b0357ca749c389bc61add20cd3e7 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.921521 24181 ts_tablet_manager.cc:1888] T 0dff0d73a933427fb1f2686bac33eba1 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:42.921708 24181 log.cc:1196] T 0dff0d73a933427fb1f2686bac33eba1 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/0dff0d73a933427fb1f2686bac33eba1
I20240502 08:16:42.922125 24181 ts_tablet_manager.cc:1909] T 0dff0d73a933427fb1f2686bac33eba1 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:42.923488 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 0dff0d73a933427fb1f2686bac33eba1 (table default.Table_6ccfa5bc466242b2a325aa1ce6c85e9c [id=9076329a97254502a8c10b167c7f5b41]) successfully deleted
I20240502 08:16:42.923889 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8b1436ec53e349658e5e39d3e10da3e1" } new_table_name: "default.Table_063b529ba90f40ffafa8016535fe31bd" modify_external_catalogs: true
I20240502 08:16:42.933157 24643 raft_consensus.cc:482] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:42.933372 24643 raft_consensus.cc:504] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:42.934198 24643 leader_election.cc:288] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:42.935660 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1be2c29d3154d9189af3266191f0b3e" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:42.936338 23993 leader_election.cc:341] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:42.937301 24180 ts_tablet_manager.cc:1888] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:42.937521 24180 log.cc:1196] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/01675f637a5a4645949abfd87b3d09a2
I20240502 08:16:42.938009 24180 ts_tablet_manager.cc:1909] T 01675f637a5a4645949abfd87b3d09a2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:42.939662 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6c98fa55fddc49b285544085ec459fd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.939864 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 001e3f4f72f64dc69dbd521724912f07 consensus state has no leader
W20240502 08:16:42.940330 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c98fa55fddc49b285544085ec459fd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:42.944468 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 2c2ed5eee2b0412ab3f385df556e1e7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c2ed5eee2b0412ab3f385df556e1e7a already in progress: creating tablet
I20240502 08:16:42.945914 24181 tablet_replica.cc:330] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:42.946285 24181 raft_consensus.cc:2229] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:42.946570 24181 raft_consensus.cc:2258] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:42.947436 24181 ts_tablet_manager.cc:1875] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:42.954176 23766 tablet_service.cc:1467] Processing CreateTablet for tablet fbcf17a3bcc844b1b9676580f0749c9c (DEFAULT_TABLE table=default.Table_20f1683b9d814b74a26458bb2c1c769b [id=f8b6167466e84ad1b0f01df0be64a9b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.954224 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_696cfb6ac5024b57aa3d8113d3be2ead"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:42.955281 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_696cfb6ac5024b57aa3d8113d3be2ead in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:42.958457 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 768167b977c14b53a7537879e349d354 (table default.Table_d10cdf44509d4b4aabd9fb5c4a035f4b [id=ab1669d2a12f4187809987ed41fa0379]) successfully deleted
I20240502 08:16:42.960538 23756 tablet_service.cc:1467] Processing CreateTablet for tablet f1d8641acec9467fbc53b2ab90e04142 (DEFAULT_TABLE table=default.Table_ce5f27ca66f44a44af682629af53654f [id=5cabdba53fd742e584e1be1e14f471e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.956028 24030 tablet_service.cc:1467] Processing CreateTablet for tablet fbcf17a3bcc844b1b9676580f0749c9c (DEFAULT_TABLE table=default.Table_20f1683b9d814b74a26458bb2c1c769b [id=f8b6167466e84ad1b0f01df0be64a9b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.963001 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbcf17a3bcc844b1b9676580f0749c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.962358 24029 tablet_service.cc:1467] Processing CreateTablet for tablet f1d8641acec9467fbc53b2ab90e04142 (DEFAULT_TABLE table=default.Table_ce5f27ca66f44a44af682629af53654f [id=5cabdba53fd742e584e1be1e14f471e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.966034 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d8641acec9467fbc53b2ab90e04142. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.972476 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 1da01e2c2f054ccfb53383e70e16da84
I20240502 08:16:42.972929 23606 catalog_manager.cc:6554] ReplaceTablet: tablet 1da01e2c2f054ccfb53383e70e16da84 deleted and replaced by tablet 3e36be367f9b4c568ce0f3278b6297d9
I20240502 08:16:42.973404 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 1da01e2c2f054ccfb53383e70e16da84 with delete_type TABLET_DATA_DELETED (replaced by tablet 3e36be367f9b4c568ce0f3278b6297d9) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.973764 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 76e0f767c6534a608023e6e6d33ca7a8
I20240502 08:16:42.974170 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet 1da01e2c2f054ccfb53383e70e16da84 with delete_type TABLET_DATA_DELETED (replaced by tablet 3e36be367f9b4c568ce0f3278b6297d9) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:42.975020 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 76e0f767c6534a608023e6e6d33ca7a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:42.977973 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet 76e0f767c6534a608023e6e6d33ca7a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:42.978374 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c44e58448899483fb6f36f8ee844e61d" } modify_external_catalogs: true
I20240502 08:16:42.978606 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c44e58448899483fb6f36f8ee844e61d" } modify_external_catalogs: true
I20240502 08:16:42.979720 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbcf17a3bcc844b1b9676580f0749c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.981900 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d8641acec9467fbc53b2ab90e04142. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:42.985059 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet fd57b0357ca749c389bc61add20cd3e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:42.985337 24179 ts_tablet_manager.cc:1888] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:42.985502 24179 log.cc:1196] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/4302fec5c51e4c6c841710bab50b2997
I20240502 08:16:42.988629 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 2411259b872e4be4b8eb8832b36094c4 (DEFAULT_TABLE table=default.Table_1f2e6b876bcf4bc39b3935a9a85e1219 [id=d74c53fec63c4d7f8bb7ff077a09e67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:42.989182 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2411259b872e4be4b8eb8832b36094c4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:42.998701 23723 consensus_peers.cc:485] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 768167b977c14b53a7537879e349d354. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:42.999310 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 01675f637a5a4645949abfd87b3d09a2 (table default.Table_c5d06caccb424872b965a751c171a0a6 [id=4411ea4d183f4694b16b6255567ab62b]) successfully deleted
W20240502 08:16:43.000082 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet b2eac3bde41c4415bdb97fc53a5b395f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2eac3bde41c4415bdb97fc53a5b395f already in progress: creating tablet
I20240502 08:16:43.000466 24180 tablet_replica.cc:330] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:43.000723 24180 raft_consensus.cc:2229] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:43.001061 24180 raft_consensus.cc:2258] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:43.001740 24180 ts_tablet_manager.cc:1875] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:43.004295 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 2411259b872e4be4b8eb8832b36094c4 (DEFAULT_TABLE table=default.Table_1f2e6b876bcf4bc39b3935a9a85e1219 [id=d74c53fec63c4d7f8bb7ff077a09e67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.004810 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2411259b872e4be4b8eb8832b36094c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.008944 24695 raft_consensus.cc:482] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:43.009112 24695 raft_consensus.cc:504] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:43.009845 24695 leader_election.cc:288] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:43.010236 24179 ts_tablet_manager.cc:1909] T 4302fec5c51e4c6c841710bab50b2997 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:43.010710 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 418ab502915c4e4ea9998dd2d0180c48 (DEFAULT_TABLE table=default.Table_417bf79bff5e4e01ab05528227bf9081 [id=d13560bbbb3749ac9fb456bb82cd6479]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.011140 24179 tablet_replica.cc:330] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:43.011348 24179 raft_consensus.cc:2229] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:43.011523 24179 raft_consensus.cc:2258] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:43.011444 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 418ab502915c4e4ea9998dd2d0180c48 (DEFAULT_TABLE table=default.Table_417bf79bff5e4e01ab05528227bf9081 [id=d13560bbbb3749ac9fb456bb82cd6479]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.011847 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 418ab502915c4e4ea9998dd2d0180c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.012085 24179 ts_tablet_manager.cc:1875] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:43.013820 23861 leader_election.cc:341] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: b6417aafbdcf416ea7bff56bce5e6ddb
I20240502 08:16:43.015964 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 418ab502915c4e4ea9998dd2d0180c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.016206 24489 raft_consensus.cc:482] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:43.016403 24489 raft_consensus.cc:504] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.017290 24489 leader_election.cc:288] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:43.018618 23723 leader_election.cc:341] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: f36aadfa1d264f23bcafa91744aed36d
I20240502 08:16:43.019722 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e5f5597402374527bf0e5e50178df713
I20240502 08:16:43.020566 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet e5f5597402374527bf0e5e50178df713 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:43.020896 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet e5f5597402374527bf0e5e50178df713 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.021147 24174 raft_consensus.cc:348] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.025084 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1be2c29d3154d9189af3266191f0b3e" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:43.025933 23996 leader_election.cc:341] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:43.026127 23996 leader_election.cc:302] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:43.026813 24643 raft_consensus.cc:2726] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:43.027626 24174 raft_consensus.cc:374] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:43.027750 24174 raft_consensus.cc:729] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:43.028059 24174 consensus_queue.cc:260] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.031011 23892 tablet_service.cc:1467] Processing CreateTablet for tablet fbcf17a3bcc844b1b9676580f0749c9c (DEFAULT_TABLE table=default.Table_20f1683b9d814b74a26458bb2c1c769b [id=f8b6167466e84ad1b0f01df0be64a9b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.031076 24024 tablet_service.cc:1467] Processing CreateTablet for tablet cb59a231a2a74ee4ada0c23b2fa77645 (DEFAULT_TABLE table=default.Table_662b6129b4274c9e851b05a110135289 [id=9b900be482144915bbbe709b20cd4acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.031424 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb59a231a2a74ee4ada0c23b2fa77645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.032533 23756 tablet_service.cc:1467] Processing CreateTablet for tablet cb59a231a2a74ee4ada0c23b2fa77645 (DEFAULT_TABLE table=default.Table_662b6129b4274c9e851b05a110135289 [id=9b900be482144915bbbe709b20cd4acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.032920 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb59a231a2a74ee4ada0c23b2fa77645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.037864 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbcf17a3bcc844b1b9676580f0749c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.038154 24174 ts_tablet_manager.cc:1398] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.187s user 0.002s sys 0.000s
I20240502 08:16:43.039424 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 1da01e2c2f054ccfb53383e70e16da84 with delete_type TABLET_DATA_DELETED (replaced by tablet 3e36be367f9b4c568ce0f3278b6297d9) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:43.039963 23603 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 36f2d7b6f4e442ae95b2d1002d571b6a (table default.Table_29ca35f287d14925b43e3d6898fd1b12 [id=a95a99a58cde44fe814b3edfcc643a80]). Expected version 1 got 0
I20240502 08:16:43.041394 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet 76e0f767c6534a608023e6e6d33ca7a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:43.041923 23898 tablet_service.cc:1467] Processing CreateTablet for tablet f1d8641acec9467fbc53b2ab90e04142 (DEFAULT_TABLE table=default.Table_ce5f27ca66f44a44af682629af53654f [id=5cabdba53fd742e584e1be1e14f471e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.042766 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet e5f5597402374527bf0e5e50178df713 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:43.042968 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 91646e7af3824520a1fe28e180eeb0a7 (DEFAULT_TABLE table=default.Table_0611d28b3eeb4bd8b06d3b02e17190b0 [id=b096761c868742e2b6eaf3f2dffdcb92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.043503 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 418ab502915c4e4ea9998dd2d0180c48 (DEFAULT_TABLE table=default.Table_417bf79bff5e4e01ab05528227bf9081 [id=d13560bbbb3749ac9fb456bb82cd6479]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.043984 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_83695afdff8e4b29a56002056ac46b93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.044626 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83695afdff8e4b29a56002056ac46b93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.044667 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 2411259b872e4be4b8eb8832b36094c4 (DEFAULT_TABLE table=default.Table_1f2e6b876bcf4bc39b3935a9a85e1219 [id=d74c53fec63c4d7f8bb7ff077a09e67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.044903 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d8641acec9467fbc53b2ab90e04142. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.045177 23603 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 36f2d7b6f4e442ae95b2d1002d571b6a (table default.Table_29ca35f287d14925b43e3d6898fd1b12 [id=a95a99a58cde44fe814b3edfcc643a80]). Expected version 1 got 0
I20240502 08:16:43.045258 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 418ab502915c4e4ea9998dd2d0180c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.046113 24181 ts_tablet_manager.cc:1888] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:43.046267 24181 log.cc:1196] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/23ce83c8001d4b2988b88c0ed9bf948c
I20240502 08:16:43.046677 24181 ts_tablet_manager.cc:1909] T 23ce83c8001d4b2988b88c0ed9bf948c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:43.047104 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2411259b872e4be4b8eb8832b36094c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.047444 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91646e7af3824520a1fe28e180eeb0a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.047652 23891 tablet_service.cc:1467] Processing CreateTablet for tablet cb59a231a2a74ee4ada0c23b2fa77645 (DEFAULT_TABLE table=default.Table_662b6129b4274c9e851b05a110135289 [id=9b900be482144915bbbe709b20cd4acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.048239 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 23ce83c8001d4b2988b88c0ed9bf948c (table default.Table_190eae2a8c64440f8d19842f8b6ac254 [id=54454efbe9394170822f237dc8e6115e]) successfully deleted
W20240502 08:16:43.048681 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 91f1ab141585441db46d3027833932b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91f1ab141585441db46d3027833932b0 already in progress: creating tablet
I20240502 08:16:43.048919 24181 tablet_replica.cc:330] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
W20240502 08:16:43.048913 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 02e6a1ba23dd4f82bc721a60c0526871 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02e6a1ba23dd4f82bc721a60c0526871 already in progress: creating tablet
I20240502 08:16:43.049140 24181 raft_consensus.cc:2229] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:43.049358 24181 raft_consensus.cc:2258] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:43.049499 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 45f02efa659e457ab57bc32ed3f66bb9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45f02efa659e457ab57bc32ed3f66bb9 already in progress: creating tablet
W20240502 08:16:43.049757 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 0f306a288b90413b9d8ea8b76d23363a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f306a288b90413b9d8ea8b76d23363a already in progress: creating tablet
I20240502 08:16:43.050232 24181 ts_tablet_manager.cc:1875] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:43.051192 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 4302fec5c51e4c6c841710bab50b2997 (table default.Table_006d60eaae72439f829df411c32c1514 [id=e3f148f833bc4d99a3ab2ec8d5258507]) successfully deleted
W20240502 08:16:43.051462 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 0a2ff8af162d40cc87d99ec4fe91229d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a2ff8af162d40cc87d99ec4fe91229d already in progress: creating tablet
W20240502 08:16:43.051687 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 91a3455e5931488da3134d6b5c18aef2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91a3455e5931488da3134d6b5c18aef2 already in progress: creating tablet
I20240502 08:16:43.053135 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 91646e7af3824520a1fe28e180eeb0a7 (DEFAULT_TABLE table=default.Table_0611d28b3eeb4bd8b06d3b02e17190b0 [id=b096761c868742e2b6eaf3f2dffdcb92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.053544 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91646e7af3824520a1fe28e180eeb0a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.053778 24179 ts_tablet_manager.cc:1888] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:43.057608 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb59a231a2a74ee4ada0c23b2fa77645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.058779 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 657bcfa3d93e4809a78c7ba5692d9b90 (DEFAULT_TABLE table=default.Table_acb191a2bab44ddd84855f88d5b7e073 [id=f1512e16988841d98807bdbb71a48598]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.059263 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657bcfa3d93e4809a78c7ba5692d9b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.062088 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 91646e7af3824520a1fe28e180eeb0a7 (DEFAULT_TABLE table=default.Table_0611d28b3eeb4bd8b06d3b02e17190b0 [id=b096761c868742e2b6eaf3f2dffdcb92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.062666 24173 tablet_bootstrap.cc:492] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:43.062808 24173 ts_tablet_manager.cc:1367] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.193s user 0.009s sys 0.047s
I20240502 08:16:43.063140 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36aadfa1d264f23bcafa91744aed36d" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:43.063678 24174 tablet_bootstrap.cc:492] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:43.064270 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 657bcfa3d93e4809a78c7ba5692d9b90 (DEFAULT_TABLE table=default.Table_acb191a2bab44ddd84855f88d5b7e073 [id=f1512e16988841d98807bdbb71a48598]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.064677 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657bcfa3d93e4809a78c7ba5692d9b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.064564 24173 raft_consensus.cc:348] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:43.064776 24173 raft_consensus.cc:374] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:43.064857 24173 raft_consensus.cc:729] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:43.065200 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91646e7af3824520a1fe28e180eeb0a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.065133 24173 consensus_queue.cc:260] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:43.065551 24175 tablet_bootstrap.cc:492] T fd57b0357ca749c389bc61add20cd3e7 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:43.065721 24173 ts_tablet_manager.cc:1398] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:16:43.067442 24173 tablet_bootstrap.cc:492] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:43.071522 23858 leader_election.cc:341] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: b6417aafbdcf416ea7bff56bce5e6ddb
I20240502 08:16:43.071671 23858 leader_election.cc:302] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:43.071789 23602 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet e134e90efb4a48bb82b53a80bda708b3 (table default.Table_cc80d1cedafa4e71906e5e9530fdd828 [id=47682c26a85842c7b8991bf3828a9336]). Expected version 1 got 0
I20240502 08:16:43.073695 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_da999521d34e478fab2c024a89fa0025" } new_table_name: "default.Table_792c73b104fe4271b1fe9ab5d3445a8c" modify_external_catalogs: true
I20240502 08:16:43.075251 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_823d8553cb624d3ab7a8df24374651ab" } new_table_name: "default.Table_92759ae885e24e0395e7a8b3f6f59181" modify_external_catalogs: true
I20240502 08:16:43.076181 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 848a0865eb2b4828a2594a17d96869d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.076699 24179 log.cc:1196] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/10adabdd31f848548ef6f2408ccc2050
I20240502 08:16:43.076783 24175 ts_tablet_manager.cc:1367] T fd57b0357ca749c389bc61add20cd3e7 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.148s user 0.019s sys 0.047s
I20240502 08:16:43.077729 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c575f9ccf4c4495390078283ae331c7b" } new_table_name: "default.Table_aded763332144f979b418931264d1eb5" modify_external_catalogs: true
I20240502 08:16:43.077919 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fc4d6fa243f34d899165684a654feb3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.078441 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc4d6fa243f34d899165684a654feb3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.079057 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_59f82dbde55b406ba5271520c5bab104" } modify_external_catalogs: true
I20240502 08:16:43.079157 24695 raft_consensus.cc:2726] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:43.079248 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_59f82dbde55b406ba5271520c5bab104" } modify_external_catalogs: true
I20240502 08:16:43.079906 23890 tablet_service.cc:1467] Processing CreateTablet for tablet d2f7ccb4385d4c3cad54758d765e8452 (DEFAULT_TABLE table=default.Table_fa986dbf19ba412dbe12164c322d7558 [id=790ab064f86b4f65a609846ba2192ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.082731 23750 tablet_service.cc:1467] Processing CreateTablet for tablet d2f7ccb4385d4c3cad54758d765e8452 (DEFAULT_TABLE table=default.Table_fa986dbf19ba412dbe12164c322d7558 [id=790ab064f86b4f65a609846ba2192ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.084509 24179 ts_tablet_manager.cc:1909] T 10adabdd31f848548ef6f2408ccc2050 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:43.084839 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 657bcfa3d93e4809a78c7ba5692d9b90 (DEFAULT_TABLE table=default.Table_acb191a2bab44ddd84855f88d5b7e073 [id=f1512e16988841d98807bdbb71a48598]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.085099 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 10adabdd31f848548ef6f2408ccc2050 (table default.Table_c45e7b653ae641b293b87c150748c864 [id=5ff5989d908243c9a58013285842dd46]) successfully deleted
I20240502 08:16:43.085676 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2f7ccb4385d4c3cad54758d765e8452. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:43.086536 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet c5c377e037894728ac084b6500ad20de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5c377e037894728ac084b6500ad20de already in progress: creating tablet
I20240502 08:16:43.087041 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet 21832f0151ec4cf48f51f9bd2cb2b5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:43.088331 23723 leader_election.cc:341] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:43.089186 24031 tablet_service.cc:1467] Processing CreateTablet for tablet a4f80e0255d34711841d8c3174e5c69d (DEFAULT_TABLE table=default.Table_8cb458b151584d4a8916e34bf519d16b [id=80cfed1c00fb4aa2a0370641f54d1d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.089412 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 324716a1b3c44be2b415695f821de8fc (DEFAULT_TABLE table=default.Table_b3352fb8a27f46c78a9919a2de2510c2 [id=c2ff07900e194320920cc91945a57910]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:43.089818 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet d16110e1b1664802a98182c73f1e41d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d16110e1b1664802a98182c73f1e41d3 already in progress: creating tablet
I20240502 08:16:43.090569 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet dcdd765895254a39b86b692b5d812c83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:43.090775 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 324716a1b3c44be2b415695f821de8fc (DEFAULT_TABLE table=default.Table_b3352fb8a27f46c78a9919a2de2510c2 [id=c2ff07900e194320920cc91945a57910]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:43.090966 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 2c2ed5eee2b0412ab3f385df556e1e7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c2ed5eee2b0412ab3f385df556e1e7a already in progress: creating tablet
I20240502 08:16:43.091181 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f80e0255d34711841d8c3174e5c69d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.091568 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_28b3620523074c23866e772b117bf0ee" } modify_external_catalogs: true
I20240502 08:16:43.091763 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_28b3620523074c23866e772b117bf0ee" } modify_external_catalogs: true
I20240502 08:16:43.093416 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4a3c195bed430e896cf7cd4c6f55e6 (DEFAULT_TABLE table=default.Table_18ad0085c7a0415c854a64868bf0735c [id=77aa3970b51046fbbd1677c051b616d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.095081 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 324716a1b3c44be2b415695f821de8fc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:43.095454 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41f2e98ef2884b05994d02114e992e80 consensus state has no leader
W20240502 08:16:43.095525 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 29144c1d42f0408f9b2987c9c66ed8dc consensus state has no leader
W20240502 08:16:43.096467 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 27093862fef54c5c85d810ebeff9819c consensus state has no leader
I20240502 08:16:43.097082 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657bcfa3d93e4809a78c7ba5692d9b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.097210 23723 leader_election.cc:302] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:43.097743 24489 raft_consensus.cc:2726] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:43.097920 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4a3c195bed430e896cf7cd4c6f55e6 (DEFAULT_TABLE table=default.Table_18ad0085c7a0415c854a64868bf0735c [id=77aa3970b51046fbbd1677c051b616d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.098970 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 324716a1b3c44be2b415695f821de8fc (DEFAULT_TABLE table=default.Table_b3352fb8a27f46c78a9919a2de2510c2 [id=c2ff07900e194320920cc91945a57910]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.099349 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 324716a1b3c44be2b415695f821de8fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.099280 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 93883f7676c148d6a9cf22690b737e80 (DEFAULT_TABLE table=default.Table_5fb99cde55ed4b33a109255d52e75384 [id=ffe60ec845b84b53bc4a63d604362018]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.099459 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet 0be4f8d5cc09483d8d1aa1e37679b544 with delete_type TABLET_DATA_DELETED (replaced by tablet e686db13bb7541c69ae88f0fb7e27061) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:43.099606 23888 tablet_service.cc:1467] Processing CreateTablet for tablet a4f80e0255d34711841d8c3174e5c69d (DEFAULT_TABLE table=default.Table_8cb458b151584d4a8916e34bf519d16b [id=80cfed1c00fb4aa2a0370641f54d1d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.097750 24175 raft_consensus.cc:348] T fd57b0357ca749c389bc61add20cd3e7 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.100749 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4a3c195bed430e896cf7cd4c6f55e6 (DEFAULT_TABLE table=default.Table_18ad0085c7a0415c854a64868bf0735c [id=77aa3970b51046fbbd1677c051b616d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.100500 24030 tablet_service.cc:1467] Processing CreateTablet for tablet d2f7ccb4385d4c3cad54758d765e8452 (DEFAULT_TABLE table=default.Table_fa986dbf19ba412dbe12164c322d7558 [id=790ab064f86b4f65a609846ba2192ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.101431 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2f7ccb4385d4c3cad54758d765e8452. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.102442 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 93883f7676c148d6a9cf22690b737e80 (DEFAULT_TABLE table=default.Table_5fb99cde55ed4b33a109255d52e75384 [id=ffe60ec845b84b53bc4a63d604362018]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.104315 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 93883f7676c148d6a9cf22690b737e80 (DEFAULT_TABLE table=default.Table_5fb99cde55ed4b33a109255d52e75384 [id=ffe60ec845b84b53bc4a63d604362018]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.105046 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6fd5def10796410c9fe46e557794e914" } modify_external_catalogs: true
I20240502 08:16:43.105266 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6fd5def10796410c9fe46e557794e914" } modify_external_catalogs: true
I20240502 08:16:43.107276 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 0e559ce43d2c440fa50715f25cc2c7e9 (DEFAULT_TABLE table=default.Table_9c8c85b0e8084a3698ebc3aae915c41c [id=e753a2d316fa48e5a0469e6e56559afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.108108 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_60a54110edb74227a1ae3a379d85e68d" } new_table_name: "default.Table_5bff1e299251446f91a962b909f49a42" modify_external_catalogs: true
I20240502 08:16:43.108837 23763 tablet_service.cc:1467] Processing CreateTablet for tablet a4f80e0255d34711841d8c3174e5c69d (DEFAULT_TABLE table=default.Table_8cb458b151584d4a8916e34bf519d16b [id=80cfed1c00fb4aa2a0370641f54d1d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.109184 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4a3c195bed430e896cf7cd4c6f55e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.109930 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93883f7676c148d6a9cf22690b737e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.109926 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e007d9b14fef433dad313c1bab5f3faa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.110389 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e007d9b14fef433dad313c1bab5f3faa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.110868 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 0e559ce43d2c440fa50715f25cc2c7e9 (DEFAULT_TABLE table=default.Table_9c8c85b0e8084a3698ebc3aae915c41c [id=e753a2d316fa48e5a0469e6e56559afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.111855 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e559ce43d2c440fa50715f25cc2c7e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.112761 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d3ec376d217c4b6781b6895c116243ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:43.113047 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d4166fd873f34357b7fe8e6078195288"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.113232 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3ec376d217c4b6781b6895c116243ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:43.113512 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4166fd873f34357b7fe8e6078195288 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.114809 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f80e0255d34711841d8c3174e5c69d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.114970 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2f7ccb4385d4c3cad54758d765e8452. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.115208 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b84ccd0af14546dd9a2dd5aab4387f27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.115661 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b84ccd0af14546dd9a2dd5aab4387f27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.116173 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e559ce43d2c440fa50715f25cc2c7e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.117254 24173 tablet_bootstrap.cc:654] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:43.120887 24028 tablet_service.cc:1467] Processing CreateTablet for tablet a60eae19da824562961dba8d52920a97 (DEFAULT_TABLE table=default.Table_d26ea2d33fbc414e84ffcc7193fa3a52 [id=7723a1a297a04f828612e7e3cc4f4bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.121759 24179 tablet_replica.cc:330] T 23ce83c8001d4b2988b88c0ed9bf948c P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:43.122002 24179 raft_consensus.cc:2229] T 23ce83c8001d4b2988b88c0ed9bf948c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:43.122376 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93883f7676c148d6a9cf22690b737e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.122915 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60eae19da824562961dba8d52920a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.127632 24175 raft_consensus.cc:374] T fd57b0357ca749c389bc61add20cd3e7 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:43.133669 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 29144c1d42f0408f9b2987c9c66ed8dc (DEFAULT_TABLE table=default.Table_92759ae885e24e0395e7a8b3f6f59181 [id=bfd52b4171794d88b1c87c681b4cff3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.133724 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 7dadd4f432554022bc2fdc3c775ac9d1 (DEFAULT_TABLE table=default.Table_5d88e7a8816a45ccb7a49f559050dad2 [id=76ef21ef044a4e8f91e7ef80e0316457]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.134121 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29144c1d42f0408f9b2987c9c66ed8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.136029 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 324716a1b3c44be2b415695f821de8fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.136147 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 7dadd4f432554022bc2fdc3c775ac9d1 (DEFAULT_TABLE table=default.Table_5d88e7a8816a45ccb7a49f559050dad2 [id=76ef21ef044a4e8f91e7ef80e0316457]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.137714 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 89f57e3a39d54b7ab372d4b145b7a696
I20240502 08:16:43.139163 24179 raft_consensus.cc:2258] T 23ce83c8001d4b2988b88c0ed9bf948c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:43.139406 23900 tablet_service.cc:1467] Processing CreateTablet for tablet d2886233097a4d19b45a4f139d82e68a (DEFAULT_TABLE table=default.Table_1b8fb2a811c54b659fb14d3bb815bc75 [id=6f910f1dce8b478fb167e12ae0bf4221]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.139735 24179 ts_tablet_manager.cc:1875] T 23ce83c8001d4b2988b88c0ed9bf948c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:43.140352 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet df08f0d3af4c4c7e8abce656714b1b7f
I20240502 08:16:43.143527 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 0e559ce43d2c440fa50715f25cc2c7e9 (DEFAULT_TABLE table=default.Table_9c8c85b0e8084a3698ebc3aae915c41c [id=e753a2d316fa48e5a0469e6e56559afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.144943 23895 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f57e3a39d54b7ab372d4b145b7a696 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:43.145972 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dadd4f432554022bc2fdc3c775ac9d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.145493 23901 tablet_service.cc:1467] Processing CreateTablet for tablet a60eae19da824562961dba8d52920a97 (DEFAULT_TABLE table=default.Table_d26ea2d33fbc414e84ffcc7193fa3a52 [id=7723a1a297a04f828612e7e3cc4f4bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.147142 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5bff0157a5e74fa8b531c2a86679a852"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.147807 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bff0157a5e74fa8b531c2a86679a852 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.149879 23897 tablet_service.cc:1467] Processing CreateTablet for tablet 29144c1d42f0408f9b2987c9c66ed8dc (DEFAULT_TABLE table=default.Table_92759ae885e24e0395e7a8b3f6f59181 [id=bfd52b4171794d88b1c87c681b4cff3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.150458 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9b5b8dc599374049ac7d7fa21ac83a27" } new_table_name: "default.Table_7c20ea5847184b48b0bd45fd5751b992" modify_external_catalogs: true
I20240502 08:16:43.151705 24175 raft_consensus.cc:729] T fd57b0357ca749c389bc61add20cd3e7 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:43.152508 23886 tablet_service.cc:1514] Processing DeleteTablet for tablet df08f0d3af4c4c7e8abce656714b1b7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:43.152709 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet b6417aafbdcf416ea7bff56bce5e6ddb with delete_type TABLET_DATA_DELETED (replaced by tablet 96b8f21c64e343d88d84e9acd0bc69b4) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:43.153301 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2886233097a4d19b45a4f139d82e68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.153556 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4a3c195bed430e896cf7cd4c6f55e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.153831 23892 tablet_service.cc:1467] Processing CreateTablet for tablet a8d8a96668e14c088de6c82d549d35ec (DEFAULT_TABLE table=default.Table_6b486df3232a4864b82575bd742573c2 [id=3b556f448f5e46cba84b95d60f7b0428]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.155669 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6 (DEFAULT_TABLE table=default.Table_f8db538a85994d0dbee1b1cbbabf5e16 [id=57a76411ce5b4a9c8b973ce5d2da53ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.163800 23759 tablet_service.cc:1467] Processing CreateTablet for tablet a8d8a96668e14c088de6c82d549d35ec (DEFAULT_TABLE table=default.Table_6b486df3232a4864b82575bd742573c2 [id=3b556f448f5e46cba84b95d60f7b0428]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.165944 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e559ce43d2c440fa50715f25cc2c7e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.166201 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 29f82ad5860843f1adfbb40cdb21a4a3
I20240502 08:16:43.166325 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dadd4f432554022bc2fdc3c775ac9d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.166599 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8d8a96668e14c088de6c82d549d35ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.166705 23752 tablet_service.cc:1467] Processing CreateTablet for tablet d2886233097a4d19b45a4f139d82e68a (DEFAULT_TABLE table=default.Table_1b8fb2a811c54b659fb14d3bb815bc75 [id=6f910f1dce8b478fb167e12ae0bf4221]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:43.166954 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92adebc0c69e4a52b975253656132495 consensus state has no leader
I20240502 08:16:43.167088 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2886233097a4d19b45a4f139d82e68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.167827 23895 tablet_service.cc:1467] Processing CreateTablet for tablet f8f43935536048a3a4a4da36d0d86ba5 (DEFAULT_TABLE table=default.Table_a52d298f258f4463a634e594b1254755 [id=425cbbc0eb39449faf67bf23887bc7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.168375 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet df08f0d3af4c4c7e8abce656714b1b7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:43.169904 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f57e3a39d54b7ab372d4b145b7a696 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:43.170097 23768 tablet_service.cc:1467] Processing CreateTablet for tablet f8f43935536048a3a4a4da36d0d86ba5 (DEFAULT_TABLE table=default.Table_a52d298f258f4463a634e594b1254755 [id=425cbbc0eb39449faf67bf23887bc7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.169543 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_14f2349fc9bb46abbdc3fef6e221184f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:43.170419 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f43935536048a3a4a4da36d0d86ba5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:43.170584 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14f2349fc9bb46abbdc3fef6e221184f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.176880 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6 (DEFAULT_TABLE table=default.Table_f8db538a85994d0dbee1b1cbbabf5e16 [id=57a76411ce5b4a9c8b973ce5d2da53ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.178236 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c1137138c8804ebfab7bdcde07e0edc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:43.178691 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:43.178814 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1137138c8804ebfab7bdcde07e0edc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.176713 24175 consensus_queue.cc:260] T fd57b0357ca749c389bc61add20cd3e7 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.182744 23757 tablet_service.cc:1467] Processing CreateTablet for tablet a60eae19da824562961dba8d52920a97 (DEFAULT_TABLE table=default.Table_d26ea2d33fbc414e84ffcc7193fa3a52 [id=7723a1a297a04f828612e7e3cc4f4bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.184322 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f43935536048a3a4a4da36d0d86ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.187512 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e52b07c276964899970877ed4cbbcdee" } modify_external_catalogs: true
I20240502 08:16:43.188635 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_bbfae571fa824e51bf838a690198e9bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.189270 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbfae571fa824e51bf838a690198e9bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.193886 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e52b07c276964899970877ed4cbbcdee" } modify_external_catalogs: true
W20240502 08:16:43.199334 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 045b0882a40f4eaba94935d9bc90da0c consensus state has no leader
I20240502 08:16:43.202440 23886 tablet_service.cc:1514] Processing DeleteTablet for tablet 29f82ad5860843f1adfbb40cdb21a4a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:43.203301 24695 raft_consensus.cc:482] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:43.203483 24695 raft_consensus.cc:504] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:43.206115 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29144c1d42f0408f9b2987c9c66ed8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.210320 23609 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet b9e31865c46d465fbd33134a0ce25d33
I20240502 08:16:43.215770 24175 ts_tablet_manager.cc:1398] T fd57b0357ca749c389bc61add20cd3e7 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.130s user 0.002s sys 0.000s
I20240502 08:16:43.216050 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 7dadd4f432554022bc2fdc3c775ac9d1 (DEFAULT_TABLE table=default.Table_5d88e7a8816a45ccb7a49f559050dad2 [id=76ef21ef044a4e8f91e7ef80e0316457]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.221349 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4a3c195bed430e896cf7cd4c6f55e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.222860 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dadd4f432554022bc2fdc3c775ac9d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.226178 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_92a3c7cbfc16461ba5b08f262bc016d3" } modify_external_catalogs: true
I20240502 08:16:43.226418 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_92a3c7cbfc16461ba5b08f262bc016d3" } modify_external_catalogs: true
I20240502 08:16:43.226568 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 29144c1d42f0408f9b2987c9c66ed8dc (DEFAULT_TABLE table=default.Table_92759ae885e24e0395e7a8b3f6f59181 [id=bfd52b4171794d88b1c87c681b4cff3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.227895 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.228143 24695 leader_election.cc:288] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:43.228312 23861 leader_election.cc:341] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: dcdd765895254a39b86b692b5d812c83
I20240502 08:16:43.228323 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f57e3a39d54b7ab372d4b145b7a696 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.229364 24039 tablet_service.cc:1467] Processing CreateTablet for tablet d2886233097a4d19b45a4f139d82e68a (DEFAULT_TABLE table=default.Table_1b8fb2a811c54b659fb14d3bb815bc75 [id=6f910f1dce8b478fb167e12ae0bf4221]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.231141 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29144c1d42f0408f9b2987c9c66ed8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.233316 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet df08f0d3af4c4c7e8abce656714b1b7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.233778 24028 tablet_service.cc:1467] Processing CreateTablet for tablet f8f43935536048a3a4a4da36d0d86ba5 (DEFAULT_TABLE table=default.Table_a52d298f258f4463a634e594b1254755 [id=425cbbc0eb39449faf67bf23887bc7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.242437 24174 tablet_bootstrap.cc:654] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:43.249056 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8d8a96668e14c088de6c82d549d35ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.249881 24175 tablet_bootstrap.cc:492] T 49b816ea1d474f2db94119441146ba78 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:43.252097 24695 raft_consensus.cc:482] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:43.258787 24695 raft_consensus.cc:504] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.259596 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2886233097a4d19b45a4f139d82e68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.260001 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60eae19da824562961dba8d52920a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.261307 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet 29f82ad5860843f1adfbb40cdb21a4a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:43.263062 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60eae19da824562961dba8d52920a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.265202 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 63e645bb6e2948e091e08433944a52f4
I20240502 08:16:43.265630 24180 ts_tablet_manager.cc:1888] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:16:43.269107 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6 (DEFAULT_TABLE table=default.Table_f8db538a85994d0dbee1b1cbbabf5e16 [id=57a76411ce5b4a9c8b973ce5d2da53ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.273680 24040 tablet_service.cc:1467] Processing CreateTablet for tablet a8d8a96668e14c088de6c82d549d35ec (DEFAULT_TABLE table=default.Table_6b486df3232a4864b82575bd742573c2 [id=3b556f448f5e46cba84b95d60f7b0428]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.274483 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f43935536048a3a4a4da36d0d86ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.274580 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 29f82ad5860843f1adfbb40cdb21a4a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.275033 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e17483d2fa3440d8acafbd48cb0f8c51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.275648 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e17483d2fa3440d8acafbd48cb0f8c51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.279497 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.281832 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9de33c88984befb056647b4c31de6f (DEFAULT_TABLE table=default.Table_c41a9c564219469d9d8b285cd4c64711 [id=ceb44c61c2b349878d7dd0bdade5df9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.282253 24033 tablet_service.cc:1467] Processing CreateTablet for tablet c05f73181ae344d8ba48fb3d3e47f983 (DEFAULT_TABLE table=default.Table_e007d9b14fef433dad313c1bab5f3faa [id=8f5c842211a54d5abfe3b3b1e0ab0373]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.283442 24180 log.cc:1196] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/768167b977c14b53a7537879e349d354
I20240502 08:16:43.283524 24596 consensus_queue.cc:1035] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 08:16:43.284238 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 48d96e42813f4169969dd5931e169923 (DEFAULT_TABLE table=default.Table_1d04c3885cac4598a01f1669c65be7c5 [id=e8ec0a5718934bb4bda87b18ed64af65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.282558 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 5185bb078dd64975b12bcbd955089f77 (DEFAULT_TABLE table=default.Table_d3ec376d217c4b6781b6895c116243ca [id=760d3fb9780b40a893a55f7b2dcdd450]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.284601 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 5185bb078dd64975b12bcbd955089f77 (DEFAULT_TABLE table=default.Table_d3ec376d217c4b6781b6895c116243ca [id=760d3fb9780b40a893a55f7b2dcdd450]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.284763 24180 ts_tablet_manager.cc:1909] T 768167b977c14b53a7537879e349d354 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:43.284991 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5185bb078dd64975b12bcbd955089f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.285207 24022 tablet_service.cc:1467] Processing CreateTablet for tablet 6d7cc1360a7d4379867e75bb7084c4fe (DEFAULT_TABLE table=default.Table_bf29da2c486e479c8e6706cddeea1731 [id=7d212635df944265af716f2f8a62c7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.285455 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e645bb6e2948e091e08433944a52f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.286448 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e645bb6e2948e091e08433944a52f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:43.290241 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8d8a96668e14c088de6c82d549d35ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.290241 24173 tablet_bootstrap.cc:492] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:43.295996 23758 tablet_service.cc:1467] Processing CreateTablet for tablet c05f73181ae344d8ba48fb3d3e47f983 (DEFAULT_TABLE table=default.Table_e007d9b14fef433dad313c1bab5f3faa [id=8f5c842211a54d5abfe3b3b1e0ab0373]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.296152 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c05f73181ae344d8ba48fb3d3e47f983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.296406 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c05f73181ae344d8ba48fb3d3e47f983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.297235 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea07009009542b59d42b9e443990c72 (DEFAULT_TABLE table=default.Table_fb665eaa632a40188077e3127c5a3e9c [id=6a7180e342bc4d39844a3fa6a234ccb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.297530 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9de33c88984befb056647b4c31de6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.299011 24029 tablet_service.cc:1467] Processing CreateTablet for tablet d47b9c1aa6da4ca4975d863d44946e32 (DEFAULT_TABLE table=default.Table_8bae47845bfb4c78804a850866569860 [id=39eb56b7743a40a79c57cc5fc331387a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.299839 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0a4ff982972e4fdaae1583619c9d6b04" } new_table_name: "default.Table_663684b67f204c6e82f363ea57b73fab" modify_external_catalogs: true
W20240502 08:16:43.248487 23858 leader_election.cc:341] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: dcdd765895254a39b86b692b5d812c83
I20240502 08:16:43.300508 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9de33c88984befb056647b4c31de6f (DEFAULT_TABLE table=default.Table_c41a9c564219469d9d8b285cd4c64711 [id=ceb44c61c2b349878d7dd0bdade5df9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.300071 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a320ac94a297479885cba64075ce2ae2" } new_table_name: "default.Table_8c96a3fc08e94e36af627e4c8dddc7ae" modify_external_catalogs: true
I20240502 08:16:43.303884 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f80e0255d34711841d8c3174e5c69d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:43.304265 23861 leader_election.cc:341] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 21832f0151ec4cf48f51f9bd2cb2b5ec
I20240502 08:16:43.305617 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 5f8a585338be4d32acdd21f7bafa8d65 (DEFAULT_TABLE table=default.Table_347c7bcdb5674f45b6f1842416643c3b [id=df6e6ccbf14344fdb1830f26fa86de73]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.305852 24173 ts_tablet_manager.cc:1367] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.223s user 0.011s sys 0.018s
I20240502 08:16:43.300463 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8a19937d3a8b4f508046d80ca85d1d53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.307368 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a19937d3a8b4f508046d80ca85d1d53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.308213 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 065723d367b34eb0af82e992b38f3ef7 (DEFAULT_TABLE table=default.Table_6c98fa55fddc49b285544085ec459fd2 [id=2085986f3bb243ddab6ea1f49d35638b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.309094 24021 tablet_service.cc:1467] Processing CreateTablet for tablet 5f8a585338be4d32acdd21f7bafa8d65 (DEFAULT_TABLE table=default.Table_347c7bcdb5674f45b6f1842416643c3b [id=df6e6ccbf14344fdb1830f26fa86de73]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.309784 23761 tablet_service.cc:1467] Processing CreateTablet for tablet d47b9c1aa6da4ca4975d863d44946e32 (DEFAULT_TABLE table=default.Table_8bae47845bfb4c78804a850866569860 [id=39eb56b7743a40a79c57cc5fc331387a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.310173 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9de33c88984befb056647b4c31de6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.310204 23858 leader_election.cc:302] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:43.310436 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d47b9c1aa6da4ca4975d863d44946e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.310565 23765 tablet_service.cc:1467] Processing CreateTablet for tablet d658ae1f967a4e0aaedf80da510bae21 (DEFAULT_TABLE table=default.Table_06aacf94362b40b9afb9c267bdcef6a3 [id=18afff00f6b6400c8410bb880bce647c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.310742 24026 tablet_service.cc:1467] Processing CreateTablet for tablet a2b7393897c54a1fa790ef47daa33525 (DEFAULT_TABLE table=default.Table_fc4d6fa243f34d899165684a654feb3e [id=2a5300f6fdc24f14baec1732261e000d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.310906 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d658ae1f967a4e0aaedf80da510bae21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.311443 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f8a585338be4d32acdd21f7bafa8d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.311559 24173 raft_consensus.cc:348] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:43.312182 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea07009009542b59d42b9e443990c72 (DEFAULT_TABLE table=default.Table_fb665eaa632a40188077e3127c5a3e9c [id=6a7180e342bc4d39844a3fa6a234ccb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.313134 24027 tablet_service.cc:1467] Processing CreateTablet for tablet d2bb4dba23ac4b4eb53b3f2d75d0442e (DEFAULT_TABLE table=default.Table_b84ccd0af14546dd9a2dd5aab4387f27 [id=0e7f0eb6033f4505b6b6052e6119e690]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.313517 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 6d7cc1360a7d4379867e75bb7084c4fe (DEFAULT_TABLE table=default.Table_bf29da2c486e479c8e6706cddeea1731 [id=7d212635df944265af716f2f8a62c7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.313830 24173 raft_consensus.cc:374] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:43.314545 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 4917c1f0decf43c2aca74c67d8b45341 (DEFAULT_TABLE table=default.Table_922974ba60ab43e5af71756d9209fd0d [id=51cd52edc7ae46738d9ffa2bd8083fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.301451 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_29ca35f287d14925b43e3d6898fd1b12" } new_table_name: "default.Table_e7dcb7ee94304b10b88d59b0872ae586" modify_external_catalogs: true
I20240502 08:16:43.315248 24034 tablet_service.cc:1467] Processing CreateTablet for tablet c48cea94856f43b6861c3ba360414888 (DEFAULT_TABLE table=default.Table_a47e8e08a0ef4797aaf83cfa8adf0019 [id=630b2f4ab433442ab82b9451c8f54f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.320201 24173 raft_consensus.cc:729] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:43.320534 24173 consensus_queue.cc:260] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:43.321177 24173 ts_tablet_manager.cc:1398] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240502 08:16:43.321936 24173 tablet_bootstrap.cc:492] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:43.322624 24695 leader_election.cc:288] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:43.324760 24181 ts_tablet_manager.cc:1888] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:16:43.324939 24181 log.cc:1196] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/b6c52f1cae3a4ad38a9f95bfaf055e76
I20240502 08:16:43.325404 24181 ts_tablet_manager.cc:1909] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:43.325711 24695 raft_consensus.cc:2726] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:43.325830 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b7393897c54a1fa790ef47daa33525. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.326424 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c48cea94856f43b6861c3ba360414888. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.326889 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93883f7676c148d6a9cf22690b737e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.326978 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065723d367b34eb0af82e992b38f3ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.327479 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bb4dba23ac4b4eb53b3f2d75d0442e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.328238 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d7cc1360a7d4379867e75bb7084c4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.328747 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48d96e42813f4169969dd5931e169923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.329258 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5185bb078dd64975b12bcbd955089f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.329792 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4917c1f0decf43c2aca74c67d8b45341. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.330304 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f8a585338be4d32acdd21f7bafa8d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.330901 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea07009009542b59d42b9e443990c72. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:43.331666 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1b175e1b55bf4102804666b6fffccead consensus state has no leader
I20240502 08:16:43.344594 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a0ebd3d8bd13425ea80ffd0c428e20da" } modify_external_catalogs: true
I20240502 08:16:43.344739 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_22b226147b894b50a865f33b9f41491c" } modify_external_catalogs: true
I20240502 08:16:43.344889 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a0ebd3d8bd13425ea80ffd0c428e20da" } modify_external_catalogs: true
I20240502 08:16:43.344938 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_22b226147b894b50a865f33b9f41491c" } modify_external_catalogs: true
W20240502 08:16:43.345891 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36f2d7b6f4e442ae95b2d1002d571b6a consensus state has no leader
I20240502 08:16:43.346451 24040 tablet_service.cc:1467] Processing CreateTablet for tablet ef03faf0a13348d7acad954256b5a1d2 (DEFAULT_TABLE table=default.Table_dd2bda6955ae4ccab8e639637c37df7f [id=36a29eba8651472d8478388f56a67a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.347002 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea07009009542b59d42b9e443990c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.355211 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d7cc1360a7d4379867e75bb7084c4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.363469 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 479) took 1229 ms. Trace:
I20240502 08:16:43.368206 24596 raft_consensus.cc:482] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:43.375447 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8a19937d3a8b4f508046d80ca85d1d53" } modify_external_catalogs: true
I20240502 08:16:43.375706 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7db6534bd8144b208a6eaa13f381cdfd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:43.375810 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8a19937d3a8b4f508046d80ca85d1d53" } modify_external_catalogs: true
W20240502 08:16:43.376330 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7db6534bd8144b208a6eaa13f381cdfd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.376713 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e82a1ff6cbbd4d07885fe037b71bf199"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.377193 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e82a1ff6cbbd4d07885fe037b71bf199 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.378026 24695 raft_consensus.cc:482] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:43.378577 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_493b1b22b2e44d92b2eb1f9775a171bd" } new_table_name: "default.Table_5b5eb6d7519d43cbb31e928e05c9569f" modify_external_catalogs: true
I20240502 08:16:43.378619 24596 raft_consensus.cc:504] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.379626 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a20c7e6c20e6416cb1934d9b830d52a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:43.379698 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_77ee071a5f1c44d987a3e71092fe4646"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.380107 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a20c7e6c20e6416cb1934d9b830d52a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:43.380167 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77ee071a5f1c44d987a3e71092fe4646 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.379571 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 48d96e42813f4169969dd5931e169923 (DEFAULT_TABLE table=default.Table_1d04c3885cac4598a01f1669c65be7c5 [id=e8ec0a5718934bb4bda87b18ed64af65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.384570 24695 raft_consensus.cc:504] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:43.387365 23858 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 148157us
I20240502 08:16:43.389438 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 643ce929e74c499da835fbe9b63b8880
I20240502 08:16:43.393667 24596 leader_election.cc:288] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:43.394575 23861 leader_election.cc:341] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 0be4f8d5cc09483d8d1aa1e37679b544
I20240502 08:16:43.396694 24695 leader_election.cc:288] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:43.398618 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e645bb6e2948e091e08433944a52f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:43.400589 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet fd57b0357ca749c389bc61add20cd3e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:43.401022 23609 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 65b676f78f354bdaa6b8642f50f096b0
I20240502 08:16:43.401508 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet a198c33a3e314b5b81a08244862b281a
I20240502 08:16:43.403467 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ee616660e58a4aaa882ad75f8078f5e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:43.363622 24180 rpcz_store.cc:276] 0502 08:16:42.126216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.134551 (+ 8335us) service_pool.cc:225] Handling call
0502 08:16:43.000396 (+865845us) ts_tablet_manager.cc:1120] Deleting tablet 768167b977c14b53a7537879e349d354
0502 08:16:43.000404 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:43.023047 (+ 22643us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:43.244108 (+221061us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:43.355719 (+111611us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":865513}
W20240502 08:16:43.404071 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee616660e58a4aaa882ad75f8078f5e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.404450 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7ee63691fc8240ad9fd8db92d778eb3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.404968 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ee63691fc8240ad9fd8db92d778eb3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.407328 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 065723d367b34eb0af82e992b38f3ef7 (DEFAULT_TABLE table=default.Table_6c98fa55fddc49b285544085ec459fd2 [id=2085986f3bb243ddab6ea1f49d35638b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.409133 24174 tablet_bootstrap.cc:492] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:43.409307 24174 ts_tablet_manager.cc:1367] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.354s user 0.004s sys 0.015s
I20240502 08:16:43.411832 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ace31be347a84e8096ec59fe11f3f0d5" } new_table_name: "default.Table_00e3ffabca274246a81fb18ce0d772a4" modify_external_catalogs: true
I20240502 08:16:43.412760 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_13623be380874b4d8b231ac4ed815b11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:43.414239 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48d96e42813f4169969dd5931e169923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.414273 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 480) took 1240 ms. Trace:
I20240502 08:16:43.414387 24180 rpcz_store.cc:276] 0502 08:16:42.170029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.170082 (+ 53us) service_pool.cc:225] Handling call
0502 08:16:43.403947 (+1233865us) ts_tablet_manager.cc:1120] Deleting tablet 0a2ff8af162d40cc87d99ec4fe91229d
0502 08:16:43.403958 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:43.410809 (+ 6851us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1205418}
I20240502 08:16:43.414911 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 481) took 1243 ms. Trace:
I20240502 08:16:43.415026 24180 rpcz_store.cc:276] 0502 08:16:42.171301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.171355 (+ 54us) service_pool.cc:225] Handling call
0502 08:16:43.414700 (+1243345us) ts_tablet_manager.cc:1120] Deleting tablet 91a3455e5931488da3134d6b5c18aef2
0502 08:16:43.414711 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:43.414839 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1216035}
I20240502 08:16:43.415421 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 484) took 1128 ms. Trace:
I20240502 08:16:43.415516 24180 rpcz_store.cc:276] 0502 08:16:42.287102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.292033 (+ 4931us) service_pool.cc:225] Handling call
0502 08:16:43.415254 (+1123221us) ts_tablet_manager.cc:1120] Deleting tablet c5c377e037894728ac084b6500ad20de
0502 08:16:43.415263 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:43.415356 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1122853}
I20240502 08:16:43.415881 23768 tablet_service.cc:1467] Processing CreateTablet for tablet a2b7393897c54a1fa790ef47daa33525 (DEFAULT_TABLE table=default.Table_fc4d6fa243f34d899165684a654feb3e [id=2a5300f6fdc24f14baec1732261e000d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:43.415954 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13623be380874b4d8b231ac4ed815b11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.416306 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b7393897c54a1fa790ef47daa33525. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.418295 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c1137138c8804ebfab7bdcde07e0edc2" } modify_external_catalogs: true
I20240502 08:16:43.418462 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c1137138c8804ebfab7bdcde07e0edc2" } modify_external_catalogs: true
I20240502 08:16:43.420153 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b0b35fed485c4223a4710de9abfd86fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.420652 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0b35fed485c4223a4710de9abfd86fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.420809 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet b6c52f1cae3a4ad38a9f95bfaf055e76 (table default.Table_35b808b0487f463dbcfa4a773a2a20f0 [id=922fb979b5894023a68820aece0b8c5e]) successfully deleted
I20240502 08:16:43.422051 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065723d367b34eb0af82e992b38f3ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.423494 23763 tablet_service.cc:1467] Processing CreateTablet for tablet d2bb4dba23ac4b4eb53b3f2d75d0442e (DEFAULT_TABLE table=default.Table_b84ccd0af14546dd9a2dd5aab4387f27 [id=0e7f0eb6033f4505b6b6052e6119e690]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.423902 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bb4dba23ac4b4eb53b3f2d75d0442e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.424638 24175 tablet_bootstrap.cc:654] T 49b816ea1d474f2db94119441146ba78 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:43.426807 24174 raft_consensus.cc:348] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.427785 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 493) took 1006 ms. Trace:
I20240502 08:16:43.427894 24180 rpcz_store.cc:276] 0502 08:16:42.409602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.409660 (+ 58us) service_pool.cc:225] Handling call
0502 08:16:43.415720 (+1006060us) ts_tablet_manager.cc:1120] Deleting tablet 2c2ed5eee2b0412ab3f385df556e1e7a
0502 08:16:43.415730 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:43.415835 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1005820}
I20240502 08:16:43.429392 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 492) took 1021 ms. Trace:
I20240502 08:16:43.429502 24180 rpcz_store.cc:276] 0502 08:16:42.407202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.421389 (+ 14187us) service_pool.cc:225] Handling call
0502 08:16:43.428162 (+1006773us) ts_tablet_manager.cc:1120] Deleting tablet d16110e1b1664802a98182c73f1e41d3
0502 08:16:43.428172 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:43.428285 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1006541}
I20240502 08:16:43.430310 24180 tablet_replica.cc:330] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:43.430531 24180 raft_consensus.cc:2229] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:43.430967 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 5185bb078dd64975b12bcbd955089f77 (DEFAULT_TABLE table=default.Table_d3ec376d217c4b6781b6895c116243ca [id=760d3fb9780b40a893a55f7b2dcdd450]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.431066 24174 raft_consensus.cc:374] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:43.432219 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5185bb078dd64975b12bcbd955089f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.432962 23891 tablet_service.cc:1467] Processing CreateTablet for tablet c05f73181ae344d8ba48fb3d3e47f983 (DEFAULT_TABLE table=default.Table_e007d9b14fef433dad313c1bab5f3faa [id=8f5c842211a54d5abfe3b3b1e0ab0373]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.349872 24643 raft_consensus.cc:482] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:43.353180 24181 tablet_replica.cc:330] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:43.356388 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 20a0519367ad4d6fae687aaeef0a8f44 (DEFAULT_TABLE table=default.Table_ace31be347a84e8096ec59fe11f3f0d5 [id=80a6ac43017e433aa8cefdc6d3a39a44]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:43.442679 23723 leader_election.cc:341] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: fd57b0357ca749c389bc61add20cd3e7
I20240502 08:16:43.442898 24174 raft_consensus.cc:729] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:43.443131 24174 consensus_queue.cc:260] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.437539 24033 tablet_service.cc:1467] Processing CreateTablet for tablet e608dd4f99004bb795de5e05ea700e08 (DEFAULT_TABLE table=default.Table_e17483d2fa3440d8acafbd48cb0f8c51 [id=60dcd9da91d44741a4ac059f0a096f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.437979 24643 raft_consensus.cc:504] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:43.445482 24174 ts_tablet_manager.cc:1398] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20240502 08:16:43.445691 24643 leader_election.cc:288] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:43.446077 24643 raft_consensus.cc:482] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:43.446236 24643 raft_consensus.cc:504] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:43.446774 24643 leader_election.cc:288] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:43.437847 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 15007d576b064c2cb0a005cb9f7ea1cd (DEFAULT_TABLE table=default.Table_c1137138c8804ebfab7bdcde07e0edc2 [id=7de541587a054eba87a194d101067ff3]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:43.448756 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c395a29c7a1485eab39ca107dd91fca consensus state has no leader
I20240502 08:16:43.438160 24181 raft_consensus.cc:2229] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:43.448990 24181 raft_consensus.cc:2258] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:43.449663 24181 ts_tablet_manager.cc:1875] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:43.373158 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 7cbf1c0a37454beb966aa61b4f8caf72 (DEFAULT_TABLE table=default.Table_b4cedcada1f94d4296418229ee7512a0 [id=5f89843a683148db9063d8e9d7d1f074]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.401438 24022 tablet_service.cc:1467] Processing CreateTablet for tablet 5c19029fc5ed416dbf4b3517da99e587 (DEFAULT_TABLE table=default.Table_d4166fd873f34357b7fe8e6078195288 [id=990c529d137b46d0a3d34c9ef9991e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.401865 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d47b9c1aa6da4ca4975d863d44946e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.451375 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 6d7cc1360a7d4379867e75bb7084c4fe (DEFAULT_TABLE table=default.Table_bf29da2c486e479c8e6706cddeea1731 [id=7d212635df944265af716f2f8a62c7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.451759 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9de33c88984befb056647b4c31de6f (DEFAULT_TABLE table=default.Table_c41a9c564219469d9d8b285cd4c64711 [id=ceb44c61c2b349878d7dd0bdade5df9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.452791 24180 raft_consensus.cc:2258] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:43.452993 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 4917c1f0decf43c2aca74c67d8b45341 (DEFAULT_TABLE table=default.Table_922974ba60ab43e5af71756d9209fd0d [id=51cd52edc7ae46738d9ffa2bd8083fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.453171 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848a0865eb2b4828a2594a17d96869d0" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:43.405628 24030 tablet_service.cc:1467] Processing CreateTablet for tablet d658ae1f967a4e0aaedf80da510bae21 (DEFAULT_TABLE table=default.Table_06aacf94362b40b9afb9c267bdcef6a3 [id=18afff00f6b6400c8410bb880bce647c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.453960 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d658ae1f967a4e0aaedf80da510bae21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.453949 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171406810543435e832a47ad5b3c80a4" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:43.455803 23993 leader_election.cc:341] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:43.456038 23993 leader_election.cc:341] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:43.423357 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 0eb7f36f7c9b47ebb93f1d846fd7352c (DEFAULT_TABLE table=default.Table_83695afdff8e4b29a56002056ac46b93 [id=3a191d0a570c4172b5b672ad8418547e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.460237 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb7f36f7c9b47ebb93f1d846fd7352c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.460286 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 48d96e42813f4169969dd5931e169923 (DEFAULT_TABLE table=default.Table_1d04c3885cac4598a01f1669c65be7c5 [id=e8ec0a5718934bb4bda87b18ed64af65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.427279 24023 tablet_service.cc:1467] Processing CreateTablet for tablet df06e0edb57f45e5ac71b568c14769cd (DEFAULT_TABLE table=default.Table_54b9e5042d19438ab2690d06917ec10c [id=27f4152aaee24a3bad52e0390b8fa649]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.430541 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 91c7167e29cb44b68f27f972652c6739 (DEFAULT_TABLE table=default.Table_c002eb9168244875bfe05cafb44aae75 [id=0435c821335b45d0bbd90ae6747690ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.431499 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 3e36be367f9b4c568ce0f3278b6297d9 (DEFAULT_TABLE table=default.Table_dcff63ecc4f740fd999047021472be23 [id=f596bcb8d6f04ac680be38e763a33750]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.432354 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9676602d1c4b37b3f4126a29f5456f (DEFAULT_TABLE table=default.Table_78883cd551c74a68b0e2a4dfe40e6955 [id=a41cc90cc9e14b70ac0ef9ed81071a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.435726 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 40fab016ed4f489aa3ccdeafcfbe1477 (DEFAULT_TABLE table=default.Table_20e4d00ccdf240fd922450cb02438505 [id=32b51b6a55f84fc28af0fbae771e46c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.436492 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 2b9f0b0afc3c47759e0e94ca44cf9711 (DEFAULT_TABLE table=default.Table_136c710cf7d94381a2afea174adb4e56 [id=88a93262111c4112a6bfcf85a4263c7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.440110 24021 tablet_service.cc:1467] Processing CreateTablet for tablet bb57cb00e884420bacc48f16739eb29c (DEFAULT_TABLE table=default.Table_bbfae571fa824e51bf838a690198e9bb [id=e5eac9b0fb414b2fa4f2c5dea6daffcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.464699 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c05f73181ae344d8ba48fb3d3e47f983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.413504 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 63a595654cff4565bcf21a7c8fabc2c4 (DEFAULT_TABLE table=default.Table_696cfb6ac5024b57aa3d8113d3be2ead [id=4bba0f71edc3419aa4f4296ed882c496]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:43.465435 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 20a0519367ad4d6fae687aaeef0a8f44 consensus state has no leader
I20240502 08:16:43.466244 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9de33c88984befb056647b4c31de6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.468068 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef03faf0a13348d7acad954256b5a1d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.468111 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d7cc1360a7d4379867e75bb7084c4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.468883 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a9676602d1c4b37b3f4126a29f5456f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.469723 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15007d576b064c2cb0a005cb9f7ea1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.470608 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e36be367f9b4c568ce0f3278b6297d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.470674 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ef79f64b4af44b88a006a32b437c3207" } modify_external_catalogs: true
I20240502 08:16:43.470850 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ef79f64b4af44b88a006a32b437c3207" } modify_external_catalogs: true
I20240502 08:16:43.471510 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a0519367ad4d6fae687aaeef0a8f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.472146 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c19029fc5ed416dbf4b3517da99e587. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.473047 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cbf1c0a37454beb966aa61b4f8caf72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.474014 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e608dd4f99004bb795de5e05ea700e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.474772 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df06e0edb57f45e5ac71b568c14769cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.475359 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48d96e42813f4169969dd5931e169923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.475410 24174 tablet_bootstrap.cc:492] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:43.475634 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40fab016ed4f489aa3ccdeafcfbe1477. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.443657 24028 tablet_service.cc:1467] Processing CreateTablet for tablet e726e0ef49b247bebe07a7e7066c1af6 (DEFAULT_TABLE table=default.Table_14f2349fc9bb46abbdc3fef6e221184f [id=e87f9d0322064232a6d8638903852842]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.478811 24180 ts_tablet_manager.cc:1875] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:43.480078 23759 tablet_service.cc:1467] Processing CreateTablet for tablet c48cea94856f43b6861c3ba360414888 (DEFAULT_TABLE table=default.Table_a47e8e08a0ef4797aaf83cfa8adf0019 [id=630b2f4ab433442ab82b9451c8f54f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.481628 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4917c1f0decf43c2aca74c67d8b45341. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.482722 23752 tablet_service.cc:1467] Processing CreateTablet for tablet 7cbf1c0a37454beb966aa61b4f8caf72 (DEFAULT_TABLE table=default.Table_b4cedcada1f94d4296418229ee7512a0 [id=5f89843a683148db9063d8e9d7d1f074]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.484526 23762 tablet_service.cc:1467] Processing CreateTablet for tablet ef03faf0a13348d7acad954256b5a1d2 (DEFAULT_TABLE table=default.Table_dd2bda6955ae4ccab8e639637c37df7f [id=36a29eba8651472d8478388f56a67a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.487524 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 768167b977c14b53a7537879e349d354 (table default.Table_d10cdf44509d4b4aabd9fb5c4a035f4b [id=ab1669d2a12f4187809987ed41fa0379]) successfully deleted
W20240502 08:16:43.487931 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 0a2ff8af162d40cc87d99ec4fe91229d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a2ff8af162d40cc87d99ec4fe91229d already in progress: creating tablet
W20240502 08:16:43.488277 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 91a3455e5931488da3134d6b5c18aef2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91a3455e5931488da3134d6b5c18aef2 already in progress: creating tablet
W20240502 08:16:43.488837 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet c5c377e037894728ac084b6500ad20de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5c377e037894728ac084b6500ad20de already in progress: creating tablet
I20240502 08:16:43.489799 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_076bc414d7814432bfd813a959dc2730"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.490372 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_076bc414d7814432bfd813a959dc2730 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:43.492192 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 2c2ed5eee2b0412ab3f385df556e1e7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c2ed5eee2b0412ab3f385df556e1e7a already in progress: creating tablet
W20240502 08:16:43.492442 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet d16110e1b1664802a98182c73f1e41d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d16110e1b1664802a98182c73f1e41d3 already in progress: creating tablet
I20240502 08:16:43.497901 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 20a0519367ad4d6fae687aaeef0a8f44 (DEFAULT_TABLE table=default.Table_ace31be347a84e8096ec59fe11f3f0d5 [id=80a6ac43017e433aa8cefdc6d3a39a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.502955 24179 ts_tablet_manager.cc:1888] T 23ce83c8001d4b2988b88c0ed9bf948c P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:43.504981 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet dead13bba95f43cb809e17f6d7fba028
I20240502 08:16:43.444239 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 9b780a7870044d94bab14f0d9c978a0c (DEFAULT_TABLE table=default.Table_5bff0157a5e74fa8b531c2a86679a852 [id=4f5fc860acec422398dbe058dddabb23]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.479935 24173 tablet_bootstrap.cc:654] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:43.507371 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c48cea94856f43b6861c3ba360414888. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.481801 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b9f0b0afc3c47759e0e94ca44cf9711. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.513013 24179 log.cc:1196] T 23ce83c8001d4b2988b88c0ed9bf948c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/23ce83c8001d4b2988b88c0ed9bf948c
I20240502 08:16:43.513814 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c7167e29cb44b68f27f972652c6739. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.518321 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea07009009542b59d42b9e443990c72 (DEFAULT_TABLE table=default.Table_fb665eaa632a40188077e3127c5a3e9c [id=6a7180e342bc4d39844a3fa6a234ccb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.518527 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b780a7870044d94bab14f0d9c978a0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.519650 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 065723d367b34eb0af82e992b38f3ef7 (DEFAULT_TABLE table=default.Table_6c98fa55fddc49b285544085ec459fd2 [id=2085986f3bb243ddab6ea1f49d35638b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.520948 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb57cb00e884420bacc48f16739eb29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.521674 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a0519367ad4d6fae687aaeef0a8f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.522275 23905 tablet_service.cc:1467] Processing CreateTablet for tablet d47b9c1aa6da4ca4975d863d44946e32 (DEFAULT_TABLE table=default.Table_8bae47845bfb4c78804a850866569860 [id=39eb56b7743a40a79c57cc5fc331387a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.522835 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 20a0519367ad4d6fae687aaeef0a8f44 (DEFAULT_TABLE table=default.Table_ace31be347a84e8096ec59fe11f3f0d5 [id=80a6ac43017e433aa8cefdc6d3a39a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.523239 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 63a595654cff4565bcf21a7c8fabc2c4 (DEFAULT_TABLE table=default.Table_696cfb6ac5024b57aa3d8113d3be2ead [id=4bba0f71edc3419aa4f4296ed882c496]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.524410 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a595654cff4565bcf21a7c8fabc2c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.526443 23882 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:33614 (request call id 119) took 1024 ms. Trace:
I20240502 08:16:43.482565 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet 643ce929e74c499da835fbe9b63b8880 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.528568 24179 ts_tablet_manager.cc:1909] T 23ce83c8001d4b2988b88c0ed9bf948c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:43.530728 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea07009009542b59d42b9e443990c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.535804 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef03faf0a13348d7acad954256b5a1d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.535732 23897 tablet_service.cc:1467] Processing CreateTablet for tablet 5f8a585338be4d32acdd21f7bafa8d65 (DEFAULT_TABLE table=default.Table_347c7bcdb5674f45b6f1842416643c3b [id=df6e6ccbf14344fdb1830f26fa86de73]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.537956 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 473) took 1012 ms. Trace:
I20240502 08:16:43.538614 23882 rpcz_store.cc:276] 0502 08:16:42.490589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.494541 (+ 3952us) service_pool.cc:225] Handling call
0502 08:16:42.825819 (+331278us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x616000142054. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:43.050825 (+225006us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x6160000d1554. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:43.251832 (+201007us) spinlock_profiling.cc:243] Waited 4.26 ms on lock 0x6180001a0c8c. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb20235cd0e 00007fb20216346f 00007fb204b60988 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58
0502 08:16:43.380952 (+129120us) spinlock_profiling.cc:243] Waited 6.95 ms on lock 0x618000160c8c. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb20235cd0e 00007fb20216346f 00007fb204b60988 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58
0502 08:16:43.515186 (+134234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":824833024}
I20240502 08:16:43.540081 23898 tablet_service.cc:1467] Processing CreateTablet for tablet a2b7393897c54a1fa790ef47daa33525 (DEFAULT_TABLE table=default.Table_fc4d6fa243f34d899165684a654feb3e [id=2a5300f6fdc24f14baec1732261e000d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.541059 23889 tablet_service.cc:1467] Processing CreateTablet for tablet d2bb4dba23ac4b4eb53b3f2d75d0442e (DEFAULT_TABLE table=default.Table_b84ccd0af14546dd9a2dd5aab4387f27 [id=0e7f0eb6033f4505b6b6052e6119e690]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.541409 23895 tablet_service.cc:1467] Processing CreateTablet for tablet d658ae1f967a4e0aaedf80da510bae21 (DEFAULT_TABLE table=default.Table_06aacf94362b40b9afb9c267bdcef6a3 [id=18afff00f6b6400c8410bb880bce647c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.541982 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 4917c1f0decf43c2aca74c67d8b45341 (DEFAULT_TABLE table=default.Table_922974ba60ab43e5af71756d9209fd0d [id=51cd52edc7ae46738d9ffa2bd8083fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.542438 23886 tablet_service.cc:1467] Processing CreateTablet for tablet c48cea94856f43b6861c3ba360414888 (DEFAULT_TABLE table=default.Table_a47e8e08a0ef4797aaf83cfa8adf0019 [id=630b2f4ab433442ab82b9451c8f54f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.542979 23904 tablet_service.cc:1467] Processing CreateTablet for tablet ef03faf0a13348d7acad954256b5a1d2 (DEFAULT_TABLE table=default.Table_dd2bda6955ae4ccab8e639637c37df7f [id=36a29eba8651472d8478388f56a67a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.544515 24181 ts_tablet_manager.cc:1888] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:43.544685 24181 log.cc:1196] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/b1fac7df16fc41d7a3297055a7258270
I20240502 08:16:43.545091 24181 ts_tablet_manager.cc:1909] T b1fac7df16fc41d7a3297055a7258270 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:43.545892 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_45e91d35e2c245d59419f7105a5b9c8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.546567 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45e91d35e2c245d59419f7105a5b9c8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.547075 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065723d367b34eb0af82e992b38f3ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.547519 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e726e0ef49b247bebe07a7e7066c1af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.547545 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef03faf0a13348d7acad954256b5a1d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.549966 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c48cea94856f43b6861c3ba360414888. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.550618 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b7393897c54a1fa790ef47daa33525. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.551157 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4917c1f0decf43c2aca74c67d8b45341. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.553097 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_01e045a6ce334744b9ab0c8d6d390054" } new_table_name: "default.Table_d790560c82534057ab7061352d8c71a2" modify_external_catalogs: true
I20240502 08:16:43.554106 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet 33c8131846dc41678827b86e87be18d3 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:43.521143 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 5c19029fc5ed416dbf4b3517da99e587 (DEFAULT_TABLE table=default.Table_d4166fd873f34357b7fe8e6078195288 [id=990c529d137b46d0a3d34c9ef9991e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.556124 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bb4dba23ac4b4eb53b3f2d75d0442e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.557816 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_77ee071a5f1c44d987a3e71092fe4646" } new_table_name: "default.Table_3a47de7a9c814ca2a1656345ebbf39c7" modify_external_catalogs: true
I20240502 08:16:43.558017 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d658ae1f967a4e0aaedf80da510bae21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.559157 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a0519367ad4d6fae687aaeef0a8f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.560823 24596 raft_consensus.cc:482] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:43.560622 24179 rpcz_store.cc:276] 0502 08:16:42.517859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.517978 (+ 119us) service_pool.cc:225] Handling call
0502 08:16:43.090611 (+572633us) ts_tablet_manager.cc:1120] Deleting tablet 23ce83c8001d4b2988b88c0ed9bf948c
0502 08:16:43.090620 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:43.367067 (+276447us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:43.491388 (+124321us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:43.530223 (+ 38835us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":571303}
I20240502 08:16:43.485370 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet a198c33a3e314b5b81a08244862b281a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.562767 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet dead13bba95f43cb809e17f6d7fba028 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.562711 24596 raft_consensus.cc:504] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.563472 24596 leader_election.cc:288] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:43.563886 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cbf1c0a37454beb966aa61b4f8caf72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.565853 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f8a585338be4d32acdd21f7bafa8d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.568583 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 474) took 1044 ms. Trace:
W20240502 08:16:43.570542 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42e11d405a2e4e518a10bf383e9ae6c4 consensus state has no leader
W20240502 08:16:43.571409 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9b67c8cd7f34cbba6401da59dec526c has no consensus state
I20240502 08:16:43.575131 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d47b9c1aa6da4ca4975d863d44946e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.579602 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a595654cff4565bcf21a7c8fabc2c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.583379 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 0eb7f36f7c9b47ebb93f1d846fd7352c (DEFAULT_TABLE table=default.Table_83695afdff8e4b29a56002056ac46b93 [id=3a191d0a570c4172b5b672ad8418547e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.587031 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c19029fc5ed416dbf4b3517da99e587. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.588374 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 91c7167e29cb44b68f27f972652c6739 (DEFAULT_TABLE table=default.Table_c002eb9168244875bfe05cafb44aae75 [id=0435c821335b45d0bbd90ae6747690ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.589427 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c7167e29cb44b68f27f972652c6739. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.591454 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e628d856ede54061bb6736c1f1b8b0c1" } modify_external_catalogs: true
I20240502 08:16:43.591698 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e628d856ede54061bb6736c1f1b8b0c1" } modify_external_catalogs: true
I20240502 08:16:43.593382 24181 tablet_replica.cc:330] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:43.593629 24181 raft_consensus.cc:2229] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:43.593832 24181 raft_consensus.cc:2258] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:43.594388 24181 ts_tablet_manager.cc:1875] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:43.594607 24174 tablet_bootstrap.cc:654] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:43.588711 23760 tablet_service.cc:1467] Processing CreateTablet for tablet df06e0edb57f45e5ac71b568c14769cd (DEFAULT_TABLE table=default.Table_54b9e5042d19438ab2690d06917ec10c [id=27f4152aaee24a3bad52e0390b8fa649]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.596871 24596 raft_consensus.cc:482] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:43.597016 24596 raft_consensus.cc:504] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.597049 24179 rpcz_store.cc:276] 0502 08:16:42.518931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.518981 (+ 50us) service_pool.cc:225] Handling call
0502 08:16:43.562326 (+1043345us) ts_tablet_manager.cc:1120] Deleting tablet 91f1ab141585441db46d3027833932b0
0502 08:16:43.562336 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:43.563676 (+ 1340us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1042457}
I20240502 08:16:43.597736 24596 leader_election.cc:288] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:43.601378 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 5c19029fc5ed416dbf4b3517da99e587 (DEFAULT_TABLE table=default.Table_d4166fd873f34357b7fe8e6078195288 [id=990c529d137b46d0a3d34c9ef9991e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.602531 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_83695afdff8e4b29a56002056ac46b93" } modify_external_catalogs: true
I20240502 08:16:43.602725 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_83695afdff8e4b29a56002056ac46b93" } modify_external_catalogs: true
I20240502 08:16:43.603999 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_14f2349fc9bb46abbdc3fef6e221184f" } modify_external_catalogs: true
I20240502 08:16:43.604007 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_59c11afb388848da8ac134294bc6c485"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:43.604211 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_14f2349fc9bb46abbdc3fef6e221184f" } modify_external_catalogs: true
W20240502 08:16:43.604733 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59c11afb388848da8ac134294bc6c485 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.611440 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 3e36be367f9b4c568ce0f3278b6297d9 (DEFAULT_TABLE table=default.Table_dcff63ecc4f740fd999047021472be23 [id=f596bcb8d6f04ac680be38e763a33750]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.612869 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet b1fac7df16fc41d7a3297055a7258270 (table default.Table_bcd1014009b046a7ba72d1d757e69e26 [id=e6823667e67a470e870f4ed70fc58a85]) successfully deleted
I20240502 08:16:43.613453 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8cea3dc4daac4846a1ed61bfe1dd17a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.614063 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cea3dc4daac4846a1ed61bfe1dd17a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:43.614375 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet b8b1fa17dad143f0a7f1751e651eb81a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8b1fa17dad143f0a7f1751e651eb81a already in progress: creating tablet
W20240502 08:16:43.615016 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 already in progress: creating tablet
W20240502 08:16:43.615681 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e47459b5f5bb4ecb96f76240fefa1f5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e47459b5f5bb4ecb96f76240fefa1f5d already in progress: creating tablet
I20240502 08:16:43.619259 24179 tablet_replica.cc:330] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:43.619542 24179 raft_consensus.cc:2229] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:43.619782 24179 raft_consensus.cc:2258] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:43.620532 24179 ts_tablet_manager.cc:1875] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:43.623051 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 0eb7f36f7c9b47ebb93f1d846fd7352c
I20240502 08:16:43.625151 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 40fab016ed4f489aa3ccdeafcfbe1477 (DEFAULT_TABLE table=default.Table_20e4d00ccdf240fd922450cb02438505 [id=32b51b6a55f84fc28af0fbae771e46c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.626509 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 2b9f0b0afc3c47759e0e94ca44cf9711 (DEFAULT_TABLE table=default.Table_136c710cf7d94381a2afea174adb4e56 [id=88a93262111c4112a6bfcf85a4263c7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.627657 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 7cbf1c0a37454beb966aa61b4f8caf72 (DEFAULT_TABLE table=default.Table_b4cedcada1f94d4296418229ee7512a0 [id=5f89843a683148db9063d8e9d7d1f074]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.628661 23750 tablet_service.cc:1467] Processing CreateTablet for tablet e608dd4f99004bb795de5e05ea700e08 (DEFAULT_TABLE table=default.Table_e17483d2fa3440d8acafbd48cb0f8c51 [id=60dcd9da91d44741a4ac059f0a096f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.629036 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb7f36f7c9b47ebb93f1d846fd7352c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.629699 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df06e0edb57f45e5ac71b568c14769cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.631098 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 15007d576b064c2cb0a005cb9f7ea1cd
I20240502 08:16:43.633001 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 523603a31b5b49e0a18a5a64f1357d64
W20240502 08:16:43.635962 23595 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 4c395a29c7a1485eab39ca107dd91fca (table default.Table_5b5eb6d7519d43cbb31e928e05c9569f [id=272acbb9a6ca4f2dbcbfea6aacbe9cfc], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4c395a29c7a1485eab39ca107dd91fca consensus state has no leader
I20240502 08:16:43.637712 23885 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:33614 (request call id 117) took 1302 ms. Trace:
I20240502 08:16:43.637835 23885 rpcz_store.cc:276] 0502 08:16:42.297258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.297428 (+ 170us) service_pool.cc:225] Handling call
0502 08:16:42.622329 (+324901us) spinlock_profiling.cc:243] Waited 65 ms on lock 0x6160001c1b54. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:42.661486 (+ 39157us) spinlock_profiling.cc:243] Waited 2.94 ms on lock 0x61800013108c. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb20235d017 00007fb204b60f29 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:42.823585 (+162099us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x616000142054. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:43.054017 (+230432us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x6160000d1554. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:43.088435 (+ 34418us) spinlock_profiling.cc:243] Waited 2.67 ms on lock 0x6180001a088c. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb20235cd0e 00007fb20216346f 00007fb204b60988 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58
0502 08:16:43.599488 (+511053us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1035498112}
I20240502 08:16:43.640604 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_063b529ba90f40ffafa8016535fe31bd" } new_table_name: "default.Table_c4e75358850844d688b8d73770caec30" modify_external_catalogs: true
I20240502 08:16:43.643077 23606 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 49b816ea1d474f2db94119441146ba78 (table default.Table_20df1b504848439c869ae3739ac54185 [id=a0606a4053d04dd3814f09419a4a70c2]). Expected version 1 got 0
I20240502 08:16:43.646643 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c19029fc5ed416dbf4b3517da99e587. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.646632 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e726e0ef49b247bebe07a7e7066c1af6
I20240502 08:16:43.648550 24181 ts_tablet_manager.cc:1888] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:43.648710 24181 log.cc:1196] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/171406810543435e832a47ad5b3c80a4
I20240502 08:16:43.649129 24181 ts_tablet_manager.cc:1909] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:43.650203 23609 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet b844458ae73f413cbf2113027a5af359 (table default.Table_663684b67f204c6e82f363ea57b73fab [id=fd93259a378b4ea582a6661a59104abb]). Expected version 2 got 0
I20240502 08:16:43.650516 23609 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet e134e90efb4a48bb82b53a80bda708b3 (table default.Table_cc80d1cedafa4e71906e5e9530fdd828 [id=47682c26a85842c7b8991bf3828a9336]). Expected version 1 got 0
I20240502 08:16:43.651563 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0c640f9efa084ae7953f217ce7601a4f" } new_table_name: "default.Table_5398d425143843de8fe5c9594fcf8503" modify_external_catalogs: true
I20240502 08:16:43.653723 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b9f0b0afc3c47759e0e94ca44cf9711. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.655017 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cbf1c0a37454beb966aa61b4f8caf72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.659821 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a19310948e248d1a85479c41c61bda2" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:43.660156 24057 raft_consensus.cc:2454] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
W20240502 08:16:43.662843 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 001e3f4f72f64dc69dbd521724912f07 consensus state has no leader
I20240502 08:16:43.663208 24179 ts_tablet_manager.cc:1888] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:16:43.663359 24179 log.cc:1196] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/b6c52f1cae3a4ad38a9f95bfaf055e76
I20240502 08:16:43.663803 24179 ts_tablet_manager.cc:1909] T b6c52f1cae3a4ad38a9f95bfaf055e76 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:43.664943 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e5391748a4874f97a3546ba6c5fb22eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:43.665225 24179 tablet_replica.cc:330] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:43.665280 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_850349fb9fa64ccb884825b10bbec754"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:43.665437 24179 raft_consensus.cc:2229] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
W20240502 08:16:43.665824 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_850349fb9fa64ccb884825b10bbec754 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.665889 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_50a6c708582c422c89aef5f2ca35a88a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.666283 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50a6c708582c422c89aef5f2ca35a88a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:43.666611 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5391748a4874f97a3546ba6c5fb22eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.666973 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_897d3dc97c8241d58a1a9a5329a1f892" } new_table_name: "default.Table_1136fadacc774d899c0461e8a0579376" modify_external_catalogs: true
I20240502 08:16:43.667047 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_02cd49f90ab14e919f913d9eb37ebdc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.667640 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02cd49f90ab14e919f913d9eb37ebdc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.668596 24181 tablet_replica.cc:330] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:43.668903 24181 raft_consensus.cc:2229] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:43.669126 24181 raft_consensus.cc:2258] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:43.669881 24181 ts_tablet_manager.cc:1875] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:43.673085 23607 master_service.cc:946] ReplaceTablet: received request to replace tablet 9467974d67c64c7eb5e488ac51f1240d from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:43.674343 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848a0865eb2b4828a2594a17d96869d0" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:43.674801 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171406810543435e832a47ad5b3c80a4" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:43.675338 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e8bb862a5508434a83ad4f5c8be82002"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.675843 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8bb862a5508434a83ad4f5c8be82002 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:43.676438 23996 leader_election.cc:341] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:43.676575 23996 leader_election.cc:302] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:43.676790 23996 raft_consensus.cc:2670] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:16:43.676834 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9676602d1c4b37b3f4126a29f5456f (DEFAULT_TABLE table=default.Table_78883cd551c74a68b0e2a4dfe40e6955 [id=a41cc90cc9e14b70ac0ef9ed81071a16]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:43.677101 23996 leader_election.cc:341] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:43.677250 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a9676602d1c4b37b3f4126a29f5456f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.677230 23996 leader_election.cc:302] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:43.677414 23996 raft_consensus.cc:2670] T 171406810543435e832a47ad5b3c80a4 P 149ce744feab49b5aa5bcaa14bd6aba5: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:16:43.677467 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2b4b1372dc684d71aa26d7e8a435eaed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:43.677852 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 63a595654cff4565bcf21a7c8fabc2c4 (DEFAULT_TABLE table=default.Table_696cfb6ac5024b57aa3d8113d3be2ead [id=4bba0f71edc3419aa4f4296ed882c496]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:43.677937 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b4b1372dc684d71aa26d7e8a435eaed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.678328 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 15007d576b064c2cb0a005cb9f7ea1cd (DEFAULT_TABLE table=default.Table_c1137138c8804ebfab7bdcde07e0edc2 [id=7de541587a054eba87a194d101067ff3]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:43.679849 23996 consensus_peers.cc:485] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:43.681846 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 0eb7f36f7c9b47ebb93f1d846fd7352c (DEFAULT_TABLE table=default.Table_83695afdff8e4b29a56002056ac46b93 [id=3a191d0a570c4172b5b672ad8418547e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.682297 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb7f36f7c9b47ebb93f1d846fd7352c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.685427 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 3e36be367f9b4c568ce0f3278b6297d9 (DEFAULT_TABLE table=default.Table_dcff63ecc4f740fd999047021472be23 [id=f596bcb8d6f04ac680be38e763a33750]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.685521 23764 tablet_service.cc:1467] Processing CreateTablet for tablet bb57cb00e884420bacc48f16739eb29c (DEFAULT_TABLE table=default.Table_bbfae571fa824e51bf838a690198e9bb [id=e5eac9b0fb414b2fa4f2c5dea6daffcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.685869 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e36be367f9b4c568ce0f3278b6297d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.686126 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40fab016ed4f489aa3ccdeafcfbe1477. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.687005 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a595654cff4565bcf21a7c8fabc2c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.692363 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eb7f36f7c9b47ebb93f1d846fd7352c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.692890 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 91c7167e29cb44b68f27f972652c6739 (DEFAULT_TABLE table=default.Table_c002eb9168244875bfe05cafb44aae75 [id=0435c821335b45d0bbd90ae6747690ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.693181 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 171406810543435e832a47ad5b3c80a4 (table default.Table_a286e67e441e43a1b4738cd219dbea98 [id=b56e3c0686244ad6b49afdd589c5ed34]) successfully deleted
I20240502 08:16:43.693975 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c7167e29cb44b68f27f972652c6739. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.694329 24753 consensus_queue.cc:1035] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:43.696265 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15007d576b064c2cb0a005cb9f7ea1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.696516 24181 ts_tablet_manager.cc:1888] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:43.696650 24181 log.cc:1196] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/848a0865eb2b4828a2594a17d96869d0
I20240502 08:16:43.697013 24181 ts_tablet_manager.cc:1909] T 848a0865eb2b4828a2594a17d96869d0 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:43.697912 23903 tablet_service.cc:1467] Processing CreateTablet for tablet df06e0edb57f45e5ac71b568c14769cd (DEFAULT_TABLE table=default.Table_54b9e5042d19438ab2690d06917ec10c [id=27f4152aaee24a3bad52e0390b8fa649]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.698357 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df06e0edb57f45e5ac71b568c14769cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.697182 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9676602d1c4b37b3f4126a29f5456f (DEFAULT_TABLE table=default.Table_78883cd551c74a68b0e2a4dfe40e6955 [id=a41cc90cc9e14b70ac0ef9ed81071a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.698841 24181 tablet_replica.cc:330] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:43.699076 24181 raft_consensus.cc:2229] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:43.699318 24181 raft_consensus.cc:2258] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:43.699896 24181 ts_tablet_manager.cc:1875] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:43.703792 23723 leader_election.cc:302] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
W20240502 08:16:43.704363 23723 leader_election.cc:341] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 5c95dfda9d3b4e2ca9397b77ad71c011
I20240502 08:16:43.707307 24489 raft_consensus.cc:2781] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:43.707448 24489 raft_consensus.cc:482] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:43.708969 24489 raft_consensus.cc:3037] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:43.710762 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 1da01e2c2f054ccfb53383e70e16da84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1da01e2c2f054ccfb53383e70e16da84 already in progress: creating tablet
W20240502 08:16:43.711091 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 76e0f767c6534a608023e6e6d33ca7a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76e0f767c6534a608023e6e6d33ca7a8 already in progress: creating tablet
W20240502 08:16:43.711360 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e5f5597402374527bf0e5e50178df713 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5f5597402374527bf0e5e50178df713 already in progress: creating tablet
W20240502 08:16:43.702803 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2085ced52fff46ebaa98324cbb2d8c11 consensus state has no leader
I20240502 08:16:43.709654 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2cc3c92f41994b1ab2144237110b1d88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.712296 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cc3c92f41994b1ab2144237110b1d88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.710388 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 9467974d67c64c7eb5e488ac51f1240d
I20240502 08:16:43.714155 23607 catalog_manager.cc:6554] ReplaceTablet: tablet 9467974d67c64c7eb5e488ac51f1240d deleted and replaced by tablet 29f6c3cea4b14364b6d073a92171119d
I20240502 08:16:43.715395 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e36be367f9b4c568ce0f3278b6297d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.715616 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5bff0157a5e74fa8b531c2a86679a852" } modify_external_catalogs: true
I20240502 08:16:43.715862 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5bff0157a5e74fa8b531c2a86679a852" } modify_external_catalogs: true
I20240502 08:16:43.716704 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb57cb00e884420bacc48f16739eb29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.717077 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e608dd4f99004bb795de5e05ea700e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.720852 23892 tablet_service.cc:1467] Processing CreateTablet for tablet e726e0ef49b247bebe07a7e7066c1af6 (DEFAULT_TABLE table=default.Table_14f2349fc9bb46abbdc3fef6e221184f [id=e87f9d0322064232a6d8638903852842]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.721359 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e726e0ef49b247bebe07a7e7066c1af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.721843 23901 tablet_service.cc:1467] Processing CreateTablet for tablet bb57cb00e884420bacc48f16739eb29c (DEFAULT_TABLE table=default.Table_bbfae571fa824e51bf838a690198e9bb [id=e5eac9b0fb414b2fa4f2c5dea6daffcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.722215 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb57cb00e884420bacc48f16739eb29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.724970 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 15007d576b064c2cb0a005cb9f7ea1cd (DEFAULT_TABLE table=default.Table_c1137138c8804ebfab7bdcde07e0edc2 [id=7de541587a054eba87a194d101067ff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.726228 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 9b780a7870044d94bab14f0d9c978a0c (DEFAULT_TABLE table=default.Table_5bff0157a5e74fa8b531c2a86679a852 [id=4f5fc860acec422398dbe058dddabb23]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.726603 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15007d576b064c2cb0a005cb9f7ea1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.726840 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b780a7870044d94bab14f0d9c978a0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.731703 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet 643ce929e74c499da835fbe9b63b8880 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:43.731236 23896 tablet_service.cc:1467] Processing CreateTablet for tablet e608dd4f99004bb795de5e05ea700e08 (DEFAULT_TABLE table=default.Table_e17483d2fa3440d8acafbd48cb0f8c51 [id=60dcd9da91d44741a4ac059f0a096f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.732013 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet a198c33a3e314b5b81a08244862b281a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:43.732295 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c95dfda9d3b4e2ca9397b77ad71c011 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:43.732677 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet dead13bba95f43cb809e17f6d7fba028 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:43.739782 24489 raft_consensus.cc:504] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.740693 24489 leader_election.cc:288] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:43.743367 24181 ts_tablet_manager.cc:1888] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:16:43.743515 24181 log.cc:1196] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/dead13bba95f43cb809e17f6d7fba028
I20240502 08:16:43.743960 24181 ts_tablet_manager.cc:1909] T dead13bba95f43cb809e17f6d7fba028 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:43.752712 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e608dd4f99004bb795de5e05ea700e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.757457 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet 643ce929e74c499da835fbe9b63b8880 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:43.757743 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet a198c33a3e314b5b81a08244862b281a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:43.764092 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_53c08a7447a44323883744c033efcbfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.764751 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53c08a7447a44323883744c033efcbfb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.770329 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 9b780a7870044d94bab14f0d9c978a0c (DEFAULT_TABLE table=default.Table_5bff0157a5e74fa8b531c2a86679a852 [id=4f5fc860acec422398dbe058dddabb23]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.772660 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2468b65097c244298ba963e7aa155353" } new_table_name: "default.Table_db86cd2cf86e48ffbc6a513b5c2bd7ab" modify_external_catalogs: true
I20240502 08:16:43.774031 23768 tablet_service.cc:1467] Processing CreateTablet for tablet e726e0ef49b247bebe07a7e7066c1af6 (DEFAULT_TABLE table=default.Table_14f2349fc9bb46abbdc3fef6e221184f [id=e87f9d0322064232a6d8638903852842]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.774389 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e726e0ef49b247bebe07a7e7066c1af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.774739 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8cea3dc4daac4846a1ed61bfe1dd17a3" } new_table_name: "default.Table_e2cd64637b7843539c015901a34d3258" modify_external_catalogs: true
I20240502 08:16:43.776255 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet 15007d576b064c2cb0a005cb9f7ea1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:43.776494 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet dead13bba95f43cb809e17f6d7fba028 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:43.779784 23895 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eb7f36f7c9b47ebb93f1d846fd7352c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:43.782092 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a9676602d1c4b37b3f4126a29f5456f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.786164 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b780a7870044d94bab14f0d9c978a0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.788100 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 523603a31b5b49e0a18a5a64f1357d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:43.794580 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 9467974d67c64c7eb5e488ac51f1240d with delete_type TABLET_DATA_DELETED (replaced by tablet 29f6c3cea4b14364b6d073a92171119d) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:43.799911 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 2b9f0b0afc3c47759e0e94ca44cf9711 (DEFAULT_TABLE table=default.Table_136c710cf7d94381a2afea174adb4e56 [id=88a93262111c4112a6bfcf85a4263c7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.800453 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b9f0b0afc3c47759e0e94ca44cf9711. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.805663 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet e726e0ef49b247bebe07a7e7066c1af6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:43.811517 23921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd57b0357ca749c389bc61add20cd3e7" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:43.819789 23921 raft_consensus.cc:2454] T fd57b0357ca749c389bc61add20cd3e7 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:43.820619 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c95dfda9d3b4e2ca9397b77ad71c011" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:43.821604 23921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a19310948e248d1a85479c41c61bda2" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:43.821173 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a19310948e248d1a85479c41c61bda2" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:43.823000 24175 tablet_bootstrap.cc:492] T 49b816ea1d474f2db94119441146ba78 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:43.823637 24175 ts_tablet_manager.cc:1367] T 49b816ea1d474f2db94119441146ba78 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.607s user 0.002s sys 0.012s
I20240502 08:16:43.824622 24175 raft_consensus.cc:348] T 49b816ea1d474f2db94119441146ba78 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.825208 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b9b22cc686024294ab1843b25cb74af7" } modify_external_catalogs: true
I20240502 08:16:43.825459 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b9b22cc686024294ab1843b25cb74af7" } modify_external_catalogs: true
I20240502 08:16:43.825685 24175 raft_consensus.cc:374] T 49b816ea1d474f2db94119441146ba78 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:43.825832 24175 raft_consensus.cc:729] T 49b816ea1d474f2db94119441146ba78 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:43.826968 24175 consensus_queue.cc:260] T 49b816ea1d474f2db94119441146ba78 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.828689 24489 raft_consensus.cc:482] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:43.828884 24489 raft_consensus.cc:504] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.829319 24175 ts_tablet_manager.cc:1398] T 49b816ea1d474f2db94119441146ba78 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:16:43.829639 24489 leader_election.cc:288] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:43.830046 23603 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 49b816ea1d474f2db94119441146ba78 (table default.Table_20df1b504848439c869ae3739ac54185 [id=a0606a4053d04dd3814f09419a4a70c2]). Expected version 1 got 0
I20240502 08:16:43.831914 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 15007d576b064c2cb0a005cb9f7ea1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.832499 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 523603a31b5b49e0a18a5a64f1357d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.832952 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet e726e0ef49b247bebe07a7e7066c1af6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.833397 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.833628 24181 tablet_replica.cc:330] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:43.833772 24175 tablet_bootstrap.cc:492] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:43.833849 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 9467974d67c64c7eb5e488ac51f1240d with delete_type TABLET_DATA_DELETED (replaced by tablet 29f6c3cea4b14364b6d073a92171119d) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.833846 24181 raft_consensus.cc:2229] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:43.834071 24181 raft_consensus.cc:2258] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:43.834367 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 848a0865eb2b4828a2594a17d96869d0 (table default.Table_7bae786c8c2e44bd9e893ff90696d216 [id=7ffacb15a8a249258fb42c4312f87d69]) successfully deleted
I20240502 08:16:43.834615 23603 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 49b816ea1d474f2db94119441146ba78 (table default.Table_20df1b504848439c869ae3739ac54185 [id=a0606a4053d04dd3814f09419a4a70c2]). Expected version 1 got 0
I20240502 08:16:43.834707 24181 ts_tablet_manager.cc:1875] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:43.834816 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 40fab016ed4f489aa3ccdeafcfbe1477 (DEFAULT_TABLE table=default.Table_20e4d00ccdf240fd922450cb02438505 [id=32b51b6a55f84fc28af0fbae771e46c1]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:43.835333 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 89f57e3a39d54b7ab372d4b145b7a696 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89f57e3a39d54b7ab372d4b145b7a696 already in progress: creating tablet
W20240502 08:16:43.835729 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet df08f0d3af4c4c7e8abce656714b1b7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df08f0d3af4c4c7e8abce656714b1b7f already in progress: creating tablet
W20240502 08:16:43.836103 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 29f82ad5860843f1adfbb40cdb21a4a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29f82ad5860843f1adfbb40cdb21a4a3 already in progress: creating tablet
I20240502 08:16:43.836315 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_42d6187c77b347e58022b68ed044f3fe" } modify_external_catalogs: true
W20240502 08:16:43.836498 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 63e645bb6e2948e091e08433944a52f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63e645bb6e2948e091e08433944a52f4 already in progress: creating tablet
I20240502 08:16:43.836519 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_42d6187c77b347e58022b68ed044f3fe" } modify_external_catalogs: true
I20240502 08:16:43.836747 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_dcbb721cb3904016a2866ef34f6bf557" } modify_external_catalogs: true
I20240502 08:16:43.836922 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_dcbb721cb3904016a2866ef34f6bf557" } modify_external_catalogs: true
W20240502 08:16:43.837288 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 643ce929e74c499da835fbe9b63b8880 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 643ce929e74c499da835fbe9b63b8880 already in progress: creating tablet
W20240502 08:16:43.837699 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet a198c33a3e314b5b81a08244862b281a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a198c33a3e314b5b81a08244862b281a already in progress: creating tablet
I20240502 08:16:43.838040 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet dead13bba95f43cb809e17f6d7fba028 (table default.Table_ef79f64b4af44b88a006a32b437c3207 [id=7409c7e1a3534932a787559a679e7098]) successfully deleted
W20240502 08:16:43.838485 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 0eb7f36f7c9b47ebb93f1d846fd7352c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0eb7f36f7c9b47ebb93f1d846fd7352c already in progress: creating tablet
I20240502 08:16:43.838905 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40fab016ed4f489aa3ccdeafcfbe1477. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:43.838900 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 15007d576b064c2cb0a005cb9f7ea1cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15007d576b064c2cb0a005cb9f7ea1cd already in progress: creating tablet
I20240502 08:16:43.839095 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_18f36b7d14194553a25aad1cc668845a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.839279 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 523603a31b5b49e0a18a5a64f1357d64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 523603a31b5b49e0a18a5a64f1357d64 already in progress: creating tablet
W20240502 08:16:43.839655 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18f36b7d14194553a25aad1cc668845a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.840170 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a19310948e248d1a85479c41c61bda2" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:43.840230 24058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2d7b6f4e442ae95b2d1002d571b6a" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:43.840389 24060 raft_consensus.cc:3037] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:43.840385 23723 consensus_peers.cc:485] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6c52f1cae3a4ad38a9f95bfaf055e76. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:43.840485 24058 raft_consensus.cc:2454] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
W20240502 08:16:43.841863 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e726e0ef49b247bebe07a7e7066c1af6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e726e0ef49b247bebe07a7e7066c1af6 already in progress: creating tablet
I20240502 08:16:43.842733 24181 ts_tablet_manager.cc:1888] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:43.842864 24181 log.cc:1196] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/9c03a4353c4e46e3bcdf3a36aac04bf5
I20240502 08:16:43.843192 24181 ts_tablet_manager.cc:1909] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:43.843403 24060 raft_consensus.cc:2454] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:43.849651 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4b967ef2b2fc4e2caa4219afcae841a6" } modify_external_catalogs: true
I20240502 08:16:43.849839 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4b967ef2b2fc4e2caa4219afcae841a6" } modify_external_catalogs: true
I20240502 08:16:43.851744 23921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2d7b6f4e442ae95b2d1002d571b6a" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:43.858906 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 523603a31b5b49e0a18a5a64f1357d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:43.859544 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet 9467974d67c64c7eb5e488ac51f1240d with delete_type TABLET_DATA_DELETED (replaced by tablet 29f6c3cea4b14364b6d073a92171119d) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:43.859607 24173 tablet_bootstrap.cc:492] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:43.859747 24173 ts_tablet_manager.cc:1367] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.538s user 0.031s sys 0.028s
I20240502 08:16:43.859908 23752 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eb7f36f7c9b47ebb93f1d846fd7352c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:43.860549 24173 raft_consensus.cc:348] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:43.860770 24173 raft_consensus.cc:374] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:43.860872 24173 raft_consensus.cc:729] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:43.861110 24173 consensus_queue.cc:260] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:43.861678 24173 ts_tablet_manager.cc:1398] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:16:43.861992 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 23ce83c8001d4b2988b88c0ed9bf948c (table default.Table_190eae2a8c64440f8d19842f8b6ac254 [id=54454efbe9394170822f237dc8e6115e]) successfully deleted
I20240502 08:16:43.862571 24173 tablet_bootstrap.cc:492] T c08228fcd1304528b9446003851f6c70 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:43.863234 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 91f1ab141585441db46d3027833932b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91f1ab141585441db46d3027833932b0 already in progress: creating tablet
W20240502 08:16:43.863885 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 02e6a1ba23dd4f82bc721a60c0526871 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02e6a1ba23dd4f82bc721a60c0526871 already in progress: creating tablet
I20240502 08:16:43.864063 24173 tablet_bootstrap.cc:654] T c08228fcd1304528b9446003851f6c70 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:43.864672 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 45f02efa659e457ab57bc32ed3f66bb9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45f02efa659e457ab57bc32ed3f66bb9 already in progress: creating tablet
I20240502 08:16:43.867494 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 15007d576b064c2cb0a005cb9f7ea1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:43.873050 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 0f306a288b90413b9d8ea8b76d23363a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f306a288b90413b9d8ea8b76d23363a already in progress: creating tablet
I20240502 08:16:43.876878 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet a5a5a410b6314818b10927e35032f95d from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:43.878060 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0611d28b3eeb4bd8b06d3b02e17190b0" } new_table_name: "default.Table_de7dd656583f4c118ba6bd8bb1945caf" modify_external_catalogs: true
W20240502 08:16:43.880101 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1be2c29d3154d9189af3266191f0b3e consensus state has no leader
I20240502 08:16:43.880885 24174 tablet_bootstrap.cc:492] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:43.881083 24174 ts_tablet_manager.cc:1367] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.425s user 0.009s sys 0.000s
I20240502 08:16:43.881816 24174 raft_consensus.cc:348] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.882062 24174 raft_consensus.cc:374] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:43.882136 24174 raft_consensus.cc:729] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:43.882411 24174 consensus_queue.cc:260] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.885403 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_05b41a1454b94a0eaca01c06499ff3b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.886020 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05b41a1454b94a0eaca01c06499ff3b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:43.887284 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe5e08ad6af2442bb0a8fcc80a99b532 has no consensus state
I20240502 08:16:43.890579 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_30df43df108e43d4b44b20c60eeb8287" } new_table_name: "default.Table_9f3af157070d4bf5a099b901f335bc52" modify_external_catalogs: true
I20240502 08:16:43.893474 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet b844458ae73f413cbf2113027a5af359 (table default.Table_663684b67f204c6e82f363ea57b73fab [id=fd93259a378b4ea582a6661a59104abb]). Expected version 2 got 0
I20240502 08:16:43.893646 24175 tablet_bootstrap.cc:654] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:43.893728 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 49b816ea1d474f2db94119441146ba78 (table default.Table_20df1b504848439c869ae3739ac54185 [id=a0606a4053d04dd3814f09419a4a70c2]). Expected version 1 got 0
I20240502 08:16:43.902207 24180 ts_tablet_manager.cc:1888] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:43.902350 24180 log.cc:1196] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/23ce83c8001d4b2988b88c0ed9bf948c
I20240502 08:16:43.903209 24180 ts_tablet_manager.cc:1909] T 23ce83c8001d4b2988b88c0ed9bf948c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:43.904362 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 498) took 1372 ms. Trace:
I20240502 08:16:43.904511 24180 rpcz_store.cc:276] 0502 08:16:42.532200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.532257 (+ 57us) service_pool.cc:225] Handling call
0502 08:16:43.429761 (+897504us) ts_tablet_manager.cc:1120] Deleting tablet 23ce83c8001d4b2988b88c0ed9bf948c
0502 08:16:43.429769 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:43.679496 (+249727us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:43.902048 (+222552us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:43.904324 (+ 2276us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":897190}
I20240502 08:16:43.905115 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 499) took 1371 ms. Trace:
I20240502 08:16:43.905232 24180 rpcz_store.cc:276] 0502 08:16:42.533281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.533329 (+ 48us) service_pool.cc:225] Handling call
0502 08:16:43.904947 (+1371618us) ts_tablet_manager.cc:1120] Deleting tablet 91f1ab141585441db46d3027833932b0
0502 08:16:43.904956 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:43.905085 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1371297}
I20240502 08:16:43.905290 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a22cf11b49c04125a8e6c9b2942634a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:43.908864 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet c1e746013fde4de4862f213792e235ed
I20240502 08:16:43.909910 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e9171ebf74424c9aae5e1c8a5118f768
I20240502 08:16:43.910444 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 9b780a7870044d94bab14f0d9c978a0c
I20240502 08:16:43.910815 24180 tablet_replica.cc:330] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:43.911301 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet b6d8410ba7cc4b03bb64a138b1f9bb14
I20240502 08:16:43.911356 23754 tablet_service.cc:1514] Processing DeleteTablet for tablet e726e0ef49b247bebe07a7e7066c1af6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:43.913352 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 91646e7af3824520a1fe28e180eeb0a7 consensus state has no leader
I20240502 08:16:43.915545 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_31d76df4cb2f4950970e912e0a8797b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.916170 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31d76df4cb2f4950970e912e0a8797b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.918576 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet fddac683eda14ec1a0140926380f2c07
I20240502 08:16:43.918754 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_990c64d081c148d79dae73c2d06b1f11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.919272 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_990c64d081c148d79dae73c2d06b1f11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.919739 23723 leader_election.cc:302] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:43.920204 24489 raft_consensus.cc:2781] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:43.920315 24489 raft_consensus.cc:482] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:43.920446 24489 raft_consensus.cc:3037] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:43.922116 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba3aa2621fdb43fd8299f81af486ef1e consensus state has no leader
I20240502 08:16:43.922552 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_3e3f39f737b84f009fc0e2b1b1da6468"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.923039 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e3f39f737b84f009fc0e2b1b1da6468 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.924530 24489 raft_consensus.cc:504] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.924513 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8040b4c1a36844658c4cdf62e9ae61de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.925025 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8040b4c1a36844658c4cdf62e9ae61de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.925784 24489 leader_election.cc:288] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:43.926141 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_827517f3c36f43e585faba823c6d068d" } new_table_name: "default.Table_3e0123590a92473dabaacc9c158859e7" modify_external_catalogs: true
I20240502 08:16:43.926585 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d0fd869295b5475c902bde5dd5ee95d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:43.927038 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0fd869295b5475c902bde5dd5ee95d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.928500 23723 leader_election.cc:302] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:43.925627 24771 tablet.cc:1722] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240502 08:16:43.929459 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a22cf11b49c04125a8e6c9b2942634a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:43.930572 24596 raft_consensus.cc:2781] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:43.935354 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 (table default.Table_9663c413d5be4a049eb7a02ececc1276 [id=ad4f0c8adc524a019beaf37e153b33a7]) successfully deleted
W20240502 08:16:43.936025 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 9467974d67c64c7eb5e488ac51f1240d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9467974d67c64c7eb5e488ac51f1240d already in progress: creating tablet
I20240502 08:16:43.936740 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet b6c52f1cae3a4ad38a9f95bfaf055e76 (table default.Table_35b808b0487f463dbcfa4a773a2a20f0 [id=922fb979b5894023a68820aece0b8c5e]) successfully deleted
W20240502 08:16:43.937008 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet b8b1fa17dad143f0a7f1751e651eb81a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8b1fa17dad143f0a7f1751e651eb81a already in progress: creating tablet
W20240502 08:16:43.937436 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 already in progress: creating tablet
W20240502 08:16:43.937806 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e47459b5f5bb4ecb96f76240fefa1f5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e47459b5f5bb4ecb96f76240fefa1f5d already in progress: creating tablet
W20240502 08:16:43.938093 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 76e0f767c6534a608023e6e6d33ca7a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76e0f767c6534a608023e6e6d33ca7a8 already in progress: creating tablet
W20240502 08:16:43.938349 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 1da01e2c2f054ccfb53383e70e16da84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1da01e2c2f054ccfb53383e70e16da84 already in progress: creating tablet
W20240502 08:16:43.938629 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e5f5597402374527bf0e5e50178df713 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5f5597402374527bf0e5e50178df713 already in progress: creating tablet
I20240502 08:16:43.941247 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet a5a5a410b6314818b10927e35032f95d
I20240502 08:16:43.941838 23602 catalog_manager.cc:6554] ReplaceTablet: tablet a5a5a410b6314818b10927e35032f95d deleted and replaced by tablet 92b830e128ac439eba286f29dbb20ee4
W20240502 08:16:43.945353 23594 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.125:36189, user_credentials={real_user=slave}} blocked reactor thread for 87652us
I20240502 08:16:43.949071 24174 ts_tablet_manager.cc:1398] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.068s user 0.002s sys 0.000s
I20240502 08:16:43.952383 24179 raft_consensus.cc:2258] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:43.952798 24038 tablet_service.cc:1467] Processing CreateTablet for tablet b123bb81d0b44d4ea6443536168d085c (DEFAULT_TABLE table=default.Table_45e91d35e2c245d59419f7105a5b9c8b [id=4f198dcf66f4473abc832b185b1c9e96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.952905 24025 tablet_service.cc:1467] Processing CreateTablet for tablet a238b67dd6e84aa58d14a7802b6ba0ee (DEFAULT_TABLE table=default.Table_076bc414d7814432bfd813a959dc2730 [id=57fc78a6eefb423da89bc738b355dcfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.953310 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet 4645143a00d440a7b6d56c5e30d249f1 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:43.953356 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a238b67dd6e84aa58d14a7802b6ba0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.953903 24755 raft_consensus.cc:482] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:43.954038 24755 raft_consensus.cc:504] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:43.955196 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c374f42f353497fa1d745164e7fbf61" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:43.955729 23993 leader_election.cc:341] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:43.958293 23603 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet b844458ae73f413cbf2113027a5af359 (table default.Table_663684b67f204c6e82f363ea57b73fab [id=fd93259a378b4ea582a6661a59104abb]). Expected version 2 got 0
I20240502 08:16:43.958827 24021 tablet_service.cc:1467] Processing CreateTablet for tablet 96013fa1fa014b1891eb621cc3a81cdc (DEFAULT_TABLE table=default.Table_7ee63691fc8240ad9fd8db92d778eb3c [id=fb2db1142b024839be0663235b8272be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.959224 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96013fa1fa014b1891eb621cc3a81cdc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:43.959317 23723 consensus_peers.cc:485] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:43.959677 23723 leader_election.cc:302] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
W20240502 08:16:43.960235 23723 leader_election.cc:341] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:43.960376 23723 leader_election.cc:302] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:43.960728 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 23ce83c8001d4b2988b88c0ed9bf948c (table default.Table_190eae2a8c64440f8d19842f8b6ac254 [id=54454efbe9394170822f237dc8e6115e]) successfully deleted
W20240502 08:16:43.960934 23723 leader_election.cc:341] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:43.960959 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 91f1ab141585441db46d3027833932b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91f1ab141585441db46d3027833932b0 already in progress: creating tablet
W20240502 08:16:43.961426 23723 leader_election.cc:341] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:43.961769 23760 tablet_service.cc:1467] Processing CreateTablet for tablet b123bb81d0b44d4ea6443536168d085c (DEFAULT_TABLE table=default.Table_45e91d35e2c245d59419f7105a5b9c8b [id=4f198dcf66f4473abc832b185b1c9e96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.961858 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 90d44be09f284f3d9042038c92f3b28d (DEFAULT_TABLE table=default.Table_13623be380874b4d8b231ac4ed815b11 [id=a16ce0f4bf7a42f3a8471b0fb85778e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.962059 24179 ts_tablet_manager.cc:1875] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:43.962097 23858 leader_election.cc:341] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 21832f0151ec4cf48f51f9bd2cb2b5ec
I20240502 08:16:43.962174 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b123bb81d0b44d4ea6443536168d085c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.962174 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d44be09f284f3d9042038c92f3b28d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.962249 23858 leader_election.cc:302] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:43.964181 24755 leader_election.cc:288] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:43.965729 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c374f42f353497fa1d745164e7fbf61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.966176 24181 tablet_replica.cc:330] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:43.966361 24181 raft_consensus.cc:2229] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:43.966537 24181 raft_consensus.cc:2258] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:43.967293 23921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c374f42f353497fa1d745164e7fbf61" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:43.967964 23996 leader_election.cc:341] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:43.968097 23996 leader_election.cc:302] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:43.968276 23996 raft_consensus.cc:2670] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:16:43.968709 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b123bb81d0b44d4ea6443536168d085c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.971297 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 843edd7092464981adad317b48871645 (DEFAULT_TABLE table=default.Table_7db6534bd8144b208a6eaa13f381cdfd [id=c27df05d1e8e4119a5fa06c26b8d7116]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.971666 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843edd7092464981adad317b48871645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.973747 24489 raft_consensus.cc:2781] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:43.973875 24489 raft_consensus.cc:482] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:43.973961 24489 raft_consensus.cc:3037] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:43.973887 24037 tablet_service.cc:1467] Processing CreateTablet for tablet d9b67c8cd7f34cbba6401da59dec526c (DEFAULT_TABLE table=default.Table_3a47de7a9c814ca2a1656345ebbf39c7 [id=0305599976c6431ba3a148af6c7edad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.974287 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9b67c8cd7f34cbba6401da59dec526c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.976997 24489 raft_consensus.cc:504] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.977846 24032 tablet_service.cc:1467] Processing CreateTablet for tablet f01069d60dbe4ed79a16238d2e89fbe2 (DEFAULT_TABLE table=default.Table_a20c7e6c20e6416cb1934d9b830d52a3 [id=7f22f3c48ea74649be53f444041bd75e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.978192 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f01069d60dbe4ed79a16238d2e89fbe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.979202 24180 raft_consensus.cc:2229] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:43.979372 24180 raft_consensus.cc:2258] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:43.979969 24180 ts_tablet_manager.cc:1875] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:43.981865 24039 tablet_service.cc:1467] Processing CreateTablet for tablet eb9c6d9019f84305b09d619d65f45738 (DEFAULT_TABLE table=default.Table_e82a1ff6cbbd4d07885fe037b71bf199 [id=e8d8c69c61e744df8a063646dcf304fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.982050 24774 raft_consensus.cc:2726] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:43.982228 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9c6d9019f84305b09d619d65f45738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.982447 24175 tablet_bootstrap.cc:492] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:43.982586 24175 ts_tablet_manager.cc:1367] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.153s user 0.001s sys 0.003s
I20240502 08:16:43.982807 24773 tablet.cc:1722] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:43.983323 24175 raft_consensus.cc:348] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.983525 24175 raft_consensus.cc:374] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:43.983621 24175 raft_consensus.cc:729] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:43.983865 24175 consensus_queue.cc:260] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.984320 24174 tablet_bootstrap.cc:492] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:43.984387 24175 ts_tablet_manager.cc:1398] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:16:43.984848 24175 tablet_bootstrap.cc:492] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:43.985879 24036 tablet_service.cc:1467] Processing CreateTablet for tablet e9941681b26d4c97b06a91155043b040 (DEFAULT_TABLE table=default.Table_b0b35fed485c4223a4710de9abfd86fb [id=b863ddadc61c42d5a3fc5a1621cfb7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.985994 24174 tablet_bootstrap.cc:654] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:43.986234 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9941681b26d4c97b06a91155043b040. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.986465 24175 tablet_bootstrap.cc:654] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:43.989605 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2d7b6f4e442ae95b2d1002d571b6a" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:43.989683 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet c1e746013fde4de4862f213792e235ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.989799 24057 raft_consensus.cc:3037] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:43.989912 24022 tablet_service.cc:1514] Processing DeleteTablet for tablet e9171ebf74424c9aae5e1c8a5118f768 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.990103 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b780a7870044d94bab14f0d9c978a0c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.990288 24034 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d8410ba7cc4b03bb64a138b1f9bb14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.990479 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet fddac683eda14ec1a0140926380f2c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.990649 24026 tablet_service.cc:1514] Processing DeleteTablet for tablet a5a5a410b6314818b10927e35032f95d with delete_type TABLET_DATA_DELETED (replaced by tablet 92b830e128ac439eba286f29dbb20ee4) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:43.991518 24179 ts_tablet_manager.cc:1888] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:43.991644 24179 log.cc:1196] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/21832f0151ec4cf48f51f9bd2cb2b5ec
I20240502 08:16:43.992997 24057 raft_consensus.cc:2454] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:43.993885 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 62a268f8c6004ebbb6c830839b842e22 (DEFAULT_TABLE table=default.Table_ee616660e58a4aaa882ad75f8078f5e3 [id=b1d5e0fda61348388e2c8d5ce62ff42e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.994247 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a268f8c6004ebbb6c830839b842e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.995777 23723 leader_election.cc:302] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:43.996109 24630 raft_consensus.cc:2781] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:43.996273 24630 raft_consensus.cc:686] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:43.996541 24630 consensus_queue.cc:237] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.996843 24179 ts_tablet_manager.cc:1909] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
W20240502 08:16:43.962443 23858 raft_consensus.cc:2670] T 21832f0151ec4cf48f51f9bd2cb2b5ec P 5b89a1f0485a44b7ac1710e2369a809b: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240502 08:16:43.997356 23858 leader_election.cc:341] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 0be4f8d5cc09483d8d1aa1e37679b544
I20240502 08:16:43.997645 24596 raft_consensus.cc:686] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:43.997746 23858 leader_election.cc:302] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:43.997874 24596 consensus_queue.cc:237] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:43.998010 24695 raft_consensus.cc:2726] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:43.998093 24181 ts_tablet_manager.cc:1875] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:43.998502 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0281c8123eab4c48bddc377f5695c6e0 consensus state has no leader
I20240502 08:16:44.000787 24179 tablet_replica.cc:330] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:44.001230 23903 tablet_service.cc:1467] Processing CreateTablet for tablet b123bb81d0b44d4ea6443536168d085c (DEFAULT_TABLE table=default.Table_45e91d35e2c245d59419f7105a5b9c8b [id=4f198dcf66f4473abc832b185b1c9e96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.001230 23896 tablet_service.cc:1467] Processing CreateTablet for tablet a238b67dd6e84aa58d14a7802b6ba0ee (DEFAULT_TABLE table=default.Table_076bc414d7814432bfd813a959dc2730 [id=57fc78a6eefb423da89bc738b355dcfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.001654 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 96013fa1fa014b1891eb621cc3a81cdc (DEFAULT_TABLE table=default.Table_7ee63691fc8240ad9fd8db92d778eb3c [id=fb2db1142b024839be0663235b8272be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.002128 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 90d44be09f284f3d9042038c92f3b28d (DEFAULT_TABLE table=default.Table_13623be380874b4d8b231ac4ed815b11 [id=a16ce0f4bf7a42f3a8471b0fb85778e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.002108 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b123bb81d0b44d4ea6443536168d085c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.002519 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 62a268f8c6004ebbb6c830839b842e22 (DEFAULT_TABLE table=default.Table_ee616660e58a4aaa882ad75f8078f5e3 [id=b1d5e0fda61348388e2c8d5ce62ff42e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.002743 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d44be09f284f3d9042038c92f3b28d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.002974 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a268f8c6004ebbb6c830839b842e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.004809 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96013fa1fa014b1891eb621cc3a81cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.009060 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a238b67dd6e84aa58d14a7802b6ba0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.017427 24179 raft_consensus.cc:2229] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:44.017488 24489 leader_election.cc:288] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:44.017684 24179 raft_consensus.cc:2258] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:44.018863 24179 ts_tablet_manager.cc:1875] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:44.023242 23434 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 76282.5us
I20240502 08:16:44.024623 23901 tablet_service.cc:1467] Processing CreateTablet for tablet e9941681b26d4c97b06a91155043b040 (DEFAULT_TABLE table=default.Table_b0b35fed485c4223a4710de9abfd86fb [id=b863ddadc61c42d5a3fc5a1621cfb7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.025135 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9941681b26d4c97b06a91155043b040. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.028744 24181 ts_tablet_manager.cc:1888] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:44.028898 24181 log.cc:1196] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/7c374f42f353497fa1d745164e7fbf61
I20240502 08:16:44.029318 24181 ts_tablet_manager.cc:1909] T 7c374f42f353497fa1d745164e7fbf61 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:44.033838 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet c1e746013fde4de4862f213792e235ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.036222 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 843edd7092464981adad317b48871645 (DEFAULT_TABLE table=default.Table_7db6534bd8144b208a6eaa13f381cdfd [id=c27df05d1e8e4119a5fa06c26b8d7116]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:44.036615 23723 leader_election.cc:341] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:44.038697 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843edd7092464981adad317b48871645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.040340 23905 tablet_service.cc:1467] Processing CreateTablet for tablet eb9c6d9019f84305b09d619d65f45738 (DEFAULT_TABLE table=default.Table_e82a1ff6cbbd4d07885fe037b71bf199 [id=e8d8c69c61e744df8a063646dcf304fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.040799 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9c6d9019f84305b09d619d65f45738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.041541 23892 tablet_service.cc:1467] Processing CreateTablet for tablet f01069d60dbe4ed79a16238d2e89fbe2 (DEFAULT_TABLE table=default.Table_a20c7e6c20e6416cb1934d9b830d52a3 [id=7f22f3c48ea74649be53f444041bd75e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.042136 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f01069d60dbe4ed79a16238d2e89fbe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.043032 23608 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_663684b67f204c6e82f363ea57b73fab [id=fd93259a378b4ea582a6661a59104abb] alter complete (version 2)
I20240502 08:16:44.043172 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet e9171ebf74424c9aae5e1c8a5118f768 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.043989 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d8410ba7cc4b03bb64a138b1f9bb14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.044523 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet fddac683eda14ec1a0140926380f2c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.044855 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet a5a5a410b6314818b10927e35032f95d with delete_type TABLET_DATA_DELETED (replaced by tablet 92b830e128ac439eba286f29dbb20ee4) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.045315 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c95dfda9d3b4e2ca9397b77ad71c011 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.046173 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2d7b6f4e442ae95b2d1002d571b6a" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:44.048707 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 4645143a00d440a7b6d56c5e30d249f1
I20240502 08:16:44.049122 23602 catalog_manager.cc:6554] ReplaceTablet: tablet 4645143a00d440a7b6d56c5e30d249f1 deleted and replaced by tablet 058e13ba3a474af3aabebc93c5a88124
I20240502 08:16:44.049194 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 7c374f42f353497fa1d745164e7fbf61 (table default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]) successfully deleted
W20240502 08:16:44.049907 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet c1e746013fde4de4862f213792e235ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1e746013fde4de4862f213792e235ed already in progress: creating tablet
W20240502 08:16:44.050945 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e9171ebf74424c9aae5e1c8a5118f768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9171ebf74424c9aae5e1c8a5118f768 already in progress: creating tablet
W20240502 08:16:44.051245 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 9b780a7870044d94bab14f0d9c978a0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b780a7870044d94bab14f0d9c978a0c already in progress: creating tablet
W20240502 08:16:44.051479 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet b6d8410ba7cc4b03bb64a138b1f9bb14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6d8410ba7cc4b03bb64a138b1f9bb14 already in progress: creating tablet
W20240502 08:16:44.051728 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet fddac683eda14ec1a0140926380f2c07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fddac683eda14ec1a0140926380f2c07 already in progress: creating tablet
W20240502 08:16:44.051959 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet a5a5a410b6314818b10927e35032f95d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5a5a410b6314818b10927e35032f95d already in progress: creating tablet
I20240502 08:16:44.066725 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 843edd7092464981adad317b48871645 (DEFAULT_TABLE table=default.Table_7db6534bd8144b208a6eaa13f381cdfd [id=c27df05d1e8e4119a5fa06c26b8d7116]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.067345 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843edd7092464981adad317b48871645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.068117 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet 5d04312bd81f46ed8c49c42b1af1aec9 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:44.068390 24033 tablet_service.cc:1467] Processing CreateTablet for tablet e712916274144e5e86661561646c3521 (DEFAULT_TABLE table=default.Table_50a6c708582c422c89aef5f2ca35a88a [id=0a4db0daf1734dea88ba36a30beef34c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.069131 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e712916274144e5e86661561646c3521. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.073962 23750 tablet_service.cc:1467] Processing CreateTablet for tablet d9b67c8cd7f34cbba6401da59dec526c (DEFAULT_TABLE table=default.Table_3a47de7a9c814ca2a1656345ebbf39c7 [id=0305599976c6431ba3a148af6c7edad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.074335 24021 tablet_service.cc:1467] Processing CreateTablet for tablet 0f19886fda394def977508c993b6e2d2 (DEFAULT_TABLE table=default.Table_e5391748a4874f97a3546ba6c5fb22eb [id=ed93837aa0004562986d05c522c490a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.074457 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9b67c8cd7f34cbba6401da59dec526c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.074810 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f19886fda394def977508c993b6e2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.081952 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 62a268f8c6004ebbb6c830839b842e22 (DEFAULT_TABLE table=default.Table_ee616660e58a4aaa882ad75f8078f5e3 [id=b1d5e0fda61348388e2c8d5ce62ff42e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.082584 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a268f8c6004ebbb6c830839b842e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.085186 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b780a7870044d94bab14f0d9c978a0c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.086195 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 3abfc0c885d64c39808a6e7c815afd7e (DEFAULT_TABLE table=default.Table_59c11afb388848da8ac134294bc6c485 [id=18516b0bae644f96b2cba96d6ea18799]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.086691 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 303a0a3748ee422faab856e1c4bff66d (DEFAULT_TABLE table=default.Table_850349fb9fa64ccb884825b10bbec754 [id=168b446deda748a8a2da7a829e845c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.086812 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet 4645143a00d440a7b6d56c5e30d249f1 with delete_type TABLET_DATA_DELETED (replaced by tablet 058e13ba3a474af3aabebc93c5a88124) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.086710 24025 tablet_service.cc:1467] Processing CreateTablet for tablet e54b09eac68f423086620617dd4a6ffc (DEFAULT_TABLE table=default.Table_e8bb862a5508434a83ad4f5c8be82002 [id=b13a6a084a8d45728e3932be14219d5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.087142 24037 tablet_service.cc:1467] Processing CreateTablet for tablet d7e2da9cff594d90819f6b216dbed552 (DEFAULT_TABLE table=default.Table_2cc3c92f41994b1ab2144237110b1d88 [id=dd1e27eb85b549c6991356a1b2af1413]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.087631 24032 tablet_service.cc:1467] Processing CreateTablet for tablet fe5e08ad6af2442bb0a8fcc80a99b532 (DEFAULT_TABLE table=default.Table_e2cd64637b7843539c015901a34d3258 [id=dc0271352a9f45538aea81c8898ff3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.087837 24021 tablet_service.cc:1467] Processing CreateTablet for tablet 725982e4712e442d8ff1fd3f796e18f5 (DEFAULT_TABLE table=default.Table_02cd49f90ab14e919f913d9eb37ebdc1 [id=b1524d4ccab346b7bb0c817441eabd09]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.088097 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 3777d2e9d4464552ad1c4c8c0d3e9888 (DEFAULT_TABLE table=default.Table_53c08a7447a44323883744c033efcbfb [id=78b34e35d763495aadf6a9eb242741c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.088573 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 29f6c3cea4b14364b6d073a92171119d (DEFAULT_TABLE table=default.Table_c1886293aee349b2b9e7177c1cfd6d26 [id=0853ffc4d48549bc93e251a5dbc55901]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.089835 24173 tablet_bootstrap.cc:492] T c08228fcd1304528b9446003851f6c70 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:44.092481 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2e579720e5df47e9875f86b878b88260"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:44.092988 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d132dfd0ffff4fd68be038259362c284" } new_table_name: "default.Table_2d6b281d29d14441ac6abe9705633102" modify_external_catalogs: true
W20240502 08:16:44.093158 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e579720e5df47e9875f86b878b88260 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.090631 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3abfc0c885d64c39808a6e7c815afd7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.095216 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_52cae77d343b4c43bb8cf4fa6f4727af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:44.095749 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52cae77d343b4c43bb8cf4fa6f4727af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.097736 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3777d2e9d4464552ad1c4c8c0d3e9888. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.098588 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_f194510125f340b58d41408e911d2721"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:44.099035 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f194510125f340b58d41408e911d2721 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.096777 23921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd57b0357ca749c389bc61add20cd3e7" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:44.100512 23896 tablet_service.cc:1467] Processing CreateTablet for tablet d9b67c8cd7f34cbba6401da59dec526c (DEFAULT_TABLE table=default.Table_3a47de7a9c814ca2a1656345ebbf39c7 [id=0305599976c6431ba3a148af6c7edad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.101701 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b84ccd0af14546dd9a2dd5aab4387f27" } new_table_name: "default.Table_6118b7d5075e4fbb9eaab8b4a9fd1627" modify_external_catalogs: true
I20240502 08:16:44.101771 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7e2da9cff594d90819f6b216dbed552. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.103909 23921 raft_consensus.cc:3037] T fd57b0357ca749c389bc61add20cd3e7 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:44.105321 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9b67c8cd7f34cbba6401da59dec526c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.105824 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe5e08ad6af2442bb0a8fcc80a99b532. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.109747 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e54b09eac68f423086620617dd4a6ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.113370 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 0f19886fda394def977508c993b6e2d2 (DEFAULT_TABLE table=default.Table_e5391748a4874f97a3546ba6c5fb22eb [id=ed93837aa0004562986d05c522c490a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.115075 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 303a0a3748ee422faab856e1c4bff66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.115464 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f6c3cea4b14364b6d073a92171119d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.124307 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_20f1683b9d814b74a26458bb2c1c769b" } modify_external_catalogs: true
I20240502 08:16:44.124596 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_20f1683b9d814b74a26458bb2c1c769b" } modify_external_catalogs: true
I20240502 08:16:44.127434 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 21832f0151ec4cf48f51f9bd2cb2b5ec (table default.Table_75cadbecaad04383945a6e0b1e226d86 [id=4c539e8625094a278029356ac44da4c0]) successfully deleted
I20240502 08:16:44.128062 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bf53572d3cc544e1aa5674239aaa62f9" } new_table_name: "default.Table_c09636e3919b4ef68dbbddda1b18605d" modify_external_catalogs: true
I20240502 08:16:44.128044 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 90d44be09f284f3d9042038c92f3b28d (DEFAULT_TABLE table=default.Table_13623be380874b4d8b231ac4ed815b11 [id=a16ce0f4bf7a42f3a8471b0fb85778e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.128592 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d44be09f284f3d9042038c92f3b28d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.129386 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_efda0c8c5d1d4e138081bfe146a06fa4" } modify_external_catalogs: true
I20240502 08:16:44.129621 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_efda0c8c5d1d4e138081bfe146a06fa4" } modify_external_catalogs: true
I20240502 08:16:44.131780 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_92759ae885e24e0395e7a8b3f6f59181" } modify_external_catalogs: true
I20240502 08:16:44.131915 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fec81a33cad74ba3bfbc7787a7bd5829" } new_table_name: "default.Table_07e89c5c963e42b19f037caf9bbfa7af" modify_external_catalogs: true
I20240502 08:16:44.131981 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_92759ae885e24e0395e7a8b3f6f59181" } modify_external_catalogs: true
I20240502 08:16:44.132243 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 96013fa1fa014b1891eb621cc3a81cdc (DEFAULT_TABLE table=default.Table_7ee63691fc8240ad9fd8db92d778eb3c [id=fb2db1142b024839be0663235b8272be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.132591 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96013fa1fa014b1891eb621cc3a81cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.133100 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_20df1b504848439c869ae3739ac54185" } modify_external_catalogs: true
I20240502 08:16:44.133236 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_20df1b504848439c869ae3739ac54185" } modify_external_catalogs: true
W20240502 08:16:44.133599 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 4645143a00d440a7b6d56c5e30d249f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4645143a00d440a7b6d56c5e30d249f1 already in progress: creating tablet
I20240502 08:16:44.134228 24033 tablet_service.cc:1467] Processing CreateTablet for tablet cda83e82168a456a8c3d42d69a1a743b (DEFAULT_TABLE table=default.Table_2b4b1372dc684d71aa26d7e8a435eaed [id=e6c56df3e6ed49da99e8ea1bae916b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.134636 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3def35abf8ad4ff49c5ae77acb410e01" } modify_external_catalogs: true
W20240502 08:16:44.134729 23434 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 73336.1us
I20240502 08:16:44.134819 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3def35abf8ad4ff49c5ae77acb410e01" } modify_external_catalogs: true
I20240502 08:16:44.135635 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda83e82168a456a8c3d42d69a1a743b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:44.137642 23723 leader_election.cc:341] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: fd57b0357ca749c389bc61add20cd3e7
I20240502 08:16:44.139601 23756 tablet_service.cc:1467] Processing CreateTablet for tablet e9941681b26d4c97b06a91155043b040 (DEFAULT_TABLE table=default.Table_b0b35fed485c4223a4710de9abfd86fb [id=b863ddadc61c42d5a3fc5a1621cfb7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.140027 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet c1e746013fde4de4862f213792e235ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.140234 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet e9171ebf74424c9aae5e1c8a5118f768 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.140374 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b780a7870044d94bab14f0d9c978a0c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.140515 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d8410ba7cc4b03bb64a138b1f9bb14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.141191 23752 tablet_service.cc:1467] Processing CreateTablet for tablet e712916274144e5e86661561646c3521 (DEFAULT_TABLE table=default.Table_50a6c708582c422c89aef5f2ca35a88a [id=0a4db0daf1734dea88ba36a30beef34c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.144277 24695 raft_consensus.cc:482] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:44.144539 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725982e4712e442d8ff1fd3f796e18f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.144516 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a1b11bbb99a24f79b6d33fbf9cae4d5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:44.144917 23750 tablet_service.cc:1467] Processing CreateTablet for tablet f01069d60dbe4ed79a16238d2e89fbe2 (DEFAULT_TABLE table=default.Table_a20c7e6c20e6416cb1934d9b830d52a3 [id=7f22f3c48ea74649be53f444041bd75e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.145043 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f19886fda394def977508c993b6e2d2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:44.145120 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1b11bbb99a24f79b6d33fbf9cae4d5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.145648 23749 tablet_service.cc:1467] Processing CreateTablet for tablet a238b67dd6e84aa58d14a7802b6ba0ee (DEFAULT_TABLE table=default.Table_076bc414d7814432bfd813a959dc2730 [id=57fc78a6eefb423da89bc738b355dcfc]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:44.145988 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 51597.4us
I20240502 08:16:44.146843 23903 tablet_service.cc:1467] Processing CreateTablet for tablet cda83e82168a456a8c3d42d69a1a743b (DEFAULT_TABLE table=default.Table_2b4b1372dc684d71aa26d7e8a435eaed [id=e6c56df3e6ed49da99e8ea1bae916b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.147364 24773 tablet.cc:1722] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240502 08:16:44.147377 24695 raft_consensus.cc:504] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:44.147691 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9941681b26d4c97b06a91155043b040. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.148079 24695 leader_election.cc:288] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:44.148455 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 725982e4712e442d8ff1fd3f796e18f5 (DEFAULT_TABLE table=default.Table_02cd49f90ab14e919f913d9eb37ebdc1 [id=b1524d4ccab346b7bb0c817441eabd09]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:44.149546 23861 leader_election.cc:341] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 5c95dfda9d3b4e2ca9397b77ad71c011
I20240502 08:16:44.151104 24173 ts_tablet_manager.cc:1367] T c08228fcd1304528b9446003851f6c70 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.289s user 0.021s sys 0.021s
W20240502 08:16:44.151829 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d2bb4dba23ac4b4eb53b3f2d75d0442e consensus state has no leader
I20240502 08:16:44.151829 24173 raft_consensus.cc:348] T c08228fcd1304528b9446003851f6c70 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:44.152019 24173 raft_consensus.cc:374] T c08228fcd1304528b9446003851f6c70 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.152081 24173 raft_consensus.cc:729] T c08228fcd1304528b9446003851f6c70 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.152052 23894 tablet_service.cc:1467] Processing CreateTablet for tablet e712916274144e5e86661561646c3521 (DEFAULT_TABLE table=default.Table_50a6c708582c422c89aef5f2ca35a88a [id=0a4db0daf1734dea88ba36a30beef34c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.152446 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda83e82168a456a8c3d42d69a1a743b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.152705 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet fddac683eda14ec1a0140926380f2c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.152923 23754 tablet_service.cc:1514] Processing DeleteTablet for tablet a5a5a410b6314818b10927e35032f95d with delete_type TABLET_DATA_DELETED (replaced by tablet 92b830e128ac439eba286f29dbb20ee4) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.154641 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725982e4712e442d8ff1fd3f796e18f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.154902 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_02b3fe4baa274e408220d9d27d9db7fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:44.155133 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 3abfc0c885d64c39808a6e7c815afd7e (DEFAULT_TABLE table=default.Table_59c11afb388848da8ac134294bc6c485 [id=18516b0bae644f96b2cba96d6ea18799]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.155445 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3abfc0c885d64c39808a6e7c815afd7e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:44.155412 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02b3fe4baa274e408220d9d27d9db7fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.157073 24173 consensus_queue.cc:260] T c08228fcd1304528b9446003851f6c70 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:44.158665 24173 ts_tablet_manager.cc:1398] T c08228fcd1304528b9446003851f6c70 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 08:16:44.159817 24173 tablet_bootstrap.cc:492] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:44.160431 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4b291bdb81a4efbac7f6691e5577bbe consensus state has no leader
I20240502 08:16:44.162575 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e712916274144e5e86661561646c3521. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.162657 24173 tablet_bootstrap.cc:654] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.164031 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7937f18be63b41d7945c72ed597adfcb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:44.164525 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7937f18be63b41d7945c72ed597adfcb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.166112 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a238b67dd6e84aa58d14a7802b6ba0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.169219 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e712916274144e5e86661561646c3521. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.169879 23757 tablet_service.cc:1467] Processing CreateTablet for tablet eb9c6d9019f84305b09d619d65f45738 (DEFAULT_TABLE table=default.Table_e82a1ff6cbbd4d07885fe037b71bf199 [id=e8d8c69c61e744df8a063646dcf304fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.170092 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet fbcf17a3bcc844b1b9676580f0749c9c
I20240502 08:16:44.170337 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9c6d9019f84305b09d619d65f45738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.143896 24779 raft_consensus.cc:482] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:44.172817 24779 raft_consensus.cc:504] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:44.174922 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_722c9cabaa9a4f43a7c8aeab8a17dc05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:44.175387 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_722c9cabaa9a4f43a7c8aeab8a17dc05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.175452 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f01069d60dbe4ed79a16238d2e89fbe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.176195 23898 tablet_service.cc:1467] Processing CreateTablet for tablet d7e2da9cff594d90819f6b216dbed552 (DEFAULT_TABLE table=default.Table_2cc3c92f41994b1ab2144237110b1d88 [id=dd1e27eb85b549c6991356a1b2af1413]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.176530 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7e2da9cff594d90819f6b216dbed552. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.178032 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 0f19886fda394def977508c993b6e2d2 (DEFAULT_TABLE table=default.Table_e5391748a4874f97a3546ba6c5fb22eb [id=ed93837aa0004562986d05c522c490a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.186132 23751 tablet_service.cc:1467] Processing CreateTablet for tablet cda83e82168a456a8c3d42d69a1a743b (DEFAULT_TABLE table=default.Table_2b4b1372dc684d71aa26d7e8a435eaed [id=e6c56df3e6ed49da99e8ea1bae916b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.186512 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f19886fda394def977508c993b6e2d2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:44.186661 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d48d1f30fb34ee6a9a68908a3442d43 consensus state has no leader
I20240502 08:16:44.188499 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 1ce5ccfd29724dc49afd0892b99f3169
I20240502 08:16:44.189929 24779 leader_election.cc:288] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:44.190353 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 4645143a00d440a7b6d56c5e30d249f1 with delete_type TABLET_DATA_DELETED (replaced by tablet 058e13ba3a474af3aabebc93c5a88124) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.189198 23886 tablet_service.cc:1467] Processing CreateTablet for tablet e54b09eac68f423086620617dd4a6ffc (DEFAULT_TABLE table=default.Table_e8bb862a5508434a83ad4f5c8be82002 [id=b13a6a084a8d45728e3932be14219d5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.190623 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 725982e4712e442d8ff1fd3f796e18f5 (DEFAULT_TABLE table=default.Table_02cd49f90ab14e919f913d9eb37ebdc1 [id=b1524d4ccab346b7bb0c817441eabd09]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.190861 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e54b09eac68f423086620617dd4a6ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.191226 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 3abfc0c885d64c39808a6e7c815afd7e (DEFAULT_TABLE table=default.Table_59c11afb388848da8ac134294bc6c485 [id=18516b0bae644f96b2cba96d6ea18799]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.193806 24179 ts_tablet_manager.cc:1888] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240502 08:16:44.194283 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet edd7244b9f5741a8aa1c6835fe544676 consensus state has no leader
I20240502 08:16:44.195326 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 303a0a3748ee422faab856e1c4bff66d (DEFAULT_TABLE table=default.Table_850349fb9fa64ccb884825b10bbec754 [id=168b446deda748a8a2da7a829e845c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.196477 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda83e82168a456a8c3d42d69a1a743b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.198658 23753 tablet_service.cc:1467] Processing CreateTablet for tablet e54b09eac68f423086620617dd4a6ffc (DEFAULT_TABLE table=default.Table_e8bb862a5508434a83ad4f5c8be82002 [id=b13a6a084a8d45728e3932be14219d5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.200140 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3abfc0c885d64c39808a6e7c815afd7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.201814 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a77d44d9077d44fb9db187597d42756b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:44.201913 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 29f6c3cea4b14364b6d073a92171119d (DEFAULT_TABLE table=default.Table_c1886293aee349b2b9e7177c1cfd6d26 [id=0853ffc4d48549bc93e251a5dbc55901]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:44.202325 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a77d44d9077d44fb9db187597d42756b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.202984 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 303a0a3748ee422faab856e1c4bff66d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:44.203033 23594 consensus_peers.cc:485] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 -> Peer 84f3c07a31414c4f8fb61c44dc9f16e1 (127.22.225.125:36189): Couldn't send request to peer 84f3c07a31414c4f8fb61c44dc9f16e1. Status: Network error: Client connection negotiation failed: client connection to 127.22.225.125:36189: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:16:44.203713 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f6c3cea4b14364b6d073a92171119d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.203889 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0ba7fb682ff94893b3ef11abc45a7f6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:44.204150 23761 tablet_service.cc:1467] Processing CreateTablet for tablet d7e2da9cff594d90819f6b216dbed552 (DEFAULT_TABLE table=default.Table_2cc3c92f41994b1ab2144237110b1d88 [id=dd1e27eb85b549c6991356a1b2af1413]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:44.204355 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ba7fb682ff94893b3ef11abc45a7f6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.205706 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6ce45bf2f8e64d759b44ee43f3df434d" } new_table_name: "default.Table_38fd1abca8cf4743bee0a0cb23748c90" modify_external_catalogs: true
I20240502 08:16:44.206751 24179 log.cc:1196] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/dcdd765895254a39b86b692b5d812c83
I20240502 08:16:44.208262 23766 tablet_service.cc:1467] Processing CreateTablet for tablet fe5e08ad6af2442bb0a8fcc80a99b532 (DEFAULT_TABLE table=default.Table_e2cd64637b7843539c015901a34d3258 [id=dc0271352a9f45538aea81c8898ff3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.208631 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 4645143a00d440a7b6d56c5e30d249f1 with delete_type TABLET_DATA_DELETED (replaced by tablet 058e13ba3a474af3aabebc93c5a88124) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.208807 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e54b09eac68f423086620617dd4a6ffc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:44.213284 23723 consensus_peers.cc:485] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6c52f1cae3a4ad38a9f95bfaf055e76. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:44.218397 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 49b816ea1d474f2db94119441146ba78
I20240502 08:16:44.224390 23921 raft_consensus.cc:2454] T fd57b0357ca749c389bc61add20cd3e7 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:44.225771 23887 tablet_service.cc:1467] Processing CreateTablet for tablet fe5e08ad6af2442bb0a8fcc80a99b532 (DEFAULT_TABLE table=default.Table_e2cd64637b7843539c015901a34d3258 [id=dc0271352a9f45538aea81c8898ff3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.226949 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 303a0a3748ee422faab856e1c4bff66d (DEFAULT_TABLE table=default.Table_850349fb9fa64ccb884825b10bbec754 [id=168b446deda748a8a2da7a829e845c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.228632 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 29f6c3cea4b14364b6d073a92171119d (DEFAULT_TABLE table=default.Table_c1886293aee349b2b9e7177c1cfd6d26 [id=0853ffc4d48549bc93e251a5dbc55901]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.229035 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet fbcf17a3bcc844b1b9676580f0749c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.231807 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe5e08ad6af2442bb0a8fcc80a99b532. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.229089 24179 ts_tablet_manager.cc:1909] T dcdd765895254a39b86b692b5d812c83 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:44.234925 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet cb7b37f776df47da9aa6e7fba82a8999
I20240502 08:16:44.235898 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 501) took 1145 ms. Trace:
I20240502 08:16:44.236006 24179 rpcz_store.cc:276] 0502 08:16:43.089245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.089533 (+ 288us) service_pool.cc:225] Handling call
0502 08:16:44.000714 (+911181us) ts_tablet_manager.cc:1120] Deleting tablet dcdd765895254a39b86b692b5d812c83
0502 08:16:44.000721 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.111330 (+110609us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:44.185307 (+ 73977us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:44.234983 (+ 49676us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":909986}
W20240502 08:16:44.236959 23723 leader_election.cc:341] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:44.237632 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c95dfda9d3b4e2ca9397b77ad71c011" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:44.235707 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 3777d2e9d4464552ad1c4c8c0d3e9888 (DEFAULT_TABLE table=default.Table_53c08a7447a44323883744c033efcbfb [id=78b34e35d763495aadf6a9eb242741c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.237841 23786 raft_consensus.cc:2454] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:44.239786 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 3777d2e9d4464552ad1c4c8c0d3e9888 (DEFAULT_TABLE table=default.Table_53c08a7447a44323883744c033efcbfb [id=78b34e35d763495aadf6a9eb242741c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.240281 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f6c3cea4b14364b6d073a92171119d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.241292 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_08a0e708a9ad4c619295ddc6c62e1a83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:44.241899 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08a0e708a9ad4c619295ddc6c62e1a83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.246361 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3777d2e9d4464552ad1c4c8c0d3e9888. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.247726 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3777d2e9d4464552ad1c4c8c0d3e9888. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.248006 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ce5ccfd29724dc49afd0892b99f3169 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.249239 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 303a0a3748ee422faab856e1c4bff66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.250864 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7e2da9cff594d90819f6b216dbed552. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.256402 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7b37f776df47da9aa6e7fba82a8999 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.256559 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 49b816ea1d474f2db94119441146ba78 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.257807 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe5e08ad6af2442bb0a8fcc80a99b532. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.258697 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725982e4712e442d8ff1fd3f796e18f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.259585 24179 tablet_replica.cc:330] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:44.273131 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0af5f4daf3149738647a08219efb245" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:44.277381 23723 leader_election.cc:302] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:44.277245 24179 raft_consensus.cc:2229] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:44.277930 24179 raft_consensus.cc:2258] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:44.277992 24779 raft_consensus.cc:2781] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:44.278575 24179 ts_tablet_manager.cc:1875] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:44.279611 23858 leader_election.cc:302] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:44.279727 24779 raft_consensus.cc:686] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:44.280099 24779 consensus_queue.cc:237] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:44.296954 24771 tablet.cc:1722] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240502 08:16:44.305611 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet dcdd765895254a39b86b692b5d812c83 (table default.Table_dd3c838f4a014e178351d135f5d4567d [id=6be0b9c539dd40059dab4bae00b7b180]) successfully deleted
I20240502 08:16:44.307619 24180 ts_tablet_manager.cc:1888] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:44.307946 24180 log.cc:1196] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/f36aadfa1d264f23bcafa91744aed36d
I20240502 08:16:44.322118 24695 raft_consensus.cc:2781] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:44.322263 24695 raft_consensus.cc:482] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:44.322638 24695 raft_consensus.cc:3037] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:44.334666 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ce5ccfd29724dc49afd0892b99f3169 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.335224 23903 tablet_service.cc:1514] Processing DeleteTablet for tablet fbcf17a3bcc844b1b9676580f0749c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.335662 24175 tablet_bootstrap.cc:492] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:44.335867 24175 ts_tablet_manager.cc:1367] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.351s user 0.010s sys 0.018s
I20240502 08:16:44.336714 24175 raft_consensus.cc:348] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:44.336953 24175 raft_consensus.cc:374] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.337028 24175 raft_consensus.cc:729] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.337339 24175 consensus_queue.cc:260] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
W20240502 08:16:44.313087 23723 leader_election.cc:341] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:44.342247 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7b37f776df47da9aa6e7fba82a8999 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.342862 23921 raft_consensus.cc:3037] T 49b816ea1d474f2db94119441146ba78 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:44.326375 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 48ae3bad6d3c48a6a067f16be24e224c (DEFAULT_TABLE table=default.Table_8040b4c1a36844658c4cdf62e9ae61de [id=105a31efe9a1492e823d95bfd38a94ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.327149 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 5704dff6114142d3b27c365ab99e2277 (DEFAULT_TABLE table=default.Table_31d76df4cb2f4950970e912e0a8797b9 [id=a8824e6224bc446b8c23af04afa99261]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.343534 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5704dff6114142d3b27c365ab99e2277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.328605 23765 tablet_service.cc:1467] Processing CreateTablet for tablet be4981ea1104472892be73c2e9f47a83 (DEFAULT_TABLE table=default.Table_05b41a1454b94a0eaca01c06499ff3b2 [id=f3f2972741104f67866e961e1c1a3f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.346009 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be4981ea1104472892be73c2e9f47a83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.347520 23903 tablet_service.cc:1514] Processing DeleteTablet for tablet 49b816ea1d474f2db94119441146ba78 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.359123 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48ae3bad6d3c48a6a067f16be24e224c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.363637 24695 raft_consensus.cc:504] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:44.364358 24695 leader_election.cc:288] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
W20240502 08:16:44.365543 23861 leader_election.cc:341] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 5c95dfda9d3b4e2ca9397b77ad71c011
I20240502 08:16:44.332649 23764 tablet_service.cc:1467] Processing CreateTablet for tablet fb6ecfcde4d14a74b47b8b23404cdd10 (DEFAULT_TABLE table=default.Table_a22cf11b49c04125a8e6c9b2942634a6 [id=d6f58fe0645e4f16b8cbb914a8408e6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.372480 24021 tablet_service.cc:1514] Processing DeleteTablet for tablet fbcf17a3bcc844b1b9676580f0749c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:44.373519 23723 leader_election.cc:341] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: e0af5f4daf3149738647a08219efb245
I20240502 08:16:44.379680 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6ecfcde4d14a74b47b8b23404cdd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.381376 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 92b830e128ac439eba286f29dbb20ee4 (DEFAULT_TABLE table=default.Table_cf112903d1344a4484d86fb9785c0f17 [id=e5b9ab35a16943af94e5108dc0d29426]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:44.306389 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1be95e0576c143e5a94c7aaf8d3e0625 consensus state has no leader
I20240502 08:16:44.382388 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7f8b1dce548042b7886ff9f5e150c3f1" } modify_external_catalogs: true
I20240502 08:16:44.382700 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7f8b1dce548042b7886ff9f5e150c3f1" } modify_external_catalogs: true
I20240502 08:16:44.315217 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_19538f34f25a4fd79dd6f076de5c021e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:44.315586 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_44bdfbf3075343e0a2ea5b79c0ca93e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:44.383608 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44bdfbf3075343e0a2ea5b79c0ca93e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:44.384744 23597 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 79185.2us
I20240502 08:16:44.343552 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8ac26bacf7bc4a05bacf7f35c2755a74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:44.348477 23607 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_5398d425143843de8fe5c9594fcf8503 [id=ba62a673079d424992daa0f6b0c5d6b9] alter complete (version 4)
I20240502 08:16:44.359797 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_663684b67f204c6e82f363ea57b73fab" } new_table_name: "default.Table_a86b6b9ac2d8498c97341dc60ca5ee34" modify_external_catalogs: true
I20240502 08:16:44.364941 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 29144c1d42f0408f9b2987c9c66ed8dc
I20240502 08:16:44.368866 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_646fc9f7ffec46cf999a349a70c20fe7" } new_table_name: "default.Table_96165587b33b444c8ec8a3b2df7a77d1" modify_external_catalogs: true
W20240502 08:16:44.385514 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ac26bacf7bc4a05bacf7f35c2755a74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:44.383610 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19538f34f25a4fd79dd6f076de5c021e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.323751 23604 catalog_manager.cc:5484] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:44.390450 23604 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 7a19310948e248d1a85479c41c61bda2 (table default.Table_785d5ddac96746379d1117bc8379566b [id=8521bfc13ef040a78cfb2cc4501b7f04]). Expected version 4 got 0
I20240502 08:16:44.390847 23604 catalog_manager.cc:5484] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:44.391290 23604 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 36f2d7b6f4e442ae95b2d1002d571b6a (table default.Table_e7dcb7ee94304b10b88d59b0872ae586 [id=a95a99a58cde44fe814b3edfcc643a80]). Expected version 2 got 0
I20240502 08:16:44.392633 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e8bb862a5508434a83ad4f5c8be82002" } new_table_name: "default.Table_16cc91d0fe274b37a9114f57e8d1cc8c" modify_external_catalogs: true
I20240502 08:16:44.393806 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7db6534bd8144b208a6eaa13f381cdfd" } modify_external_catalogs: true
I20240502 08:16:44.393996 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7db6534bd8144b208a6eaa13f381cdfd" } modify_external_catalogs: true
I20240502 08:16:44.404093 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet c08228fcd1304528b9446003851f6c70 with delete_type TABLET_DATA_DELETED (replaced by tablet b59c2f93d6514c9cbf358ab96f1a06b5) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.404548 24181 tablet_replica.cc:330] T c08228fcd1304528b9446003851f6c70 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:44.404527 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3e9140f5996646cbb7cad1837c94d43e" } new_table_name: "default.Table_9f3ea2fdcdc2477b89d509e868ffe4de" modify_external_catalogs: true
I20240502 08:16:44.404889 24181 raft_consensus.cc:2229] T c08228fcd1304528b9446003851f6c70 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:44.405126 24181 raft_consensus.cc:2258] T c08228fcd1304528b9446003851f6c70 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:44.406061 24181 ts_tablet_manager.cc:1875] T c08228fcd1304528b9446003851f6c70 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:44.362860 24180 ts_tablet_manager.cc:1909] T f36aadfa1d264f23bcafa91744aed36d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:44.406073 23723 leader_election.cc:302] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:44.406255 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 890fd44075ad4b5699f2fcf0a1ab7804 (DEFAULT_TABLE table=default.Table_d0fd869295b5475c902bde5dd5ee95d3 [id=7a2e8a5daae14e259454e99fee9d1dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.331717 23762 tablet_service.cc:1467] Processing CreateTablet for tablet a059351d148e4376adfbe7128c88c351 (DEFAULT_TABLE table=default.Table_18f36b7d14194553a25aad1cc668845a [id=15c77ab4af364973bac5f7f6a3c78b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.410189 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ce5ccfd29724dc49afd0892b99f3169 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.361730 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 793d364d9dec429484660a86036f5fb9 (DEFAULT_TABLE table=default.Table_990c64d081c148d79dae73c2d06b1f11 [id=bc79353ff53247dcb3a7117c7bdfa749]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.411271 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 49b816ea1d474f2db94119441146ba78 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.411991 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7b37f776df47da9aa6e7fba82a8999 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.412957 24181 ts_tablet_manager.cc:1888] T c08228fcd1304528b9446003851f6c70 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:44.413126 24181 log.cc:1196] T c08228fcd1304528b9446003851f6c70 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/c08228fcd1304528b9446003851f6c70
I20240502 08:16:44.413280 24025 tablet_service.cc:1467] Processing CreateTablet for tablet fb6ecfcde4d14a74b47b8b23404cdd10 (DEFAULT_TABLE table=default.Table_a22cf11b49c04125a8e6c9b2942634a6 [id=d6f58fe0645e4f16b8cbb914a8408e6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.413785 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6ecfcde4d14a74b47b8b23404cdd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.413895 24181 ts_tablet_manager.cc:1909] T c08228fcd1304528b9446003851f6c70 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:44.415050 24181 tablet_replica.cc:330] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:44.415365 24181 raft_consensus.cc:2229] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:44.415675 24181 raft_consensus.cc:2258] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:44.416353 24181 ts_tablet_manager.cc:1875] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:44.420786 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 48ae3bad6d3c48a6a067f16be24e224c (DEFAULT_TABLE table=default.Table_8040b4c1a36844658c4cdf62e9ae61de [id=105a31efe9a1492e823d95bfd38a94ad]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:44.420979 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2da8b9d3f374d8a812336057c0ce7d2 consensus state has no leader
I20240502 08:16:44.421387 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48ae3bad6d3c48a6a067f16be24e224c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.425046 24021 tablet_service.cc:1467] Processing CreateTablet for tablet 5704dff6114142d3b27c365ab99e2277 (DEFAULT_TABLE table=default.Table_31d76df4cb2f4950970e912e0a8797b9 [id=a8824e6224bc446b8c23af04afa99261]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.425132 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b3352fb8a27f46c78a9919a2de2510c2" } modify_external_catalogs: true
I20240502 08:16:44.425338 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b3352fb8a27f46c78a9919a2de2510c2" } modify_external_catalogs: true
I20240502 08:16:44.428048 24780 raft_consensus.cc:2726] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:44.435580 24175 ts_tablet_manager.cc:1398] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.100s user 0.006s sys 0.001s
I20240502 08:16:44.436105 24175 tablet_bootstrap.cc:492] T 36f2d7b6f4e442ae95b2d1002d571b6a P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:44.437844 24175 tablet_bootstrap.cc:654] T 36f2d7b6f4e442ae95b2d1002d571b6a P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.439680 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d0fd869295b5475c902bde5dd5ee95d3" } modify_external_catalogs: true
I20240502 08:16:44.439894 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d0fd869295b5475c902bde5dd5ee95d3" } modify_external_catalogs: true
I20240502 08:16:44.449438 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 29144c1d42f0408f9b2987c9c66ed8dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.449916 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_28e9d1b5045f4818a01c228a3d25994d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:44.450125 23753 tablet_service.cc:1467] Processing CreateTablet for tablet ddcb9f236489402593d59ba801615103 (DEFAULT_TABLE table=default.Table_3e3f39f737b84f009fc0e2b1b1da6468 [id=f514e0944fa548849851b5b03ed8f5a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.450320 23896 tablet_service.cc:1467] Processing CreateTablet for tablet fb6ecfcde4d14a74b47b8b23404cdd10 (DEFAULT_TABLE table=default.Table_a22cf11b49c04125a8e6c9b2942634a6 [id=d6f58fe0645e4f16b8cbb914a8408e6e]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:44.450505 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28e9d1b5045f4818a01c228a3d25994d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.450879 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6ecfcde4d14a74b47b8b23404cdd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.456579 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b3315254654d4234a879012f47feeb19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:44.457123 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3315254654d4234a879012f47feeb19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.459756 23903 tablet_service.cc:1467] Processing CreateTablet for tablet be4981ea1104472892be73c2e9f47a83 (DEFAULT_TABLE table=default.Table_05b41a1454b94a0eaca01c06499ff3b2 [id=f3f2972741104f67866e961e1c1a3f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.460301 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be4981ea1104472892be73c2e9f47a83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.467320 24175 tablet_bootstrap.cc:492] T 36f2d7b6f4e442ae95b2d1002d571b6a P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:44.467525 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 48ae3bad6d3c48a6a067f16be24e224c (DEFAULT_TABLE table=default.Table_8040b4c1a36844658c4cdf62e9ae61de [id=105a31efe9a1492e823d95bfd38a94ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.468057 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48ae3bad6d3c48a6a067f16be24e224c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.467603 24175 ts_tablet_manager.cc:1367] T 36f2d7b6f4e442ae95b2d1002d571b6a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240502 08:16:44.469486 24175 raft_consensus.cc:348] T 36f2d7b6f4e442ae95b2d1002d571b6a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:44.469822 24175 raft_consensus.cc:374] T 36f2d7b6f4e442ae95b2d1002d571b6a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.469974 24175 raft_consensus.cc:729] T 36f2d7b6f4e442ae95b2d1002d571b6a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.470297 24175 consensus_queue.cc:260] T 36f2d7b6f4e442ae95b2d1002d571b6a P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:44.470911 24175 ts_tablet_manager.cc:1398] T 36f2d7b6f4e442ae95b2d1002d571b6a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:44.471314 24175 tablet_bootstrap.cc:492] T b844458ae73f413cbf2113027a5af359 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:44.474300 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 890fd44075ad4b5699f2fcf0a1ab7804 (DEFAULT_TABLE table=default.Table_d0fd869295b5475c902bde5dd5ee95d3 [id=7a2e8a5daae14e259454e99fee9d1dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.474731 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890fd44075ad4b5699f2fcf0a1ab7804. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.477372 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0f88db2e7c844740bcb3df0fb75508b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:44.477957 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f88db2e7c844740bcb3df0fb75508b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:44.479285 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e54b09eac68f423086620617dd4a6ffc consensus state has no leader
I20240502 08:16:44.479740 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7991f9ca2d6f4a79aa0aeb12b23f1000"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:44.480239 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7991f9ca2d6f4a79aa0aeb12b23f1000 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.481742 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 497) took 1901 ms. Trace:
I20240502 08:16:44.481773 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c95dfda9d3b4e2ca9397b77ad71c011" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:44.481933 23901 tablet_service.cc:1467] Processing CreateTablet for tablet a059351d148e4376adfbe7128c88c351 (DEFAULT_TABLE table=default.Table_18f36b7d14194553a25aad1cc668845a [id=15c77ab4af364973bac5f7f6a3c78b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.482072 23786 raft_consensus.cc:3037] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:44.481904 24180 rpcz_store.cc:276] 0502 08:16:42.528724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.549093 (+ 20369us) service_pool.cc:225] Handling call
0502 08:16:43.905685 (+1356592us) ts_tablet_manager.cc:1120] Deleting tablet f36aadfa1d264f23bcafa91744aed36d
0502 08:16:43.905698 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.189413 (+283715us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:44.279591 (+ 90178us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:44.430438 (+150847us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":68101,"tablet-delete.queue_time_us":1355720}
I20240502 08:16:44.482339 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a059351d148e4376adfbe7128c88c351. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.482687 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 506) took 1879 ms. Trace:
I20240502 08:16:44.482903 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet e0af5f4daf3149738647a08219efb245 with delete_type TABLET_DATA_DELETED (replaced by tablet b8b1fa17dad143f0a7f1751e651eb81a) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.483145 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 843edd7092464981adad317b48871645
I20240502 08:16:44.484266 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet f36aadfa1d264f23bcafa91744aed36d (table default.Table_1136fadacc774d899c0461e8a0579376 [id=e18870bff56747b192b4698f1ca3a8ba]) successfully deleted
I20240502 08:16:44.485963 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8eddc9a260a04c339e7bb130c9822c62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:44.486467 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8eddc9a260a04c339e7bb130c9822c62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.486516 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a059351d148e4376adfbe7128c88c351. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.488371 24175 tablet_bootstrap.cc:654] T b844458ae73f413cbf2113027a5af359 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.445855 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 058e13ba3a474af3aabebc93c5a88124 (DEFAULT_TABLE table=default.Table_b3278c5d6d0b4593ac33bc9977f5c3b1 [id=e86031550cec46a793785c0b3859fa92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.482826 24180 rpcz_store.cc:276] 0502 08:16:42.603564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.603735 (+ 171us) service_pool.cc:225] Handling call
0502 08:16:44.482361 (+1878626us) ts_tablet_manager.cc:1120] Deleting tablet 02e6a1ba23dd4f82bc721a60c0526871
0502 08:16:44.482376 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.482604 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1878357}
I20240502 08:16:44.489845 24174 tablet_bootstrap.cc:492] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
W20240502 08:16:44.489877 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 02e6a1ba23dd4f82bc721a60c0526871 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02e6a1ba23dd4f82bc721a60c0526871 already in progress: creating tablet
I20240502 08:16:44.490034 24174 ts_tablet_manager.cc:1367] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.506s user 0.022s sys 0.008s
I20240502 08:16:44.492689 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5704dff6114142d3b27c365ab99e2277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.488417 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 058e13ba3a474af3aabebc93c5a88124 (DEFAULT_TABLE table=default.Table_b3278c5d6d0b4593ac33bc9977f5c3b1 [id=e86031550cec46a793785c0b3859fa92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.493259 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 058e13ba3a474af3aabebc93c5a88124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.493593 24771 tablet.cc:1722] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:44.495067 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 843edd7092464981adad317b48871645 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.493613 24174 raft_consensus.cc:348] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:44.495564 24174 raft_consensus.cc:374] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.497092 24174 raft_consensus.cc:729] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.497373 24174 consensus_queue.cc:260] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:44.488916 23893 tablet_service.cc:1467] Processing CreateTablet for tablet ddcb9f236489402593d59ba801615103 (DEFAULT_TABLE table=default.Table_3e3f39f737b84f009fc0e2b1b1da6468 [id=f514e0944fa548849851b5b03ed8f5a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.498122 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddcb9f236489402593d59ba801615103. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.498678 24181 ts_tablet_manager.cc:1888] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:16:44.499001 24181 log.cc:1196] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/1ce5ccfd29724dc49afd0892b99f3169
I20240502 08:16:44.489264 23886 tablet_service.cc:1514] Processing DeleteTablet for tablet 29144c1d42f0408f9b2987c9c66ed8dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.504822 24181 ts_tablet_manager.cc:1909] T 1ce5ccfd29724dc49afd0892b99f3169 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:44.491053 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 92b830e128ac439eba286f29dbb20ee4 (DEFAULT_TABLE table=default.Table_cf112903d1344a4484d86fb9785c0f17 [id=e5b9ab35a16943af94e5108dc0d29426]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.491483 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 5704dff6114142d3b27c365ab99e2277 (DEFAULT_TABLE table=default.Table_31d76df4cb2f4950970e912e0a8797b9 [id=a8824e6224bc446b8c23af04afa99261]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.492444 24795 tablet.cc:1722] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240502 08:16:44.506438 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b830e128ac439eba286f29dbb20ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.506697 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5704dff6114142d3b27c365ab99e2277. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.508776 24181 tablet_replica.cc:330] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:44.509732 24181 raft_consensus.cc:2229] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:44.510337 24181 raft_consensus.cc:2258] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:44.513159 24173 tablet_bootstrap.cc:492] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:44.513332 24173 ts_tablet_manager.cc:1367] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.354s user 0.030s sys 0.070s
I20240502 08:16:44.514243 24173 raft_consensus.cc:348] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:44.514489 24173 raft_consensus.cc:374] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.514603 24173 raft_consensus.cc:729] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.514798 24181 ts_tablet_manager.cc:1875] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:44.514917 24173 consensus_queue.cc:260] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:44.515597 24173 ts_tablet_manager.cc:1398] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:44.516001 24173 tablet_bootstrap.cc:492] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:44.518739 24173 tablet_bootstrap.cc:654] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.520351 24773 tablet.cc:1722] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:44.522460 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 508) took 1763 ms. Trace:
I20240502 08:16:44.522593 24180 rpcz_store.cc:276] 0502 08:16:42.757736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.757824 (+ 88us) service_pool.cc:225] Handling call
0502 08:16:44.489058 (+1731234us) ts_tablet_manager.cc:1120] Deleting tablet 0f306a288b90413b9d8ea8b76d23363a
0502 08:16:44.504759 (+ 15701us) spinlock_profiling.cc:243] Waited 4.33 ms on lock 0x61800000fd18. stack: 00007f1068831756 00007f1068831510 00007f106786448d 0000000000955804 00000000009557cd 000000000095519a 00007f106882719d 00007f106882820e 00007f107bd2d4ed 00007f107bca96b3 00007f1077e0ab06 00007f1077e0be07 00007f1077e3e8d4 00007f1077e1fbc0 00007f1077e1f94d 0000000000a60bcc
0502 08:16:44.513306 (+ 8547us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.520897 (+ 7591us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9959808,"tablet-delete.queue_time_us":1730966}
W20240502 08:16:44.523566 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 0f306a288b90413b9d8ea8b76d23363a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f306a288b90413b9d8ea8b76d23363a already in progress: creating tablet
I20240502 08:16:44.523856 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b830e128ac439eba286f29dbb20ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.525239 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 507) took 1852 ms. Trace:
I20240502 08:16:44.525364 24180 rpcz_store.cc:276] 0502 08:16:42.670513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.679050 (+ 8537us) service_pool.cc:225] Handling call
0502 08:16:44.522965 (+1843915us) ts_tablet_manager.cc:1120] Deleting tablet 45f02efa659e457ab57bc32ed3f66bb9
0502 08:16:44.522981 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.523121 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1732214}
W20240502 08:16:44.526546 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 45f02efa659e457ab57bc32ed3f66bb9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45f02efa659e457ab57bc32ed3f66bb9 already in progress: creating tablet
I20240502 08:16:44.526774 24025 tablet_service.cc:1467] Processing CreateTablet for tablet be4981ea1104472892be73c2e9f47a83 (DEFAULT_TABLE table=default.Table_05b41a1454b94a0eaca01c06499ff3b2 [id=f3f2972741104f67866e961e1c1a3f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.527320 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be4981ea1104472892be73c2e9f47a83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.530714 24179 ts_tablet_manager.cc:1888] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:44.530892 24179 log.cc:1196] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/0be4f8d5cc09483d8d1aa1e37679b544
I20240502 08:16:44.531293 24179 ts_tablet_manager.cc:1909] T 0be4f8d5cc09483d8d1aa1e37679b544 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:44.531148 24021 tablet_service.cc:1467] Processing CreateTablet for tablet a059351d148e4376adfbe7128c88c351 (DEFAULT_TABLE table=default.Table_18f36b7d14194553a25aad1cc668845a [id=15c77ab4af364973bac5f7f6a3c78b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.531692 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 500) took 1443 ms. Trace:
I20240502 08:16:44.531826 24179 rpcz_store.cc:276] 0502 08:16:43.088041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.088304 (+ 263us) service_pool.cc:225] Handling call
0502 08:16:44.236320 (+1148016us) ts_tablet_manager.cc:1120] Deleting tablet 0be4f8d5cc09483d8d1aa1e37679b544
0502 08:16:44.236328 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.388347 (+152019us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:44.530547 (+142200us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:44.531643 (+ 1096us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1136739}
I20240502 08:16:44.532295 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 510) took 1389 ms. Trace:
I20240502 08:16:44.532420 24179 rpcz_store.cc:276] 0502 08:16:43.142562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.142627 (+ 65us) service_pool.cc:225] Handling call
0502 08:16:44.532127 (+1389500us) ts_tablet_manager.cc:1120] Deleting tablet 89f57e3a39d54b7ab372d4b145b7a696
0502 08:16:44.532137 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.532257 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1387032}
I20240502 08:16:44.532801 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 511) took 1388 ms. Trace:
I20240502 08:16:44.532837 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a059351d148e4376adfbe7128c88c351. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.532924 24179 rpcz_store.cc:276] 0502 08:16:43.144160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.144487 (+ 327us) service_pool.cc:225] Handling call
0502 08:16:44.532672 (+1388185us) ts_tablet_manager.cc:1120] Deleting tablet df08f0d3af4c4c7e8abce656714b1b7f
0502 08:16:44.532680 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.532761 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1380031}
I20240502 08:16:44.533270 24179 tablet_replica.cc:330] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:44.533500 24179 raft_consensus.cc:2229] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:44.534198 24179 raft_consensus.cc:2258] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:44.534905 24179 ts_tablet_manager.cc:1875] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:44.533872 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 793d364d9dec429484660a86036f5fb9 (DEFAULT_TABLE table=default.Table_990c64d081c148d79dae73c2d06b1f11 [id=bc79353ff53247dcb3a7117c7bdfa749]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.534991 24174 ts_tablet_manager.cc:1398] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.030s user 0.003s sys 0.000s
I20240502 08:16:44.535392 24174 tablet_bootstrap.cc:492] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:44.535418 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 793d364d9dec429484660a86036f5fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.536644 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 793d364d9dec429484660a86036f5fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.537307 24180 tablet_replica.cc:330] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:44.537546 24180 raft_consensus.cc:2229] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:44.542106 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 793d364d9dec429484660a86036f5fb9 (DEFAULT_TABLE table=default.Table_990c64d081c148d79dae73c2d06b1f11 [id=bc79353ff53247dcb3a7117c7bdfa749]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.542533 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 793d364d9dec429484660a86036f5fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.546484 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddcb9f236489402593d59ba801615103. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.548215 24179 ts_tablet_manager.cc:1888] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:44.548373 24179 log.cc:1196] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/b6417aafbdcf416ea7bff56bce5e6ddb
I20240502 08:16:44.548799 24179 ts_tablet_manager.cc:1909] T b6417aafbdcf416ea7bff56bce5e6ddb P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:44.549129 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 506) took 1420 ms. Trace:
I20240502 08:16:44.549230 24179 rpcz_store.cc:276] 0502 08:16:43.128751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.128841 (+ 90us) service_pool.cc:225] Handling call
0502 08:16:44.533195 (+1404354us) ts_tablet_manager.cc:1120] Deleting tablet b6417aafbdcf416ea7bff56bce5e6ddb
0502 08:16:44.533205 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.545130 (+ 11925us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:44.548113 (+ 2983us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:44.549070 (+ 957us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1380364}
I20240502 08:16:44.549717 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 515) took 1380 ms. Trace:
I20240502 08:16:44.549885 24179 rpcz_store.cc:276] 0502 08:16:43.169128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.176208 (+ 7080us) service_pool.cc:225] Handling call
0502 08:16:44.549511 (+1373303us) ts_tablet_manager.cc:1120] Deleting tablet 29f82ad5860843f1adfbb40cdb21a4a3
0502 08:16:44.549521 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.549667 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1346899}
I20240502 08:16:44.550029 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 058e13ba3a474af3aabebc93c5a88124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.550220 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 517) took 1154 ms. Trace:
I20240502 08:16:44.550292 24179 rpcz_store.cc:276] 0502 08:16:43.396041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.396094 (+ 53us) service_pool.cc:225] Handling call
0502 08:16:44.550081 (+1153987us) ts_tablet_manager.cc:1120] Deleting tablet 63e645bb6e2948e091e08433944a52f4
0502 08:16:44.550089 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.550185 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1151340}
I20240502 08:16:44.550534 24179 tablet_replica.cc:330] T fd57b0357ca749c389bc61add20cd3e7 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:44.550690 24179 raft_consensus.cc:2229] T fd57b0357ca749c389bc61add20cd3e7 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:44.550841 24179 raft_consensus.cc:2258] T fd57b0357ca749c389bc61add20cd3e7 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:44.551326 24179 ts_tablet_manager.cc:1875] T fd57b0357ca749c389bc61add20cd3e7 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:44.552488 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 890fd44075ad4b5699f2fcf0a1ab7804 (DEFAULT_TABLE table=default.Table_d0fd869295b5475c902bde5dd5ee95d3 [id=7a2e8a5daae14e259454e99fee9d1dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.553264 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890fd44075ad4b5699f2fcf0a1ab7804. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.556923 23786 raft_consensus.cc:2454] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:44.557193 24179 ts_tablet_manager.cc:1888] T fd57b0357ca749c389bc61add20cd3e7 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:44.557297 24179 log.cc:1196] T fd57b0357ca749c389bc61add20cd3e7 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/fd57b0357ca749c389bc61add20cd3e7
I20240502 08:16:44.557667 24179 ts_tablet_manager.cc:1909] T fd57b0357ca749c389bc61add20cd3e7 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:44.558120 24179 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 516) took 1164 ms. Trace:
I20240502 08:16:44.558249 24179 rpcz_store.cc:276] 0502 08:16:43.394109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.394264 (+ 155us) service_pool.cc:225] Handling call
0502 08:16:44.550460 (+1156196us) ts_tablet_manager.cc:1120] Deleting tablet fd57b0357ca749c389bc61add20cd3e7
0502 08:16:44.550466 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.554166 (+ 3700us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:44.557078 (+ 2912us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:44.558073 (+ 995us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1149700}
I20240502 08:16:44.558336 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 058e13ba3a474af3aabebc93c5a88124 (DEFAULT_TABLE table=default.Table_b3278c5d6d0b4593ac33bc9977f5c3b1 [id=e86031550cec46a793785c0b3859fa92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.558799 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 058e13ba3a474af3aabebc93c5a88124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.559007 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890fd44075ad4b5699f2fcf0a1ab7804. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.559116 24179 tablet_replica.cc:330] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:44.559311 24179 raft_consensus.cc:2229] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:44.559540 24179 raft_consensus.cc:2258] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:44.560118 24179 ts_tablet_manager.cc:1875] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:44.560320 24180 raft_consensus.cc:2258] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:44.565690 24180 ts_tablet_manager.cc:1875] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:44.565781 24039 tablet_service.cc:1467] Processing CreateTablet for tablet ddcb9f236489402593d59ba801615103 (DEFAULT_TABLE table=default.Table_3e3f39f737b84f009fc0e2b1b1da6468 [id=f514e0944fa548849851b5b03ed8f5a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.566462 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddcb9f236489402593d59ba801615103. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.566622 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 0be4f8d5cc09483d8d1aa1e37679b544 (table default.Table_178527d31d7044fc98fa41886d651049 [id=6dc275aed42b42118a70099d3eae3c21]) successfully deleted
W20240502 08:16:44.567085 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 89f57e3a39d54b7ab372d4b145b7a696 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89f57e3a39d54b7ab372d4b145b7a696 already in progress: creating tablet
W20240502 08:16:44.567523 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet df08f0d3af4c4c7e8abce656714b1b7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df08f0d3af4c4c7e8abce656714b1b7f already in progress: creating tablet
I20240502 08:16:44.569540 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet b6417aafbdcf416ea7bff56bce5e6ddb (table default.Table_9d372763eb03435f8b7074a802d86ffe [id=9ff8ea75a9bc4de194a70fe2088a94bc]) successfully deleted
W20240502 08:16:44.569947 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 29f82ad5860843f1adfbb40cdb21a4a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29f82ad5860843f1adfbb40cdb21a4a3 already in progress: creating tablet
W20240502 08:16:44.570255 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 63e645bb6e2948e091e08433944a52f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63e645bb6e2948e091e08433944a52f4 already in progress: creating tablet
I20240502 08:16:44.570569 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet fd57b0357ca749c389bc61add20cd3e7 (table default.Table_f52a4fa0e0c04f19a517f946b59a850f [id=fd141979a86c45bf9648396d086f80d4]) successfully deleted
W20240502 08:16:44.570881 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 643ce929e74c499da835fbe9b63b8880 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 643ce929e74c499da835fbe9b63b8880 already in progress: creating tablet
W20240502 08:16:44.571233 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet a198c33a3e314b5b81a08244862b281a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a198c33a3e314b5b81a08244862b281a already in progress: creating tablet
W20240502 08:16:44.571583 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 15007d576b064c2cb0a005cb9f7ea1cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15007d576b064c2cb0a005cb9f7ea1cd already in progress: creating tablet
I20240502 08:16:44.572335 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 29144c1d42f0408f9b2987c9c66ed8dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.576730 23858 leader_election.cc:302] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:44.577033 24790 raft_consensus.cc:2781] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:44.577148 24790 raft_consensus.cc:686] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:16:44.577431 24790 consensus_queue.cc:237] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:44.584434 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 92b830e128ac439eba286f29dbb20ee4 (DEFAULT_TABLE table=default.Table_cf112903d1344a4484d86fb9785c0f17 [id=e5b9ab35a16943af94e5108dc0d29426]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.588282 24181 ts_tablet_manager.cc:1888] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:44.588507 24181 log.cc:1196] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/49b816ea1d474f2db94119441146ba78
I20240502 08:16:44.588966 24181 ts_tablet_manager.cc:1909] T 49b816ea1d474f2db94119441146ba78 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:44.590668 24175 tablet_bootstrap.cc:492] T b844458ae73f413cbf2113027a5af359 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:44.590941 24175 ts_tablet_manager.cc:1367] T b844458ae73f413cbf2113027a5af359 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.120s user 0.006s sys 0.000s
I20240502 08:16:44.591849 24175 raft_consensus.cc:348] T b844458ae73f413cbf2113027a5af359 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:44.592181 24175 raft_consensus.cc:374] T b844458ae73f413cbf2113027a5af359 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.592314 24175 raft_consensus.cc:729] T b844458ae73f413cbf2113027a5af359 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.592617 24175 consensus_queue.cc:260] T b844458ae73f413cbf2113027a5af359 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:44.593016 24795 tablet.cc:1722] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240502 08:16:44.594247 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet fbcf17a3bcc844b1b9676580f0749c9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbcf17a3bcc844b1b9676580f0749c9c already in progress: creating tablet
I20240502 08:16:44.594704 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet c08228fcd1304528b9446003851f6c70 (table default.Table_bbf9d7694fe942de8aa8bd4ed61336a5 [id=f1d6eb661b0143ee82e797cde61391c0]) successfully deleted
I20240502 08:16:44.595057 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 843edd7092464981adad317b48871645 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.598532 24173 tablet_bootstrap.cc:492] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:44.593381 24175 ts_tablet_manager.cc:1398] T b844458ae73f413cbf2113027a5af359 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:44.598937 24173 ts_tablet_manager.cc:1367] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.005s
I20240502 08:16:44.599009 24175 tablet_bootstrap.cc:492] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:44.600284 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b830e128ac439eba286f29dbb20ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.600730 24175 tablet_bootstrap.cc:654] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.603915 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 843edd7092464981adad317b48871645 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.605070 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 1ce5ccfd29724dc49afd0892b99f3169 (table default.Table_efda0c8c5d1d4e138081bfe146a06fa4 [id=3a9d7e8fe2c84cb4942fee9053a012b0]) successfully deleted
I20240502 08:16:44.606542 24790 raft_consensus.cc:482] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:44.606763 24790 raft_consensus.cc:504] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:44.607582 24790 leader_election.cc:288] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:44.607771 24174 tablet_bootstrap.cc:654] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.607899 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a19310948e248d1a85479c41c61bda2" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:44.608203 24057 raft_consensus.cc:2379] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:16:44.608626 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 49b816ea1d474f2db94119441146ba78 (table default.Table_20df1b504848439c869ae3739ac54185 [id=a0606a4053d04dd3814f09419a4a70c2]) successfully deleted
W20240502 08:16:44.609169 23723 consensus_peers.cc:485] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dead13bba95f43cb809e17f6d7fba028. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:44.610342 24179 ts_tablet_manager.cc:1888] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:16:44.610505 24179 log.cc:1196] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/dead13bba95f43cb809e17f6d7fba028
I20240502 08:16:44.610895 24179 ts_tablet_manager.cc:1909] T dead13bba95f43cb809e17f6d7fba028 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:44.611030 23923 raft_consensus.cc:3037] T b844458ae73f413cbf2113027a5af359 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:44.611534 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet dead13bba95f43cb809e17f6d7fba028 (table default.Table_ef79f64b4af44b88a006a32b437c3207 [id=7409c7e1a3534932a787559a679e7098]) successfully deleted
W20240502 08:16:44.612205 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 0eb7f36f7c9b47ebb93f1d846fd7352c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0eb7f36f7c9b47ebb93f1d846fd7352c already in progress: creating tablet
W20240502 08:16:44.612864 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 523603a31b5b49e0a18a5a64f1357d64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 523603a31b5b49e0a18a5a64f1357d64 already in progress: creating tablet
I20240502 08:16:44.613333 24175 tablet_bootstrap.cc:492] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
W20240502 08:16:44.613438 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 9467974d67c64c7eb5e488ac51f1240d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9467974d67c64c7eb5e488ac51f1240d already in progress: creating tablet
I20240502 08:16:44.613494 24175 ts_tablet_manager.cc:1367] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.002s
W20240502 08:16:44.614068 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e726e0ef49b247bebe07a7e7066c1af6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e726e0ef49b247bebe07a7e7066c1af6 already in progress: creating tablet
I20240502 08:16:44.614423 24175 raft_consensus.cc:348] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:44.614681 24175 raft_consensus.cc:374] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.614804 24175 raft_consensus.cc:729] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
W20240502 08:16:44.614821 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet c1e746013fde4de4862f213792e235ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1e746013fde4de4862f213792e235ed already in progress: creating tablet
I20240502 08:16:44.615099 24175 consensus_queue.cc:260] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:44.615662 24175 ts_tablet_manager.cc:1398] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:44.615998 24175 tablet_bootstrap.cc:492] T d8f7b5f018d84680ac5533bf99102d31 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:44.617000 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e9171ebf74424c9aae5e1c8a5118f768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9171ebf74424c9aae5e1c8a5118f768 already in progress: creating tablet
W20240502 08:16:44.617702 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet b6d8410ba7cc4b03bb64a138b1f9bb14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6d8410ba7cc4b03bb64a138b1f9bb14 already in progress: creating tablet
I20240502 08:16:44.617848 24175 tablet_bootstrap.cc:654] T d8f7b5f018d84680ac5533bf99102d31 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:44.618393 23723 consensus_peers.cc:485] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49b816ea1d474f2db94119441146ba78. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:16:44.618799 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet fddac683eda14ec1a0140926380f2c07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fddac683eda14ec1a0140926380f2c07 already in progress: creating tablet
W20240502 08:16:44.619469 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet a5a5a410b6314818b10927e35032f95d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5a5a410b6314818b10927e35032f95d already in progress: creating tablet
I20240502 08:16:44.619923 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a19310948e248d1a85479c41c61bda2" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:44.620270 24179 tablet_replica.cc:330] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:44.620508 23858 leader_election.cc:302] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:44.620792 24790 raft_consensus.cc:3037] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:44.623689 24790 raft_consensus.cc:2726] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:44.624343 24179 raft_consensus.cc:2229] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:44.624619 24179 pending_rounds.cc:62] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b: Trying to abort 1 pending ops.
I20240502 08:16:44.624715 24179 pending_rounds.cc:68] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156447552065536 op_type: NO_OP noop_request { }
I20240502 08:16:44.624861 24179 raft_consensus.cc:2866] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:44.624984 24179 raft_consensus.cc:2258] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:44.625169 23434 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 56755.5us
I20240502 08:16:44.625963 24179 ts_tablet_manager.cc:1875] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:44.633056 24179 ts_tablet_manager.cc:1888] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:44.633214 24179 log.cc:1196] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/5c95dfda9d3b4e2ca9397b77ad71c011
I20240502 08:16:44.633659 24179 ts_tablet_manager.cc:1909] T 5c95dfda9d3b4e2ca9397b77ad71c011 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:44.634313 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 5c95dfda9d3b4e2ca9397b77ad71c011 (table default.Table_c201704a319a4e078054899061f76615 [id=71234c2c912b4a79b7b08d87cab6fad0]) successfully deleted
W20240502 08:16:44.635147 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 9b780a7870044d94bab14f0d9c978a0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b780a7870044d94bab14f0d9c978a0c already in progress: creating tablet
W20240502 08:16:44.636644 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 4645143a00d440a7b6d56c5e30d249f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4645143a00d440a7b6d56c5e30d249f1 already in progress: creating tablet
I20240502 08:16:44.637019 24179 tablet_replica.cc:330] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:44.637275 24179 raft_consensus.cc:2229] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:44.637555 24179 raft_consensus.cc:2258] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:44.638196 24179 ts_tablet_manager.cc:1875] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:44.599885 24173 raft_consensus.cc:348] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:44.639361 24173 raft_consensus.cc:374] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.639501 24173 raft_consensus.cc:729] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.639961 24173 consensus_queue.cc:260] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:44.645965 24179 ts_tablet_manager.cc:1888] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:16:44.646165 24179 log.cc:1196] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/1ce5ccfd29724dc49afd0892b99f3169
W20240502 08:16:44.646211 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet cb7b37f776df47da9aa6e7fba82a8999 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb7b37f776df47da9aa6e7fba82a8999 already in progress: creating tablet
I20240502 08:16:44.646697 24179 ts_tablet_manager.cc:1909] T 1ce5ccfd29724dc49afd0892b99f3169 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
W20240502 08:16:44.646929 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 29144c1d42f0408f9b2987c9c66ed8dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29144c1d42f0408f9b2987c9c66ed8dc already in progress: creating tablet
I20240502 08:16:44.647359 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 1ce5ccfd29724dc49afd0892b99f3169 (table default.Table_efda0c8c5d1d4e138081bfe146a06fa4 [id=3a9d7e8fe2c84cb4942fee9053a012b0]) successfully deleted
W20240502 08:16:44.647636 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 843edd7092464981adad317b48871645 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 843edd7092464981adad317b48871645 already in progress: creating tablet
W20240502 08:16:44.648418 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet fbcf17a3bcc844b1b9676580f0749c9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbcf17a3bcc844b1b9676580f0749c9c already in progress: creating tablet
W20240502 08:16:44.649072 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet cb7b37f776df47da9aa6e7fba82a8999 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb7b37f776df47da9aa6e7fba82a8999 already in progress: creating tablet
I20240502 08:16:44.649466 24179 tablet_replica.cc:330] T 49b816ea1d474f2db94119441146ba78 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:44.649907 24179 raft_consensus.cc:2229] T 49b816ea1d474f2db94119441146ba78 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:44.651016 24179 raft_consensus.cc:2258] T 49b816ea1d474f2db94119441146ba78 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:44.651180 24771 tablet.cc:1722] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:44.652253 24179 ts_tablet_manager.cc:1875] T 49b816ea1d474f2db94119441146ba78 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:44.654479 24180 ts_tablet_manager.cc:1888] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:16:44.654623 24180 log.cc:1196] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/b6c52f1cae3a4ad38a9f95bfaf055e76
I20240502 08:16:44.656194 24180 ts_tablet_manager.cc:1909] T b6c52f1cae3a4ad38a9f95bfaf055e76 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:44.657106 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 509) took 1745 ms. Trace:
I20240502 08:16:44.657234 24180 rpcz_store.cc:276] 0502 08:16:42.912043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.912704 (+ 661us) service_pool.cc:225] Handling call
0502 08:16:44.526012 (+1613308us) ts_tablet_manager.cc:1120] Deleting tablet b6c52f1cae3a4ad38a9f95bfaf055e76
0502 08:16:44.526028 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.600295 (+ 74267us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:44.654317 (+ 54022us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:44.657062 (+ 2745us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1613092}
I20240502 08:16:44.657774 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet b6c52f1cae3a4ad38a9f95bfaf055e76 (table default.Table_35b808b0487f463dbcfa4a773a2a20f0 [id=922fb979b5894023a68820aece0b8c5e]) successfully deleted
I20240502 08:16:44.659296 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 512) took 1746 ms. Trace:
I20240502 08:16:44.661147 24173 ts_tablet_manager.cc:1398] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.062s user 0.001s sys 0.001s
W20240502 08:16:44.664577 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 63afd6e4fee24edeaba1209105f1c3e6 consensus state has no leader
I20240502 08:16:44.670784 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c2b1aa8d133542ffb45da2eacecf427b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:44.671541 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2b1aa8d133542ffb45da2eacecf427b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.674225 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 890fd44075ad4b5699f2fcf0a1ab7804
I20240502 08:16:44.680773 24173 tablet_bootstrap.cc:492] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:44.682660 23597 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 20a0519367ad4d6fae687aaeef0a8f44 (table default.Table_00e3ffabca274246a81fb18ce0d772a4 [id=80a6ac43017e433aa8cefdc6d3a39a44], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 20a0519367ad4d6fae687aaeef0a8f44 consensus state has no leader
I20240502 08:16:44.683207 24057 raft_consensus.cc:1261] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:44.683275 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 890fd44075ad4b5699f2fcf0a1ab7804 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.689076 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 890fd44075ad4b5699f2fcf0a1ab7804 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.675293 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet 890fd44075ad4b5699f2fcf0a1ab7804 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.679700 24773 tablet.cc:1722] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20240502 08:16:44.689399 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 890fd44075ad4b5699f2fcf0a1ab7804 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 890fd44075ad4b5699f2fcf0a1ab7804 already in progress: creating tablet
I20240502 08:16:44.681882 24174 tablet_bootstrap.cc:492] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:44.685376 24779 consensus_queue.cc:1035] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:44.698803 24174 ts_tablet_manager.cc:1367] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.163s user 0.011s sys 0.000s
I20240502 08:16:44.699103 24798 tablet.cc:1722] T b844458ae73f413cbf2113027a5af359 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:44.699601 24174 raft_consensus.cc:348] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:44.699846 24174 raft_consensus.cc:374] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.699916 24174 raft_consensus.cc:729] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.700091 24180 rpcz_store.cc:276] 0502 08:16:42.912640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.913314 (+ 674us) service_pool.cc:225] Handling call
0502 08:16:44.659089 (+1745775us) ts_tablet_manager.cc:1120] Deleting tablet e47459b5f5bb4ecb96f76240fefa1f5d
0502 08:16:44.659101 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.659259 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1740998}
I20240502 08:16:44.700517 23923 raft_consensus.cc:3037] T 36f2d7b6f4e442ae95b2d1002d571b6a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:44.700616 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e47459b5f5bb4ecb96f76240fefa1f5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e47459b5f5bb4ecb96f76240fefa1f5d already in progress: creating tablet
I20240502 08:16:44.701951 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 511) took 1789 ms. Trace:
I20240502 08:16:44.703336 24180 rpcz_store.cc:276] 0502 08:16:42.912523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.913295 (+ 772us) service_pool.cc:225] Handling call
0502 08:16:44.701787 (+1788492us) ts_tablet_manager.cc:1120] Deleting tablet d6a19bb8705c45c0bfc537d3a6bf8ae2
0502 08:16:44.701799 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.701917 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1783542}
W20240502 08:16:44.704391 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 already in progress: creating tablet
I20240502 08:16:44.703797 24174 consensus_queue.cc:260] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:44.707940 24174 ts_tablet_manager.cc:1398] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 08:16:44.708127 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 510) took 1795 ms. Trace:
I20240502 08:16:44.708628 24779 raft_consensus.cc:482] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:44.710062 24779 raft_consensus.cc:504] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:44.710713 24779 leader_election.cc:288] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:44.711422 24180 rpcz_store.cc:276] 0502 08:16:42.912407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.913269 (+ 862us) service_pool.cc:225] Handling call
0502 08:16:44.705038 (+1791769us) ts_tablet_manager.cc:1120] Deleting tablet b8b1fa17dad143f0a7f1751e651eb81a
0502 08:16:44.705049 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.708082 (+ 3033us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1786657}
I20240502 08:16:44.711498 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9700ea9dc58442e3ab1b7835a8c1e22c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:44.711884 23723 leader_election.cc:341] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 9700ea9dc58442e3ab1b7835a8c1e22c
W20240502 08:16:44.712227 23723 leader_election.cc:341] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:44.712323 23723 leader_election.cc:302] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:44.712805 24779 raft_consensus.cc:2726] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:44.713322 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet b8b1fa17dad143f0a7f1751e651eb81a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8b1fa17dad143f0a7f1751e651eb81a already in progress: creating tablet
I20240502 08:16:44.713835 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 515) took 1739 ms. Trace:
I20240502 08:16:44.719377 24180 rpcz_store.cc:276] 0502 08:16:42.973930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.974061 (+ 131us) service_pool.cc:225] Handling call
0502 08:16:44.713680 (+1739619us) ts_tablet_manager.cc:1120] Deleting tablet 1da01e2c2f054ccfb53383e70e16da84
0502 08:16:44.713691 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.713804 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1739377}
I20240502 08:16:44.715030 24174 tablet_bootstrap.cc:492] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:44.721155 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 516) took 1743 ms. Trace:
I20240502 08:16:44.721799 24180 rpcz_store.cc:276] 0502 08:16:42.977816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.977887 (+ 71us) service_pool.cc:225] Handling call
0502 08:16:44.720769 (+1742882us) ts_tablet_manager.cc:1120] Deleting tablet 76e0f767c6534a608023e6e6d33ca7a8
0502 08:16:44.720780 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.721122 (+ 342us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1742666}
I20240502 08:16:44.723378 24179 ts_tablet_manager.cc:1888] T 49b816ea1d474f2db94119441146ba78 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:44.723681 24179 log.cc:1196] T 49b816ea1d474f2db94119441146ba78 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/49b816ea1d474f2db94119441146ba78
I20240502 08:16:44.724148 24179 ts_tablet_manager.cc:1909] T 49b816ea1d474f2db94119441146ba78 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:44.723457 24180 tablet_replica.cc:330] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:44.724511 24180 raft_consensus.cc:2229] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:44.724838 24180 pending_rounds.cc:62] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:16:44.725529 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 49b816ea1d474f2db94119441146ba78 (table default.Table_20df1b504848439c869ae3739ac54185 [id=a0606a4053d04dd3814f09419a4a70c2]) successfully deleted
W20240502 08:16:44.726101 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 29144c1d42f0408f9b2987c9c66ed8dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29144c1d42f0408f9b2987c9c66ed8dc already in progress: creating tablet
I20240502 08:16:44.728811 24180 pending_rounds.cc:68] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156446352338944 op_type: NO_OP noop_request { }
I20240502 08:16:44.728986 24180 raft_consensus.cc:2866] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:44.729113 24180 raft_consensus.cc:2258] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:44.728706 24174 tablet_bootstrap.cc:654] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.730815 24180 ts_tablet_manager.cc:1875] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:44.735980 24173 tablet_bootstrap.cc:654] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:44.736464 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 843edd7092464981adad317b48871645 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 843edd7092464981adad317b48871645 already in progress: creating tablet
W20240502 08:16:44.738574 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 890fd44075ad4b5699f2fcf0a1ab7804 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 890fd44075ad4b5699f2fcf0a1ab7804 already in progress: creating tablet
I20240502 08:16:44.744246 24173 tablet_bootstrap.cc:492] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:44.744426 24173 ts_tablet_manager.cc:1367] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.080s user 0.002s sys 0.002s
I20240502 08:16:44.745247 24173 raft_consensus.cc:348] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:44.745468 24173 raft_consensus.cc:374] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.745625 24173 raft_consensus.cc:729] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.745929 24173 consensus_queue.cc:260] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:44.746208 23923 raft_consensus.cc:1261] T 36f2d7b6f4e442ae95b2d1002d571b6a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:44.746573 24175 tablet_bootstrap.cc:492] T d8f7b5f018d84680ac5533bf99102d31 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:44.751006 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet 92adebc0c69e4a52b975253656132495 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:44.753618 24173 ts_tablet_manager.cc:1398] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 08:16:44.753546 24779 consensus_queue.cc:1035] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:44.753916 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2f5e51d6ee144f3cbda8aabaee177251"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:44.754513 24173 tablet_bootstrap.cc:492] T 71aa916247c449fea718b35869a6df20 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:44.754513 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f5e51d6ee144f3cbda8aabaee177251 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.756189 24173 tablet_bootstrap.cc:654] T 71aa916247c449fea718b35869a6df20 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.757606 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_696cfb6ac5024b57aa3d8113d3be2ead" } new_table_name: "default.Table_86cb4c92da184779acc53787cbce4b4d" modify_external_catalogs: true
I20240502 08:16:44.757752 24773 tablet.cc:1722] T 36f2d7b6f4e442ae95b2d1002d571b6a P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:44.759138 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_de4adf089f7347969051330f6f43e162" } modify_external_catalogs: true
I20240502 08:16:44.759323 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_de4adf089f7347969051330f6f43e162" } modify_external_catalogs: true
I20240502 08:16:44.763455 23602 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_cc80d1cedafa4e71906e5e9530fdd828 [id=47682c26a85842c7b8991bf3828a9336] alter complete (version 1)
I20240502 08:16:44.763900 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_01f87df24c614973b564898fe3bfe511"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:44.764443 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01f87df24c614973b564898fe3bfe511 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.764494 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9f3af157070d4bf5a099b901f335bc52" } new_table_name: "default.Table_41458994cfa3412b8d5c9e2948df25e3" modify_external_catalogs: true
I20240502 08:16:44.766163 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1b8fb2a811c54b659fb14d3bb815bc75" } new_table_name: "default.Table_79bacd8e921e4c1bafaf368c3e64bfaa" modify_external_catalogs: true
I20240502 08:16:44.767634 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 7a19310948e248d1a85479c41c61bda2 (table default.Table_785d5ddac96746379d1117bc8379566b [id=8521bfc13ef040a78cfb2cc4501b7f04]). Expected version 4 got 0
I20240502 08:16:44.768370 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_91a1219d5c2542d9a45c7b8499786337"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:44.768802 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91a1219d5c2542d9a45c7b8499786337 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.769830 24771 tablet.cc:1722] T 36f2d7b6f4e442ae95b2d1002d571b6a P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:44.771831 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 4baa2bc2752c450fa212471c150cf098 (DEFAULT_TABLE table=default.Table_0ba7fb682ff94893b3ef11abc45a7f6d [id=88e62eedebd44c2f9b0d2e1ccbac8e59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.772255 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4baa2bc2752c450fa212471c150cf098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.784307 24021 tablet_service.cc:1467] Processing CreateTablet for tablet f685cbe705b149e69f9536e270a74a42 (DEFAULT_TABLE table=default.Table_2e579720e5df47e9875f86b878b88260 [id=9f6c06dc2a2f4a50ac0e512b26348c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.784899 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f685cbe705b149e69f9536e270a74a42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.785552 24173 tablet_bootstrap.cc:492] T 71aa916247c449fea718b35869a6df20 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:44.785815 24173 ts_tablet_manager.cc:1367] T 71aa916247c449fea718b35869a6df20 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.014s
I20240502 08:16:44.786617 24173 raft_consensus.cc:348] T 71aa916247c449fea718b35869a6df20 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:44.788301 24173 raft_consensus.cc:374] T 71aa916247c449fea718b35869a6df20 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.788709 24173 raft_consensus.cc:729] T 71aa916247c449fea718b35869a6df20 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.788859 24180 ts_tablet_manager.cc:1888] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:44.789016 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 68bdb761ff36426c96949e97279e2756
I20240502 08:16:44.789345 24173 consensus_queue.cc:260] T 71aa916247c449fea718b35869a6df20 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:44.790300 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 23ba091d8efb44858bf90a104c0f442f
I20240502 08:16:44.791586 24173 ts_tablet_manager.cc:1398] T 71aa916247c449fea718b35869a6df20 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:16:44.792536 24173 tablet_bootstrap.cc:492] T 643ce929e74c499da835fbe9b63b8880 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:44.794068 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d2886233097a4d19b45a4f139d82e68a consensus state has no leader
I20240502 08:16:44.794380 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet 1305dbdac3994224a047df139cf8d518 from {username='slave'} at 127.0.0.1:36668
W20240502 08:16:44.795051 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 63a595654cff4565bcf21a7c8fabc2c4 consensus state has no leader
I20240502 08:16:44.795433 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet 7a97427e819b4826b1ed89e6d021c8d4 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:44.795759 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_102eeba031aa4389820d28985935dd4c" } new_table_name: "default.Table_9aa2ee2fdc9b4b91996a406f42e5c440" modify_external_catalogs: true
I20240502 08:16:44.796450 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0692aa66a8ad444eab3af22ad78c527e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:44.797191 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 324716a1b3c44be2b415695f821de8fc
W20240502 08:16:44.798327 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0692aa66a8ad444eab3af22ad78c527e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.798509 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_de7dd656583f4c118ba6bd8bb1945caf" } modify_external_catalogs: true
I20240502 08:16:44.799099 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet c5d28098dcbc4761a34ee65769d6fd7c from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:44.800395 24173 tablet_bootstrap.cc:654] T 643ce929e74c499da835fbe9b63b8880 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:44.802525 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba3aa2621fdb43fd8299f81af486ef1e consensus state has no leader
I20240502 08:16:44.802726 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_de7dd656583f4c118ba6bd8bb1945caf" } modify_external_catalogs: true
I20240502 08:16:44.803501 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 92adebc0c69e4a52b975253656132495
I20240502 08:16:44.805104 24175 ts_tablet_manager.cc:1367] T d8f7b5f018d84680ac5533bf99102d31 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.131s user 0.025s sys 0.034s
I20240502 08:16:44.805454 23604 catalog_manager.cc:6554] ReplaceTablet: tablet 92adebc0c69e4a52b975253656132495 deleted and replaced by tablet ae2b20ae3a014c85af33d5ee51b704e1
I20240502 08:16:44.805714 23610 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 36f2d7b6f4e442ae95b2d1002d571b6a (table default.Table_e7dcb7ee94304b10b88d59b0872ae586 [id=a95a99a58cde44fe814b3edfcc643a80]). Expected version 2 got 0
I20240502 08:16:44.806102 24798 tablet.cc:1722] T b844458ae73f413cbf2113027a5af359 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:44.807348 23607 master_service.cc:946] ReplaceTablet: received request to replace tablet 592eeecbf5514215b6464bd030081e4b from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:44.808012 24180 log.cc:1196] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/fd57b0357ca749c389bc61add20cd3e7
I20240502 08:16:44.808167 24790 raft_consensus.cc:482] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:44.808321 24790 raft_consensus.cc:504] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:44.814389 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 40964817cf2a4712be42f40e37639bc9 (DEFAULT_TABLE table=default.Table_7937f18be63b41d7945c72ed597adfcb [id=362af6ee46a740b28b0fa3e9ae3391fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.814842 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40964817cf2a4712be42f40e37639bc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.818087 24180 ts_tablet_manager.cc:1909] T fd57b0357ca749c389bc61add20cd3e7 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:44.819530 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 517) took 1834 ms. Trace:
I20240502 08:16:44.819684 24180 rpcz_store.cc:276] 0502 08:16:42.984868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:42.984929 (+ 61us) service_pool.cc:225] Handling call
0502 08:16:44.723358 (+1738429us) ts_tablet_manager.cc:1120] Deleting tablet fd57b0357ca749c389bc61add20cd3e7
0502 08:16:44.723369 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.785707 (+ 62338us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:44.788758 (+ 3051us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:44.819492 (+ 30734us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1738170}
I20240502 08:16:44.821447 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 520) took 1801 ms. Trace:
I20240502 08:16:44.821556 24180 rpcz_store.cc:276] 0502 08:16:43.020389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.020452 (+ 63us) service_pool.cc:225] Handling call
0502 08:16:44.821297 (+1800845us) ts_tablet_manager.cc:1120] Deleting tablet e5f5597402374527bf0e5e50178df713
0502 08:16:44.821305 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.821416 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1800556}
W20240502 08:16:44.821830 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90d4e2f23c9649a4b1dfb6a179ca7be5 consensus state has no leader
I20240502 08:16:44.822654 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 535) took 1659 ms. Trace:
I20240502 08:16:44.822758 24180 rpcz_store.cc:276] 0502 08:16:43.163100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.163312 (+ 212us) service_pool.cc:225] Handling call
0502 08:16:44.822523 (+1659211us) ts_tablet_manager.cc:1120] Deleting tablet df08f0d3af4c4c7e8abce656714b1b7f
0502 08:16:44.822531 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.822622 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1653982}
I20240502 08:16:44.822853 23610 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_a86b6b9ac2d8498c97341dc60ca5ee34 [id=fd93259a378b4ea582a6661a59104abb] alter complete (version 3)
I20240502 08:16:44.823621 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 3cc189507d0b49728fb22ca590126312 (DEFAULT_TABLE table=default.Table_52cae77d343b4c43bb8cf4fa6f4727af [id=0966e7e735fb48e3b751721caee9eff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.824043 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cc189507d0b49728fb22ca590126312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.823671 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 202fa612a085450ca9c1225434cb7fe4 (DEFAULT_TABLE table=default.Table_722c9cabaa9a4f43a7c8aeab8a17dc05 [id=98df2952710847e9af30703324a08d66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.824427 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 202fa612a085450ca9c1225434cb7fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.824630 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 22240da6da2241bd93b6547a0aae917e (DEFAULT_TABLE table=default.Table_02b3fe4baa274e408220d9d27d9db7fa [id=2ebd33263cb94055a28bdc71f5082869]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.825006 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22240da6da2241bd93b6547a0aae917e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.827406 23610 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 36f2d7b6f4e442ae95b2d1002d571b6a (table default.Table_e7dcb7ee94304b10b88d59b0872ae586 [id=a95a99a58cde44fe814b3edfcc643a80]). Expected version 2 got 0
I20240502 08:16:44.827589 23610 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet b844458ae73f413cbf2113027a5af359 (table default.Table_a86b6b9ac2d8498c97341dc60ca5ee34 [id=fd93259a378b4ea582a6661a59104abb]). Expected version 3 got 2
I20240502 08:16:44.827729 23610 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 7a19310948e248d1a85479c41c61bda2 (table default.Table_785d5ddac96746379d1117bc8379566b [id=8521bfc13ef040a78cfb2cc4501b7f04]). Expected version 4 got 0
I20240502 08:16:44.829849 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 2d16819a129448d582a23d49efdaabd2 (DEFAULT_TABLE table=default.Table_a77d44d9077d44fb9db187597d42756b [id=6261d1c2ceb745a5bd2fc6feda69d6f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.830209 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d16819a129448d582a23d49efdaabd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.839596 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 2d16819a129448d582a23d49efdaabd2 (DEFAULT_TABLE table=default.Table_a77d44d9077d44fb9db187597d42756b [id=6261d1c2ceb745a5bd2fc6feda69d6f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.840040 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d16819a129448d582a23d49efdaabd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.840314 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 4baa2bc2752c450fa212471c150cf098 (DEFAULT_TABLE table=default.Table_0ba7fb682ff94893b3ef11abc45a7f6d [id=88e62eedebd44c2f9b0d2e1ccbac8e59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.840649 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4baa2bc2752c450fa212471c150cf098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.841835 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 40964817cf2a4712be42f40e37639bc9 (DEFAULT_TABLE table=default.Table_7937f18be63b41d7945c72ed597adfcb [id=362af6ee46a740b28b0fa3e9ae3391fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.842171 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40964817cf2a4712be42f40e37639bc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.845875 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 202fa612a085450ca9c1225434cb7fe4 (DEFAULT_TABLE table=default.Table_722c9cabaa9a4f43a7c8aeab8a17dc05 [id=98df2952710847e9af30703324a08d66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.846163 23760 tablet_service.cc:1467] Processing CreateTablet for tablet d935b59497bc48a190061a6111e9ae1a (DEFAULT_TABLE table=default.Table_a1b11bbb99a24f79b6d33fbf9cae4d5f [id=de36429b0ca34ce592e4405254ea65c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.846271 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 202fa612a085450ca9c1225434cb7fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.846457 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d935b59497bc48a190061a6111e9ae1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.846736 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 3cc189507d0b49728fb22ca590126312 (DEFAULT_TABLE table=default.Table_52cae77d343b4c43bb8cf4fa6f4727af [id=0966e7e735fb48e3b751721caee9eff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.847059 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cc189507d0b49728fb22ca590126312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.847540 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 324716a1b3c44be2b415695f821de8fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.848647 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 23ba091d8efb44858bf90a104c0f442f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.849836 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 982333cbce18439da3b35693b3f6ca0a (DEFAULT_TABLE table=default.Table_f194510125f340b58d41408e911d2721 [id=f4ccf1af90aa43ad9ce620becec6f4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.850203 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 982333cbce18439da3b35693b3f6ca0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.853217 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet 68bdb761ff36426c96949e97279e2756 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.853459 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet 92adebc0c69e4a52b975253656132495 with delete_type TABLET_DATA_DELETED (replaced by tablet ae2b20ae3a014c85af33d5ee51b704e1) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.845880 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 22240da6da2241bd93b6547a0aae917e (DEFAULT_TABLE table=default.Table_02b3fe4baa274e408220d9d27d9db7fa [id=2ebd33263cb94055a28bdc71f5082869]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.853979 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22240da6da2241bd93b6547a0aae917e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.857117 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 22240da6da2241bd93b6547a0aae917e (DEFAULT_TABLE table=default.Table_02b3fe4baa274e408220d9d27d9db7fa [id=2ebd33263cb94055a28bdc71f5082869]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.857267 23759 tablet_service.cc:1467] Processing CreateTablet for tablet f685cbe705b149e69f9536e270a74a42 (DEFAULT_TABLE table=default.Table_2e579720e5df47e9875f86b878b88260 [id=9f6c06dc2a2f4a50ac0e512b26348c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.857530 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22240da6da2241bd93b6547a0aae917e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.857630 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f685cbe705b149e69f9536e270a74a42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.861631 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet 324716a1b3c44be2b415695f821de8fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.863308 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 68bdb761ff36426c96949e97279e2756 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:44.863570 23861 leader_election.cc:341] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: f36aadfa1d264f23bcafa91744aed36d
I20240502 08:16:44.863723 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet f36aadfa1d264f23bcafa91744aed36d with delete_type TABLET_DATA_DELETED (replaced by tablet 2085ced52fff46ebaa98324cbb2d8c11) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.863847 24790 leader_election.cc:288] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:44.863961 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 23ba091d8efb44858bf90a104c0f442f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.864079 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 92adebc0c69e4a52b975253656132495 with delete_type TABLET_DATA_DELETED (replaced by tablet ae2b20ae3a014c85af33d5ee51b704e1) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:44.864457 23723 consensus_peers.cc:485] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ce5ccfd29724dc49afd0892b99f3169. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:44.864728 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 8810b6f61db14fed8b12d36fed413dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.865684 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 3cc189507d0b49728fb22ca590126312 (DEFAULT_TABLE table=default.Table_52cae77d343b4c43bb8cf4fa6f4727af [id=0966e7e735fb48e3b751721caee9eff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.865893 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 40964817cf2a4712be42f40e37639bc9 (DEFAULT_TABLE table=default.Table_7937f18be63b41d7945c72ed597adfcb [id=362af6ee46a740b28b0fa3e9ae3391fd]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:44.867148 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 324716a1b3c44be2b415695f821de8fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 324716a1b3c44be2b415695f821de8fc already in progress: creating tablet
I20240502 08:16:44.867616 23887 tablet_service.cc:1467] Processing CreateTablet for tablet f685cbe705b149e69f9536e270a74a42 (DEFAULT_TABLE table=default.Table_2e579720e5df47e9875f86b878b88260 [id=9f6c06dc2a2f4a50ac0e512b26348c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.865525 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 202fa612a085450ca9c1225434cb7fe4 (DEFAULT_TABLE table=default.Table_722c9cabaa9a4f43a7c8aeab8a17dc05 [id=98df2952710847e9af30703324a08d66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.868549 23886 tablet_service.cc:1467] Processing CreateTablet for tablet d935b59497bc48a190061a6111e9ae1a (DEFAULT_TABLE table=default.Table_a1b11bbb99a24f79b6d33fbf9cae4d5f [id=de36429b0ca34ce592e4405254ea65c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.868817 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 2d16819a129448d582a23d49efdaabd2 (DEFAULT_TABLE table=default.Table_a77d44d9077d44fb9db187597d42756b [id=6261d1c2ceb745a5bd2fc6feda69d6f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.869680 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 982333cbce18439da3b35693b3f6ca0a (DEFAULT_TABLE table=default.Table_f194510125f340b58d41408e911d2721 [id=f4ccf1af90aa43ad9ce620becec6f4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.870545 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3a47de7a9c814ca2a1656345ebbf39c7" } new_table_name: "default.Table_b465ef0aa8664f5d86736ed308c81e32" modify_external_catalogs: true
I20240502 08:16:44.870584 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 4baa2bc2752c450fa212471c150cf098 (DEFAULT_TABLE table=default.Table_0ba7fb682ff94893b3ef11abc45a7f6d [id=88e62eedebd44c2f9b0d2e1ccbac8e59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.871376 24179 tablet_replica.cc:330] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:44.871577 24179 raft_consensus.cc:2229] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:44.871718 24039 tablet_service.cc:1467] Processing CreateTablet for tablet d935b59497bc48a190061a6111e9ae1a (DEFAULT_TABLE table=default.Table_a1b11bbb99a24f79b6d33fbf9cae4d5f [id=de36429b0ca34ce592e4405254ea65c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.864248 24175 raft_consensus.cc:348] T d8f7b5f018d84680ac5533bf99102d31 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:44.872571 24179 raft_consensus.cc:2258] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:44.873129 24179 ts_tablet_manager.cc:1875] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:44.873853 23858 leader_election.cc:341] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: f36aadfa1d264f23bcafa91744aed36d
I20240502 08:16:44.874047 24175 raft_consensus.cc:374] T d8f7b5f018d84680ac5533bf99102d31 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.874154 24175 raft_consensus.cc:729] T d8f7b5f018d84680ac5533bf99102d31 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.874359 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 592eeecbf5514215b6464bd030081e4b
I20240502 08:16:44.874843 23607 catalog_manager.cc:6554] ReplaceTablet: tablet 592eeecbf5514215b6464bd030081e4b deleted and replaced by tablet 28a947b7e7484120b3f4069f6eef5520
I20240502 08:16:44.874805 24175 consensus_queue.cc:260] T d8f7b5f018d84680ac5533bf99102d31 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:44.875027 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 1da01e2c2f054ccfb53383e70e16da84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1da01e2c2f054ccfb53383e70e16da84 already in progress: creating tablet
W20240502 08:16:44.875337 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 76e0f767c6534a608023e6e6d33ca7a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76e0f767c6534a608023e6e6d33ca7a8 already in progress: creating tablet
I20240502 08:16:44.875401 24175 ts_tablet_manager.cc:1398] T d8f7b5f018d84680ac5533bf99102d31 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.070s user 0.002s sys 0.000s
I20240502 08:16:44.875600 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet fd57b0357ca749c389bc61add20cd3e7 (table default.Table_f52a4fa0e0c04f19a517f946b59a850f [id=fd141979a86c45bf9648396d086f80d4]) successfully deleted
W20240502 08:16:44.875972 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e5f5597402374527bf0e5e50178df713 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5f5597402374527bf0e5e50178df713 already in progress: creating tablet
I20240502 08:16:44.876227 24175 tablet_bootstrap.cc:492] T e0af5f4daf3149738647a08219efb245 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:44.876777 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet df08f0d3af4c4c7e8abce656714b1b7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df08f0d3af4c4c7e8abce656714b1b7f already in progress: creating tablet
I20240502 08:16:44.879824 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_dd2bda6955ae4ccab8e639637c37df7f" } modify_external_catalogs: true
I20240502 08:16:44.880017 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_dd2bda6955ae4ccab8e639637c37df7f" } modify_external_catalogs: true
I20240502 08:16:44.881418 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_f26e93806daa446280d6312001630c82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:44.881927 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f26e93806daa446280d6312001630c82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.882285 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet 592eeecbf5514215b6464bd030081e4b with delete_type TABLET_DATA_DELETED (replaced by tablet 28a947b7e7484120b3f4069f6eef5520) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.882758 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 91646e7af3824520a1fe28e180eeb0a7
W20240502 08:16:44.883061 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9b67c8cd7f34cbba6401da59dec526c consensus state has no leader
I20240502 08:16:44.883916 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 91646e7af3824520a1fe28e180eeb0a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.886754 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d935b59497bc48a190061a6111e9ae1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.887182 23858 leader_election.cc:302] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:44.887530 24790 raft_consensus.cc:2726] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:44.888077 23723 consensus_peers.cc:485] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dead13bba95f43cb809e17f6d7fba028. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:44.888271 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 202fa612a085450ca9c1225434cb7fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.888548 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40964817cf2a4712be42f40e37639bc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.888778 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cc189507d0b49728fb22ca590126312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.888967 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d16819a129448d582a23d49efdaabd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.889199 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4baa2bc2752c450fa212471c150cf098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.890040 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 982333cbce18439da3b35693b3f6ca0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.890667 24173 tablet_bootstrap.cc:492] T 643ce929e74c499da835fbe9b63b8880 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:44.890852 24173 ts_tablet_manager.cc:1367] T 643ce929e74c499da835fbe9b63b8880 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.098s user 0.008s sys 0.004s
I20240502 08:16:44.891144 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f685cbe705b149e69f9536e270a74a42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.891736 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d935b59497bc48a190061a6111e9ae1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.895393 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet ef03faf0a13348d7acad954256b5a1d2
I20240502 08:16:44.897744 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 23ba091d8efb44858bf90a104c0f442f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.898623 23607 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_e7dcb7ee94304b10b88d59b0872ae586 [id=a95a99a58cde44fe814b3edfcc643a80] alter complete (version 2)
I20240502 08:16:44.898816 24173 raft_consensus.cc:348] T 643ce929e74c499da835fbe9b63b8880 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:44.899061 24173 raft_consensus.cc:374] T 643ce929e74c499da835fbe9b63b8880 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.899230 24173 raft_consensus.cc:729] T 643ce929e74c499da835fbe9b63b8880 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.899468 24173 consensus_queue.cc:260] T 643ce929e74c499da835fbe9b63b8880 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:44.901206 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 982333cbce18439da3b35693b3f6ca0a (DEFAULT_TABLE table=default.Table_f194510125f340b58d41408e911d2721 [id=f4ccf1af90aa43ad9ce620becec6f4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.901664 24173 ts_tablet_manager.cc:1398] T 643ce929e74c499da835fbe9b63b8880 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240502 08:16:44.901718 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 982333cbce18439da3b35693b3f6ca0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.901954 24173 tablet_bootstrap.cc:492] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:44.909679 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet 9700ea9dc58442e3ab1b7835a8c1e22c with delete_type TABLET_DATA_DELETED (replaced by tablet 5d04312bd81f46ed8c49c42b1af1aec9) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.911068 24173 tablet_bootstrap.cc:654] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.914000 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 592eeecbf5514215b6464bd030081e4b with delete_type TABLET_DATA_DELETED (replaced by tablet 28a947b7e7484120b3f4069f6eef5520) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.920536 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet ef03faf0a13348d7acad954256b5a1d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.920914 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 91646e7af3824520a1fe28e180eeb0a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:44.922498 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet ef03faf0a13348d7acad954256b5a1d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:44.926604 23925 raft_consensus.cc:1261] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:44.927809 24175 tablet_bootstrap.cc:654] T e0af5f4daf3149738647a08219efb245 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.933336 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 68bdb761ff36426c96949e97279e2756 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.934862 24181 tablet_replica.cc:330] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:44.935151 24181 raft_consensus.cc:2229] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:44.935420 24181 raft_consensus.cc:2258] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:44.936055 24174 tablet_bootstrap.cc:492] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:44.936106 24181 ts_tablet_manager.cc:1875] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:44.936237 24174 ts_tablet_manager.cc:1367] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.221s user 0.005s sys 0.004s
I20240502 08:16:44.937111 24174 raft_consensus.cc:348] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:44.937364 24174 raft_consensus.cc:374] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.937476 24174 raft_consensus.cc:729] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.937765 24174 consensus_queue.cc:260] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:44.937876 24780 consensus_queue.cc:1035] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:44.938376 24174 ts_tablet_manager.cc:1398] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:44.939116 24174 tablet_bootstrap.cc:492] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:44.940943 24174 tablet_bootstrap.cc:654] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.943449 24181 ts_tablet_manager.cc:1888] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 08:16:44.949561 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 533) took 1786 ms. Trace:
I20240502 08:16:44.949751 24180 rpcz_store.cc:276] 0502 08:16:43.162719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.163510 (+ 791us) service_pool.cc:225] Handling call
0502 08:16:44.949348 (+1785838us) ts_tablet_manager.cc:1120] Deleting tablet 89f57e3a39d54b7ab372d4b145b7a696
0502 08:16:44.949358 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.949521 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1779312}
I20240502 08:16:44.951627 24174 tablet_bootstrap.cc:492] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:44.951831 24174 ts_tablet_manager.cc:1367] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20240502 08:16:44.952534 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 539) took 1771 ms. Trace:
I20240502 08:16:44.952663 24180 rpcz_store.cc:276] 0502 08:16:43.180794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.180849 (+ 55us) service_pool.cc:225] Handling call
0502 08:16:44.952349 (+1771500us) ts_tablet_manager.cc:1120] Deleting tablet 29f82ad5860843f1adfbb40cdb21a4a3
0502 08:16:44.952360 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.952484 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1690806}
I20240502 08:16:44.952788 24174 raft_consensus.cc:348] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:44.953065 24174 raft_consensus.cc:374] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.954277 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 540) took 1686 ms. Trace:
I20240502 08:16:44.954519 24180 rpcz_store.cc:276] 0502 08:16:43.268204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.268269 (+ 65us) service_pool.cc:225] Handling call
0502 08:16:44.954146 (+1685877us) ts_tablet_manager.cc:1120] Deleting tablet 63e645bb6e2948e091e08433944a52f4
0502 08:16:44.954157 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.954245 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1667544}
I20240502 08:16:44.955025 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 572) took 1469 ms. Trace:
I20240502 08:16:44.955133 24180 rpcz_store.cc:276] 0502 08:16:43.485449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.731609 (+246160us) service_pool.cc:225] Handling call
0502 08:16:44.954896 (+1223287us) ts_tablet_manager.cc:1120] Deleting tablet 643ce929e74c499da835fbe9b63b8880
0502 08:16:44.954905 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.954998 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1222966}
I20240502 08:16:44.956528 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 573) took 1469 ms. Trace:
I20240502 08:16:44.956629 24180 rpcz_store.cc:276] 0502 08:16:43.487013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.731960 (+244947us) service_pool.cc:225] Handling call
0502 08:16:44.956381 (+1224421us) ts_tablet_manager.cc:1120] Deleting tablet a198c33a3e314b5b81a08244862b281a
0502 08:16:44.956388 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.956498 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1224163}
I20240502 08:16:44.956612 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6e14e5ad32c946d286dc2416e1a16f61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:44.956612 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b5aef32596c94f51bb917536c2bc7341"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:44.957286 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5aef32596c94f51bb917536c2bc7341 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:44.957281 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e14e5ad32c946d286dc2416e1a16f61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.960695 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5964ac143b9f461287f7b52b806195d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:44.960837 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8c1e23fe00844168a9f1755bedd3625a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:44.961014 24181 log.cc:1196] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/68bdb761ff36426c96949e97279e2756
W20240502 08:16:44.961304 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5964ac143b9f461287f7b52b806195d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:44.961413 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c1e23fe00844168a9f1755bedd3625a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:44.950212 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 324716a1b3c44be2b415695f821de8fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.951056 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 592eeecbf5514215b6464bd030081e4b with delete_type TABLET_DATA_DELETED (replaced by tablet 28a947b7e7484120b3f4069f6eef5520) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.951570 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet a3043c5ed98c401b82c6e967ca3f3613 with delete_type TABLET_DATA_DELETED (replaced by tablet 0281c8123eab4c48bddc377f5695c6e0) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.951733 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet ef03faf0a13348d7acad954256b5a1d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.952204 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet 71aa916247c449fea718b35869a6df20 with delete_type TABLET_DATA_DELETED (replaced by tablet 0501575a84ec454780f48574bf3155fe) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.952510 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet 643ce929e74c499da835fbe9b63b8880 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.953287 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet bcfacc302ff04178bfffc48111cd7a3f with delete_type TABLET_DATA_DELETED (replaced by tablet b0cf1a6de3d649b984c50f45393ec0f1) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.953449 24021 tablet_service.cc:1514] Processing DeleteTablet for tablet 0882b7509a74475780eb700818aab398 with delete_type TABLET_DATA_DELETED (replaced by tablet eb96ad2289754e428a1c109b2d59c53b) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.953697 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 92adebc0c69e4a52b975253656132495 with delete_type TABLET_DATA_DELETED (replaced by tablet ae2b20ae3a014c85af33d5ee51b704e1) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.953895 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 91646e7af3824520a1fe28e180eeb0a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:44.962383 24180 tablet_replica.cc:330] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:44.962568 24180 raft_consensus.cc:2229] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:44.962721 24180 raft_consensus.cc:2258] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:44.963280 24180 ts_tablet_manager.cc:1875] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:44.967993 24174 raft_consensus.cc:729] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.968423 24174 consensus_queue.cc:260] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:44.969179 24174 ts_tablet_manager.cc:1398] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240502 08:16:44.969240 24180 ts_tablet_manager.cc:1888] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:44.969365 24180 log.cc:1196] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/5c95dfda9d3b4e2ca9397b77ad71c011
I20240502 08:16:44.969735 24174 tablet_bootstrap.cc:492] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:44.971446 24174 tablet_bootstrap.cc:654] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.978539 24181 ts_tablet_manager.cc:1909] T 68bdb761ff36426c96949e97279e2756 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
W20240502 08:16:44.980875 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 23ba091d8efb44858bf90a104c0f442f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23ba091d8efb44858bf90a104c0f442f already in progress: creating tablet
W20240502 08:16:44.981138 24715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:44.982753 24715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20240502 08:16:44.982975 24179 ts_tablet_manager.cc:1888] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 08:16:44.983096 24179 log.cc:1196] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/68bdb761ff36426c96949e97279e2756
I20240502 08:16:44.983088 24181 tablet_replica.cc:330] T 71aa916247c449fea718b35869a6df20 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
W20240502 08:16:44.983318 24715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 08:16:44.983457 24179 ts_tablet_manager.cc:1909] T 68bdb761ff36426c96949e97279e2756 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:44.984318 24179 tablet_replica.cc:330] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:44.984532 24179 raft_consensus.cc:2229] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:44.984732 24179 raft_consensus.cc:2258] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:44.986002 24179 ts_tablet_manager.cc:1875] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:44.990445 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 68bdb761ff36426c96949e97279e2756 (table default.Table_de4adf089f7347969051330f6f43e162 [id=eca4080e8ebf46e9b58c85ee1d8675ca]) successfully deleted
I20240502 08:16:44.990643 24057 raft_consensus.cc:1261] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:16:44.993698 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 592eeecbf5514215b6464bd030081e4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 592eeecbf5514215b6464bd030081e4b already in progress: creating tablet
W20240502 08:16:44.994753 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 324716a1b3c44be2b415695f821de8fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 324716a1b3c44be2b415695f821de8fc already in progress: creating tablet
I20240502 08:16:44.995580 24753 raft_consensus.cc:482] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:16:44.995795 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet ef03faf0a13348d7acad954256b5a1d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef03faf0a13348d7acad954256b5a1d2 already in progress: creating tablet
I20240502 08:16:44.995764 24753 raft_consensus.cc:504] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:44.996305 24181 raft_consensus.cc:2229] T 71aa916247c449fea718b35869a6df20 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:44.996519 24181 raft_consensus.cc:2258] T 71aa916247c449fea718b35869a6df20 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:44.996578 24753 leader_election.cc:288] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:44.997519 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3043c5ed98c401b82c6e967ca3f3613" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:44.997637 24033 tablet_service.cc:1467] Processing CreateTablet for tablet abdcdf7db3e34e77980d4bcc07bcd090 (DEFAULT_TABLE table=default.Table_01f87df24c614973b564898fe3bfe511 [id=0746ccdaa1f44f00a4f3a4e4d8b2946c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.998152 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdcdf7db3e34e77980d4bcc07bcd090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.009660 23606 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 36f2d7b6f4e442ae95b2d1002d571b6a (table default.Table_e7dcb7ee94304b10b88d59b0872ae586 [id=a95a99a58cde44fe814b3edfcc643a80]). Expected version 2 got 0
I20240502 08:16:45.014307 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 68bdb761ff36426c96949e97279e2756 (table default.Table_de4adf089f7347969051330f6f43e162 [id=eca4080e8ebf46e9b58c85ee1d8675ca]) successfully deleted
I20240502 08:16:45.014336 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 54619769ce264106b1ee3765572f13ba (DEFAULT_TABLE table=default.Table_0f88db2e7c844740bcb3df0fb75508b8 [id=0135506624894f12bcf045ece0cec5b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.016881 23894 tablet_service.cc:1467] Processing CreateTablet for tablet e0eb928812a24d1cb82e00b549b36516 (DEFAULT_TABLE table=default.Table_2f5e51d6ee144f3cbda8aabaee177251 [id=65fca5e9756c40808bbfbc4b436e40d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.017697 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 73d7633fd23541f3a553af008dd6a354 (DEFAULT_TABLE table=default.Table_19538f34f25a4fd79dd6f076de5c021e [id=c8e08fefe5c14979a06c8780d4bb5687]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.018890 24173 tablet_bootstrap.cc:492] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:45.018960 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3043c5ed98c401b82c6e967ca3f3613" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:45.019131 24173 ts_tablet_manager.cc:1367] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.117s user 0.002s sys 0.003s
W20240502 08:16:45.019980 23993 leader_election.cc:341] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:45.020193 24173 raft_consensus.cc:348] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.020522 24173 raft_consensus.cc:374] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.020618 24173 raft_consensus.cc:729] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
W20240502 08:16:45.021934 24715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20240502 08:16:45.023447 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 0d446aae7dec4734b112490ed77f26a1 (DEFAULT_TABLE table=default.Table_8eddc9a260a04c339e7bb130c9822c62 [id=8b7bfc79afbf4effa6a8ed7204065aec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.023561 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 397003e47c78418182e0167c3b876d4e (DEFAULT_TABLE table=default.Table_c2b1aa8d133542ffb45da2eacecf427b [id=1b4287236fcc4d4abf9d009d14bf8dde]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:45.023659 24715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:45.023754 24715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:16:45.023842 24715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:16:45.023846 23896 tablet_service.cc:1467] Processing CreateTablet for tablet abdcdf7db3e34e77980d4bcc07bcd090 (DEFAULT_TABLE table=default.Table_01f87df24c614973b564898fe3bfe511 [id=0746ccdaa1f44f00a4f3a4e4d8b2946c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.024178 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 397003e47c78418182e0167c3b876d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.024542 23905 tablet_service.cc:1467] Processing CreateTablet for tablet e9133e69e1724a26bc3650ea658ca349 (DEFAULT_TABLE table=default.Table_44bdfbf3075343e0a2ea5b79c0ca93e5 [id=513068e8e8fa48898966cbe95fa52422]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.025301 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 33bbebbca7464ea6ad59b81d7b7c560d (DEFAULT_TABLE table=default.Table_08a0e708a9ad4c619295ddc6c62e1a83 [id=563645aa3e16496bb1b7f1fd8f9c2132]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.026016 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54619769ce264106b1ee3765572f13ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.026791 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9133e69e1724a26bc3650ea658ca349. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.027514 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d446aae7dec4734b112490ed77f26a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.029279 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33bbebbca7464ea6ad59b81d7b7c560d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.029914 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d7633fd23541f3a553af008dd6a354. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.030105 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0eb928812a24d1cb82e00b549b36516. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.030355 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdcdf7db3e34e77980d4bcc07bcd090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.030344 24180 ts_tablet_manager.cc:1909] T 5c95dfda9d3b4e2ca9397b77ad71c011 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:45.031360 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 574) took 1519 ms. Trace:
I20240502 08:16:45.031258 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_32c806f7aa664335b36da222d5fd5618"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.031831 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32c806f7aa664335b36da222d5fd5618 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.032258 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 25f06e16c1ae4ed7875ab463071581a5 (DEFAULT_TABLE table=default.Table_7991f9ca2d6f4a79aa0aeb12b23f1000 [id=6a34778cf31f4e089bedcc1d9fc4aa0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.032694 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f06e16c1ae4ed7875ab463071581a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.031489 24180 rpcz_store.cc:276] 0502 08:16:43.513641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.732234 (+218593us) service_pool.cc:225] Handling call
0502 08:16:44.962321 (+1230087us) ts_tablet_manager.cc:1120] Deleting tablet 5c95dfda9d3b4e2ca9397b77ad71c011
0502 08:16:44.962330 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:44.966299 (+ 3969us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:44.969148 (+ 2849us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:45.031254 (+ 62106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1229737}
I20240502 08:16:45.038043 24180 tablet_replica.cc:330] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:45.038314 24180 raft_consensus.cc:2229] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:45.038635 24780 consensus_queue.cc:1035] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:45.038719 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 397003e47c78418182e0167c3b876d4e (DEFAULT_TABLE table=default.Table_c2b1aa8d133542ffb45da2eacecf427b [id=1b4287236fcc4d4abf9d009d14bf8dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.040017 24174 tablet_bootstrap.cc:492] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:45.040181 24174 ts_tablet_manager.cc:1367] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.071s user 0.012s sys 0.005s
I20240502 08:16:45.032433 24181 ts_tablet_manager.cc:1875] T 71aa916247c449fea718b35869a6df20 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.033160 24021 tablet_service.cc:1467] Processing CreateTablet for tablet 54619769ce264106b1ee3765572f13ba (DEFAULT_TABLE table=default.Table_0f88db2e7c844740bcb3df0fb75508b8 [id=0135506624894f12bcf045ece0cec5b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.040740 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54619769ce264106b1ee3765572f13ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.041102 24174 raft_consensus.cc:348] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.041317 24174 raft_consensus.cc:374] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.041428 24174 raft_consensus.cc:729] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.041707 24174 consensus_queue.cc:260] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.042372 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 13cbab4352bb44f389d70d4379e92457 (DEFAULT_TABLE table=default.Table_8ac26bacf7bc4a05bacf7f35c2755a74 [id=a29e7a86803442cfb88b97e0575c3700]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.043839 24174 ts_tablet_manager.cc:1398] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240502 08:16:45.044184 24174 tablet_bootstrap.cc:492] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:45.033874 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 73d7633fd23541f3a553af008dd6a354 (DEFAULT_TABLE table=default.Table_19538f34f25a4fd79dd6f076de5c021e [id=c8e08fefe5c14979a06c8780d4bb5687]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.044637 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d7633fd23541f3a553af008dd6a354. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.045702 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 0ae92276246c4d39b194d7e9b868fc02 (DEFAULT_TABLE table=default.Table_91a1219d5c2542d9a45c7b8499786337 [id=a96384f644c64d3184405e02fe3d2d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.034395 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 25f06e16c1ae4ed7875ab463071581a5 (DEFAULT_TABLE table=default.Table_7991f9ca2d6f4a79aa0aeb12b23f1000 [id=6a34778cf31f4e089bedcc1d9fc4aa0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.034888 24036 tablet_service.cc:1467] Processing CreateTablet for tablet e0eb928812a24d1cb82e00b549b36516 (DEFAULT_TABLE table=default.Table_2f5e51d6ee144f3cbda8aabaee177251 [id=65fca5e9756c40808bbfbc4b436e40d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.049978 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0eb928812a24d1cb82e00b549b36516. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.051506 24174 tablet_bootstrap.cc:654] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.035436 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 0ae92276246c4d39b194d7e9b868fc02 (DEFAULT_TABLE table=default.Table_91a1219d5c2542d9a45c7b8499786337 [id=a96384f644c64d3184405e02fe3d2d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.054026 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ae92276246c4d39b194d7e9b868fc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.035928 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 33bbebbca7464ea6ad59b81d7b7c560d (DEFAULT_TABLE table=default.Table_08a0e708a9ad4c619295ddc6c62e1a83 [id=563645aa3e16496bb1b7f1fd8f9c2132]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.057891 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33bbebbca7464ea6ad59b81d7b7c560d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:45.057920 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 89f57e3a39d54b7ab372d4b145b7a696 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89f57e3a39d54b7ab372d4b145b7a696 already in progress: creating tablet
W20240502 08:16:45.058228 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 29f82ad5860843f1adfbb40cdb21a4a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29f82ad5860843f1adfbb40cdb21a4a3 already in progress: creating tablet
W20240502 08:16:45.058461 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 63e645bb6e2948e091e08433944a52f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63e645bb6e2948e091e08433944a52f4 already in progress: creating tablet
W20240502 08:16:45.058652 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 643ce929e74c499da835fbe9b63b8880 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 643ce929e74c499da835fbe9b63b8880 already in progress: creating tablet
W20240502 08:16:45.058866 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet a198c33a3e314b5b81a08244862b281a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a198c33a3e314b5b81a08244862b281a already in progress: creating tablet
I20240502 08:16:45.059031 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 5c95dfda9d3b4e2ca9397b77ad71c011 (table default.Table_c201704a319a4e078054899061f76615 [id=71234c2c912b4a79b7b08d87cab6fad0]) successfully deleted
I20240502 08:16:45.061096 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e61ff7f9f4074074885941b862adf04d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:45.061501 23750 tablet_service.cc:1467] Processing CreateTablet for tablet abdcdf7db3e34e77980d4bcc07bcd090 (DEFAULT_TABLE table=default.Table_01f87df24c614973b564898fe3bfe511 [id=0746ccdaa1f44f00a4f3a4e4d8b2946c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.062042 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 397003e47c78418182e0167c3b876d4e (DEFAULT_TABLE table=default.Table_c2b1aa8d133542ffb45da2eacecf427b [id=1b4287236fcc4d4abf9d009d14bf8dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.062193 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4fb53d5cc9465888dce5ebb7a79379 (DEFAULT_TABLE table=default.Table_b3315254654d4234a879012f47feeb19 [id=7cac81e039f04d46afd1f05eaa52acef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.062449 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 0d446aae7dec4734b112490ed77f26a1 (DEFAULT_TABLE table=default.Table_8eddc9a260a04c339e7bb130c9822c62 [id=8b7bfc79afbf4effa6a8ed7204065aec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.062532 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13cbab4352bb44f389d70d4379e92457. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.062687 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8bae47845bfb4c78804a850866569860" } modify_external_catalogs: true
I20240502 08:16:45.062852 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdcdf7db3e34e77980d4bcc07bcd090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.062880 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 54619769ce264106b1ee3765572f13ba (DEFAULT_TABLE table=default.Table_0f88db2e7c844740bcb3df0fb75508b8 [id=0135506624894f12bcf045ece0cec5b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.062891 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8bae47845bfb4c78804a850866569860" } modify_external_catalogs: true
I20240502 08:16:45.063046 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 397003e47c78418182e0167c3b876d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.063328 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 73d7633fd23541f3a553af008dd6a354 (DEFAULT_TABLE table=default.Table_19538f34f25a4fd79dd6f076de5c021e [id=c8e08fefe5c14979a06c8780d4bb5687]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.063526 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ae92276246c4d39b194d7e9b868fc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.063756 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 25f06e16c1ae4ed7875ab463071581a5 (DEFAULT_TABLE table=default.Table_7991f9ca2d6f4a79aa0aeb12b23f1000 [id=6a34778cf31f4e089bedcc1d9fc4aa0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.064163 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 0ae92276246c4d39b194d7e9b868fc02 (DEFAULT_TABLE table=default.Table_91a1219d5c2542d9a45c7b8499786337 [id=a96384f644c64d3184405e02fe3d2d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.064404 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8f820382d8484e16834f76af57d0ab45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:45.064602 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 33bbebbca7464ea6ad59b81d7b7c560d (DEFAULT_TABLE table=default.Table_08a0e708a9ad4c619295ddc6c62e1a83 [id=563645aa3e16496bb1b7f1fd8f9c2132]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:45.064971 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f820382d8484e16834f76af57d0ab45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.065042 23767 tablet_service.cc:1467] Processing CreateTablet for tablet e9133e69e1724a26bc3650ea658ca349 (DEFAULT_TABLE table=default.Table_44bdfbf3075343e0a2ea5b79c0ca93e5 [id=513068e8e8fa48898966cbe95fa52422]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.065515 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 13cbab4352bb44f389d70d4379e92457 (DEFAULT_TABLE table=default.Table_8ac26bacf7bc4a05bacf7f35c2755a74 [id=a29e7a86803442cfb88b97e0575c3700]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:45.065960 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e61ff7f9f4074074885941b862adf04d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.066002 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ee8e3c4f16e74576b385099a280d5312"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:45.066404 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_dcff63ecc4f740fd999047021472be23" } new_table_name: "default.Table_b532c18aef184e139289a03b8f0d3c48" modify_external_catalogs: true
W20240502 08:16:45.066536 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee8e3c4f16e74576b385099a280d5312 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.067631 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_95df2e76edc746a08164b4ff25809909" } modify_external_catalogs: true
I20240502 08:16:45.067749 23756 tablet_service.cc:1467] Processing CreateTablet for tablet f87fe1c2c57748f0a17ee14503890c2f (DEFAULT_TABLE table=default.Table_28e9d1b5045f4818a01c228a3d25994d [id=0a89aa8a42b24f4d93b37780eeeeec4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.067824 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_95df2e76edc746a08164b4ff25809909" } modify_external_catalogs: true
I20240502 08:16:45.068563 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cc80d1cedafa4e71906e5e9530fdd828" } new_table_name: "default.Table_af464bf1ebe54d7281005831aa79b8bf" modify_external_catalogs: true
I20240502 08:16:45.070125 24780 raft_consensus.cc:482] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:45.070281 24780 raft_consensus.cc:504] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:45.071010 24780 leader_election.cc:288] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:45.071609 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54619769ce264106b1ee3765572f13ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.077418 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f06e16c1ae4ed7875ab463071581a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.078878 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d7633fd23541f3a553af008dd6a354. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.081837 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f06e16c1ae4ed7875ab463071581a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.085438 23892 tablet_service.cc:1467] Processing CreateTablet for tablet f87fe1c2c57748f0a17ee14503890c2f (DEFAULT_TABLE table=default.Table_28e9d1b5045f4818a01c228a3d25994d [id=0a89aa8a42b24f4d93b37780eeeeec4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.085855 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ae92276246c4d39b194d7e9b868fc02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.087085 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4fb53d5cc9465888dce5ebb7a79379 (DEFAULT_TABLE table=default.Table_b3315254654d4234a879012f47feeb19 [id=7cac81e039f04d46afd1f05eaa52acef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.090170 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33bbebbca7464ea6ad59b81d7b7c560d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.090566 24798 tablet.cc:1722] T 36f2d7b6f4e442ae95b2d1002d571b6a P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:45.092749 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f87fe1c2c57748f0a17ee14503890c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.093362 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4fb53d5cc9465888dce5ebb7a79379. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.036620 24031 tablet_service.cc:1467] Processing CreateTablet for tablet e9133e69e1724a26bc3650ea658ca349 (DEFAULT_TABLE table=default.Table_44bdfbf3075343e0a2ea5b79c0ca93e5 [id=513068e8e8fa48898966cbe95fa52422]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.037040 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 13cbab4352bb44f389d70d4379e92457 (DEFAULT_TABLE table=default.Table_8ac26bacf7bc4a05bacf7f35c2755a74 [id=a29e7a86803442cfb88b97e0575c3700]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.094360 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9133e69e1724a26bc3650ea658ca349. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.094640 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13cbab4352bb44f389d70d4379e92457. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.037469 24026 tablet_service.cc:1467] Processing CreateTablet for tablet f87fe1c2c57748f0a17ee14503890c2f (DEFAULT_TABLE table=default.Table_28e9d1b5045f4818a01c228a3d25994d [id=0a89aa8a42b24f4d93b37780eeeeec4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.097833 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4fb53d5cc9465888dce5ebb7a79379. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.097872 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f87fe1c2c57748f0a17ee14503890c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.038105 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4fb53d5cc9465888dce5ebb7a79379 (DEFAULT_TABLE table=default.Table_b3315254654d4234a879012f47feeb19 [id=7cac81e039f04d46afd1f05eaa52acef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.102475 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4fb53d5cc9465888dce5ebb7a79379. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.104343 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f87fe1c2c57748f0a17ee14503890c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.106163 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9133e69e1724a26bc3650ea658ca349. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.107515 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fac7df16fc41d7a3297055a7258270 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.108058 24773 tablet.cc:1722] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240502 08:16:45.108731 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d446aae7dec4734b112490ed77f26a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.110749 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13cbab4352bb44f389d70d4379e92457. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.112672 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 397003e47c78418182e0167c3b876d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.113533 23761 tablet_service.cc:1467] Processing CreateTablet for tablet e0eb928812a24d1cb82e00b549b36516 (DEFAULT_TABLE table=default.Table_2f5e51d6ee144f3cbda8aabaee177251 [id=65fca5e9756c40808bbfbc4b436e40d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.119771 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c4e75358850844d688b8d73770caec30" } new_table_name: "default.Table_d7cff78803d44e5aba3b50cd8ff11e92" modify_external_catalogs: true
I20240502 08:16:45.120618 23602 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 7a19310948e248d1a85479c41c61bda2 (table default.Table_785d5ddac96746379d1117bc8379566b [id=8521bfc13ef040a78cfb2cc4501b7f04]). Expected version 4 got 0
I20240502 08:16:45.121791 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_24d1bd1b4dcc4882884e13873cf896ff" } modify_external_catalogs: true
I20240502 08:16:45.122010 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_24d1bd1b4dcc4882884e13873cf896ff" } modify_external_catalogs: true
I20240502 08:16:45.123919 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0eb928812a24d1cb82e00b549b36516. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.130162 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_4179387181314b89ba7db097b4ee2318"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.130724 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4179387181314b89ba7db097b4ee2318 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.131225 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_480b6f612ca64d2caaa95f87d6ce8464" } new_table_name: "default.Table_c2a25d0d51cd40769e5045ff80c9c3fc" modify_external_catalogs: true
I20240502 08:16:45.136660 24753 raft_consensus.cc:482] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:45.136821 24753 raft_consensus.cc:504] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.137624 24753 leader_election.cc:288] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:45.138465 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_02b3fe4baa274e408220d9d27d9db7fa" } modify_external_catalogs: true
I20240502 08:16:45.138852 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet d47b9c1aa6da4ca4975d863d44946e32
I20240502 08:16:45.139487 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1991a52161ad40d0bbccd98a692364a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.140216 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1991a52161ad40d0bbccd98a692364a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.141278 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet f5ccd43e78574f3288b199fa14bcc0e8
I20240502 08:16:45.141980 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_02b3fe4baa274e408220d9d27d9db7fa" } modify_external_catalogs: true
I20240502 08:16:45.142745 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3e3f39f737b84f009fc0e2b1b1da6468" } modify_external_catalogs: true
I20240502 08:16:45.142897 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3e3f39f737b84f009fc0e2b1b1da6468" } modify_external_catalogs: true
I20240502 08:16:45.143844 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_59c11afb388848da8ac134294bc6c485" } new_table_name: "default.Table_e6b05d7b8b474e1bb7d208fd747bca47" modify_external_catalogs: true
I20240502 08:16:45.146140 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet d47b9c1aa6da4ca4975d863d44946e32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.148172 23921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8810b6f61db14fed8b12d36fed413dc8" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:45.148672 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet c08228fcd1304528b9446003851f6c70 with delete_type TABLET_DATA_DELETED (replaced by tablet b59c2f93d6514c9cbf358ab96f1a06b5) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.148988 24715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.84:40725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.22.225.126:43265,127.22.225.125:36189,127.22.225.124:41745
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.125:36189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.225.125
--webserver_port=41449
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:45.149780 24813 tablet.cc:1722] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240502 08:16:45.152205 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0882b7509a74475780eb700818aab398" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:45.153508 23993 leader_election.cc:341] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:45.155376 24753 raft_consensus.cc:482] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:45.155573 24753 raft_consensus.cc:504] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.156136 24175 tablet_bootstrap.cc:492] T e0af5f4daf3149738647a08219efb245 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:45.156211 24753 leader_election.cc:288] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:45.156275 24175 ts_tablet_manager.cc:1367] T e0af5f4daf3149738647a08219efb245 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.281s user 0.003s sys 0.001s
I20240502 08:16:45.156618 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f7b5f018d84680ac5533bf99102d31 with delete_type TABLET_DATA_DELETED (replaced by tablet c17e1be6ef294769a01981f4d361be21) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:45.156626 24715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:45.154327 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0882b7509a74475780eb700818aab398" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:45.158893 24715 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:45.159217 23723 leader_election.cc:341] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:45.161196 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet f5ccd43e78574f3288b199fa14bcc0e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:45.162894 23996 leader_election.cc:341] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:45.163054 23996 leader_election.cc:302] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:45.163592 24753 raft_consensus.cc:2726] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:45.164460 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcfacc302ff04178bfffc48111cd7a3f" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:45.165213 23993 leader_election.cc:341] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:45.167469 24779 raft_consensus.cc:482] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:45.167603 24779 raft_consensus.cc:504] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.168186 24779 leader_election.cc:288] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:45.173348 23996 leader_election.cc:341] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:45.173527 23996 leader_election.cc:302] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:45.174125 24753 raft_consensus.cc:2726] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:45.175711 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet f5ccd43e78574f3288b199fa14bcc0e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.177120 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5453cc0b93ac4cbb90a0588bb2100d60" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:45.177647 23723 leader_election.cc:341] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:45.170514 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcfacc302ff04178bfffc48111cd7a3f" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:45.181234 23996 leader_election.cc:341] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:45.181382 23996 leader_election.cc:302] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:45.182037 24753 raft_consensus.cc:2726] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:45.182184 23888 tablet_service.cc:1514] Processing DeleteTablet for tablet d47b9c1aa6da4ca4975d863d44946e32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:45.038748 24180 raft_consensus.cc:2258] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:45.196462 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e36be367f9b4c568ce0f3278b6297d9 consensus state has no leader
W20240502 08:16:45.197048 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 001e3f4f72f64dc69dbd521724912f07 consensus state has no leader
I20240502 08:16:45.201292 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet d47b9c1aa6da4ca4975d863d44946e32 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:45.201668 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet f5ccd43e78574f3288b199fa14bcc0e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:45.204712 24813 tablet.cc:1717] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b: Skipping requested alter to schema version 4, tablet already version 4
I20240502 08:16:45.205529 24175 raft_consensus.cc:348] T e0af5f4daf3149738647a08219efb245 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:45.205871 24175 raft_consensus.cc:374] T e0af5f4daf3149738647a08219efb245 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.205876 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 5453cc0b93ac4cbb90a0588bb2100d60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.206051 24175 raft_consensus.cc:729] T e0af5f4daf3149738647a08219efb245 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.209064 24175 consensus_queue.cc:260] T e0af5f4daf3149738647a08219efb245 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:45.213950 24773 tablet.cc:1717] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2: Skipping requested alter to schema version 4, tablet already version 4
I20240502 08:16:45.219786 23604 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 7a19310948e248d1a85479c41c61bda2 (table default.Table_785d5ddac96746379d1117bc8379566b [id=8521bfc13ef040a78cfb2cc4501b7f04]). Expected version 4 got 0
W20240502 08:16:45.222123 24831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:45.222435 24175 ts_tablet_manager.cc:1398] T e0af5f4daf3149738647a08219efb245 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.066s user 0.002s sys 0.000s
I20240502 08:16:45.227504 24175 tablet_bootstrap.cc:492] T 9700ea9dc58442e3ab1b7835a8c1e22c P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:45.231624 24837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:45.232419 24715 server_base.cc:1034] running on GCE node
W20240502 08:16:45.232969 24830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:45.240312 24715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20240502 08:16:45.244765 23723 consensus_peers.cc:485] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68bdb761ff36426c96949e97279e2756. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:45.246553 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 0d446aae7dec4734b112490ed77f26a1 (DEFAULT_TABLE table=default.Table_8eddc9a260a04c339e7bb130c9822c62 [id=8b7bfc79afbf4effa6a8ed7204065aec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.247113 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d446aae7dec4734b112490ed77f26a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.247735 24180 ts_tablet_manager.cc:1875] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.251489 24174 tablet_bootstrap.cc:492] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:45.253123 24179 ts_tablet_manager.cc:1888] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.254524 24174 ts_tablet_manager.cc:1367] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.208s user 0.018s sys 0.018s
I20240502 08:16:45.255486 24174 raft_consensus.cc:348] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.255746 24174 raft_consensus.cc:374] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.255843 24174 raft_consensus.cc:729] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.256145 24179 log.cc:1196] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/f36aadfa1d264f23bcafa91744aed36d
I20240502 08:16:45.256188 24174 consensus_queue.cc:260] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.256726 24175 tablet_bootstrap.cc:654] T 9700ea9dc58442e3ab1b7835a8c1e22c P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.258607 24174 ts_tablet_manager.cc:1398] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:16:45.259006 24174 tablet_bootstrap.cc:492] T 848a0865eb2b4828a2594a17d96869d0 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:45.259562 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet d9fcc7accb1f45ad9cf55b8f64d21b91
I20240502 08:16:45.260311 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet a17fb4f2f02e43dc9f7cfdb7c73f7774 from {username='slave'} at 127.0.0.1:36668
W20240502 08:16:45.260854 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3abfc0c885d64c39808a6e7c815afd7e consensus state has no leader
W20240502 08:16:45.261436 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a17fb4f2f02e43dc9f7cfdb7c73f7774 consensus state has no leader
I20240502 08:16:45.261734 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet ddcb9f236489402593d59ba801615103
I20240502 08:16:45.262763 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet d9fcc7accb1f45ad9cf55b8f64d21b91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.256675 24179 ts_tablet_manager.cc:1909] T f36aadfa1d264f23bcafa91744aed36d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:45.263128 23608 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_785d5ddac96746379d1117bc8379566b [id=8521bfc13ef040a78cfb2cc4501b7f04] alter complete (version 4)
I20240502 08:16:45.265127 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet ddcb9f236489402593d59ba801615103 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.267040 24175 tablet_bootstrap.cc:492] T 9700ea9dc58442e3ab1b7835a8c1e22c P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:45.267336 24175 ts_tablet_manager.cc:1367] T 9700ea9dc58442e3ab1b7835a8c1e22c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.000s
I20240502 08:16:45.268739 24175 raft_consensus.cc:348] T 9700ea9dc58442e3ab1b7835a8c1e22c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:45.269047 24175 raft_consensus.cc:374] T 9700ea9dc58442e3ab1b7835a8c1e22c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.269191 24175 raft_consensus.cc:729] T 9700ea9dc58442e3ab1b7835a8c1e22c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.269552 24175 consensus_queue.cc:260] T 9700ea9dc58442e3ab1b7835a8c1e22c P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:45.270748 24175 ts_tablet_manager.cc:1398] T 9700ea9dc58442e3ab1b7835a8c1e22c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:45.271468 24174 tablet_bootstrap.cc:654] T 848a0865eb2b4828a2594a17d96869d0 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.272130 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d28098dcbc4761a34ee65769d6fd7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.272284 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet ddcb9f236489402593d59ba801615103 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:45.272528 24715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:45.272663 24833 tablet.cc:1717] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2: Skipping requested alter to schema version 4, tablet already version 4
I20240502 08:16:45.273068 24798 tablet.cc:1717] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b: Skipping requested alter to schema version 4, tablet already version 4
I20240502 08:16:45.274469 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_20b409a694114049bf0d966535b376a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.275249 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20b409a694114049bf0d966535b376a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.275990 24715 hybrid_clock.cc:648] HybridClock initialized: now 1714637805275797 us; error 147 us; skew 500 ppm
I20240502 08:16:45.276417 24715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:45.276517 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5ffe8637fd9b4cd3b5b5aa0eb542b267"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.277009 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ffe8637fd9b4cd3b5b5aa0eb542b267 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.282464 24174 tablet_bootstrap.cc:492] T 848a0865eb2b4828a2594a17d96869d0 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:45.282651 24174 ts_tablet_manager.cc:1367] T 848a0865eb2b4828a2594a17d96869d0 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.005s
I20240502 08:16:45.283531 24174 raft_consensus.cc:348] T 848a0865eb2b4828a2594a17d96869d0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.283727 24174 raft_consensus.cc:374] T 848a0865eb2b4828a2594a17d96869d0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.283721 24180 ts_tablet_manager.cc:1888] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:16:45.283854 24174 raft_consensus.cc:729] T 848a0865eb2b4828a2594a17d96869d0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.283908 24180 log.cc:1196] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/dead13bba95f43cb809e17f6d7fba028
I20240502 08:16:45.284333 24180 ts_tablet_manager.cc:1909] T dead13bba95f43cb809e17f6d7fba028 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:45.285190 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 575) took 1746 ms. Trace:
I20240502 08:16:45.285385 24180 rpcz_store.cc:276] 0502 08:16:43.562464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.732602 (+170138us) service_pool.cc:225] Handling call
0502 08:16:45.037937 (+1305335us) ts_tablet_manager.cc:1120] Deleting tablet dead13bba95f43cb809e17f6d7fba028
0502 08:16:45.037950 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.273509 (+235559us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:45.283570 (+ 10061us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:45.285151 (+ 1581us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1305108}
I20240502 08:16:45.285434 24174 consensus_queue.cc:260] T 848a0865eb2b4828a2594a17d96869d0 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.286392 24174 ts_tablet_manager.cc:1398] T 848a0865eb2b4828a2594a17d96869d0 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:16:45.286823 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet d9fcc7accb1f45ad9cf55b8f64d21b91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:45.287766 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 578) took 1429 ms. Trace:
W20240502 08:16:45.288439 23723 consensus_peers.cc:485] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49b816ea1d474f2db94119441146ba78. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:45.289175 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet e0af5f4daf3149738647a08219efb245 with delete_type TABLET_DATA_DELETED (replaced by tablet b8b1fa17dad143f0a7f1751e651eb81a) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:45.293846 23723 leader_election.cc:341] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 8810b6f61db14fed8b12d36fed413dc8
I20240502 08:16:45.294005 23723 leader_election.cc:302] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:45.294111 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 4a0684db8dd947d79343afa5a2090614 (DEFAULT_TABLE table=default.Table_b5aef32596c94f51bb917536c2bc7341 [id=4a7bc331d5f840019bcc43dfd11f1fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.294433 24774 raft_consensus.cc:2726] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:45.294975 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 7d07efb303b64ce19d993c4c76a47dfc (DEFAULT_TABLE table=default.Table_6e14e5ad32c946d286dc2416e1a16f61 [id=fb99cdfbcdc74ed588f620e61a0dc24a]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:45.295071 23723 consensus_peers.cc:485] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68bdb761ff36426c96949e97279e2756. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:45.295539 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d07efb303b64ce19d993c4c76a47dfc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:45.296535 23723 leader_election.cc:341] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 5453cc0b93ac4cbb90a0588bb2100d60
I20240502 08:16:45.296682 23723 leader_election.cc:302] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:45.296909 24774 raft_consensus.cc:2726] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:45.299309 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a0684db8dd947d79343afa5a2090614. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.301124 24798 tablet.cc:1722] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:45.288040 24180 rpcz_store.cc:276] 0502 08:16:43.858311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.858793 (+ 482us) service_pool.cc:225] Handling call
0502 08:16:45.286012 (+1427219us) ts_tablet_manager.cc:1120] Deleting tablet 523603a31b5b49e0a18a5a64f1357d64
0502 08:16:45.287604 (+ 1592us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.287723 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1426936}
I20240502 08:16:45.310180 24179 tablet_replica.cc:330] T d8f7b5f018d84680ac5533bf99102d31 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:45.310366 24179 raft_consensus.cc:2229] T d8f7b5f018d84680ac5533bf99102d31 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:45.310511 24179 raft_consensus.cc:2258] T d8f7b5f018d84680ac5533bf99102d31 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:45.310954 24179 ts_tablet_manager.cc:1875] T d8f7b5f018d84680ac5533bf99102d31 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.311890 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 581) took 1453 ms. Trace:
I20240502 08:16:45.312017 24180 rpcz_store.cc:276] 0502 08:16:43.858625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.859483 (+ 858us) service_pool.cc:225] Handling call
0502 08:16:45.310574 (+1451091us) ts_tablet_manager.cc:1120] Deleting tablet 9467974d67c64c7eb5e488ac51f1240d
0502 08:16:45.310586 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.311854 (+ 1268us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1450844}
I20240502 08:16:45.313582 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 576) took 1455 ms. Trace:
I20240502 08:16:45.313694 24180 rpcz_store.cc:276] 0502 08:16:43.858063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.859762 (+ 1699us) service_pool.cc:225] Handling call
0502 08:16:45.313418 (+1453656us) ts_tablet_manager.cc:1120] Deleting tablet 0eb7f36f7c9b47ebb93f1d846fd7352c
0502 08:16:45.313429 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.313534 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1453161}
I20240502 08:16:45.313908 24175 tablet_bootstrap.cc:492] T 8810b6f61db14fed8b12d36fed413dc8 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:45.315464 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 577) took 1457 ms. Trace:
I20240502 08:16:45.315495 24175 tablet_bootstrap.cc:654] T 8810b6f61db14fed8b12d36fed413dc8 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.315573 24180 rpcz_store.cc:276] 0502 08:16:43.858206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.867396 (+ 9190us) service_pool.cc:225] Handling call
0502 08:16:45.315334 (+1447938us) ts_tablet_manager.cc:1120] Deleting tablet 15007d576b064c2cb0a005cb9f7ea1cd
0502 08:16:45.315344 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.315432 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1447656}
I20240502 08:16:45.317732 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 579) took 1459 ms. Trace:
I20240502 08:16:45.318514 24180 rpcz_store.cc:276] 0502 08:16:43.858385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:43.911260 (+ 52875us) service_pool.cc:225] Handling call
0502 08:16:45.317563 (+1406303us) ts_tablet_manager.cc:1120] Deleting tablet e726e0ef49b247bebe07a7e7066c1af6
0502 08:16:45.317573 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.317696 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1405389}
I20240502 08:16:45.319538 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 592) took 1180 ms. Trace:
I20240502 08:16:45.320693 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_3d50b45ce5014fafae3e3a7dbe1d25ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.321276 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d50b45ce5014fafae3e3a7dbe1d25ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.321817 24180 rpcz_store.cc:276] 0502 08:16:44.138743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.139955 (+ 1212us) service_pool.cc:225] Handling call
0502 08:16:45.319424 (+1179469us) ts_tablet_manager.cc:1120] Deleting tablet c1e746013fde4de4862f213792e235ed
0502 08:16:45.319432 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.319510 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1179278}
I20240502 08:16:45.322314 24174 tablet_bootstrap.cc:492] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:45.323065 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 22240da6da2241bd93b6547a0aae917e
I20240502 08:16:45.323557 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet a17fb4f2f02e43dc9f7cfdb7c73f7774
I20240502 08:16:45.324061 23602 catalog_manager.cc:6554] ReplaceTablet: tablet a17fb4f2f02e43dc9f7cfdb7c73f7774 deleted and replaced by tablet 7a23c06cb3704a7b83143a02efe2a8b3
I20240502 08:16:45.328161 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5c021393c18a4030b8894b73c1b8acff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.328811 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c021393c18a4030b8894b73c1b8acff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.329761 24174 tablet_bootstrap.cc:654] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.331292 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet d47b9c1aa6da4ca4975d863d44946e32 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:45.331622 24179 ts_tablet_manager.cc:1888] T d8f7b5f018d84680ac5533bf99102d31 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.331740 24179 log.cc:1196] T d8f7b5f018d84680ac5533bf99102d31 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/d8f7b5f018d84680ac5533bf99102d31
I20240502 08:16:45.332069 24179 ts_tablet_manager.cc:1909] T d8f7b5f018d84680ac5533bf99102d31 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:45.332487 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 593) took 1193 ms. Trace:
I20240502 08:16:45.332968 24179 tablet_replica.cc:330] T e0af5f4daf3149738647a08219efb245 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:45.333158 24179 raft_consensus.cc:2229] T e0af5f4daf3149738647a08219efb245 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:45.333329 24179 raft_consensus.cc:2258] T e0af5f4daf3149738647a08219efb245 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:45.333541 24180 rpcz_store.cc:276] 0502 08:16:44.138811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.140181 (+ 1370us) service_pool.cc:225] Handling call
0502 08:16:45.332280 (+1192099us) ts_tablet_manager.cc:1120] Deleting tablet e9171ebf74424c9aae5e1c8a5118f768
0502 08:16:45.332295 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.332445 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1191962}
I20240502 08:16:45.339154 24175 tablet_bootstrap.cc:492] T 8810b6f61db14fed8b12d36fed413dc8 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:45.339370 24175 ts_tablet_manager.cc:1367] T 8810b6f61db14fed8b12d36fed413dc8 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.002s
I20240502 08:16:45.339996 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet dead13bba95f43cb809e17f6d7fba028 (table default.Table_ef79f64b4af44b88a006a32b437c3207 [id=7409c7e1a3534932a787559a679e7098]) successfully deleted
W20240502 08:16:45.340232 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 523603a31b5b49e0a18a5a64f1357d64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 523603a31b5b49e0a18a5a64f1357d64 already in progress: creating tablet
W20240502 08:16:45.340411 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 9467974d67c64c7eb5e488ac51f1240d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9467974d67c64c7eb5e488ac51f1240d already in progress: creating tablet
W20240502 08:16:45.340581 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 0eb7f36f7c9b47ebb93f1d846fd7352c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0eb7f36f7c9b47ebb93f1d846fd7352c already in progress: creating tablet
W20240502 08:16:45.340757 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 15007d576b064c2cb0a005cb9f7ea1cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15007d576b064c2cb0a005cb9f7ea1cd already in progress: creating tablet
W20240502 08:16:45.340920 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e726e0ef49b247bebe07a7e7066c1af6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e726e0ef49b247bebe07a7e7066c1af6 already in progress: creating tablet
W20240502 08:16:45.341099 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet c1e746013fde4de4862f213792e235ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1e746013fde4de4862f213792e235ed already in progress: creating tablet
W20240502 08:16:45.341338 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e9171ebf74424c9aae5e1c8a5118f768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9171ebf74424c9aae5e1c8a5118f768 already in progress: creating tablet
I20240502 08:16:45.342190 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8e0659186349e1a02afc4f25ae9cff (DEFAULT_TABLE table=default.Table_f26e93806daa446280d6312001630c82 [id=306fe7ee33a747298ba42f338459ef6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.342737 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b8e0659186349e1a02afc4f25ae9cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.342736 24175 raft_consensus.cc:348] T 8810b6f61db14fed8b12d36fed413dc8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:45.343139 24175 raft_consensus.cc:374] T 8810b6f61db14fed8b12d36fed413dc8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.343248 24175 raft_consensus.cc:729] T 8810b6f61db14fed8b12d36fed413dc8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.343602 24179 ts_tablet_manager.cc:1875] T e0af5f4daf3149738647a08219efb245 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.343525 24175 consensus_queue.cc:260] T 8810b6f61db14fed8b12d36fed413dc8 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:45.344277 24175 ts_tablet_manager.cc:1398] T 8810b6f61db14fed8b12d36fed413dc8 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:16:45.344568 24175 tablet_bootstrap.cc:492] T 01675f637a5a4645949abfd87b3d09a2 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:45.345890 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 28a947b7e7484120b3f4069f6eef5520 (DEFAULT_TABLE table=default.Table_cda7da2a134f43a8a3c79a6609c3e52e [id=07050bd496e84a97a86fdc4f0b7811ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.346154 24175 tablet_bootstrap.cc:654] T 01675f637a5a4645949abfd87b3d09a2 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.346231 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28a947b7e7484120b3f4069f6eef5520. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.352635 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 4a0684db8dd947d79343afa5a2090614 (DEFAULT_TABLE table=default.Table_b5aef32596c94f51bb917536c2bc7341 [id=4a7bc331d5f840019bcc43dfd11f1fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.353060 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a0684db8dd947d79343afa5a2090614. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.354178 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet ddcb9f236489402593d59ba801615103 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:45.355089 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8e0659186349e1a02afc4f25ae9cff (DEFAULT_TABLE table=default.Table_f26e93806daa446280d6312001630c82 [id=306fe7ee33a747298ba42f338459ef6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.355609 24021 tablet_service.cc:1467] Processing CreateTablet for tablet 7d07efb303b64ce19d993c4c76a47dfc (DEFAULT_TABLE table=default.Table_6e14e5ad32c946d286dc2416e1a16f61 [id=fb99cdfbcdc74ed588f620e61a0dc24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.357242 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet d9fcc7accb1f45ad9cf55b8f64d21b91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:45.357648 24032 tablet_service.cc:1467] Processing CreateTablet for tablet c234ea188e104d87a951bc195e968c0a (DEFAULT_TABLE table=default.Table_0692aa66a8ad444eab3af22ad78c527e [id=bebf071aca394bf2932b9e89ff110ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.358008 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 28a947b7e7484120b3f4069f6eef5520 (DEFAULT_TABLE table=default.Table_cda7da2a134f43a8a3c79a6609c3e52e [id=07050bd496e84a97a86fdc4f0b7811ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.358304 24181 ts_tablet_manager.cc:1888] T 71aa916247c449fea718b35869a6df20 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.358395 24181 log.cc:1196] T 71aa916247c449fea718b35869a6df20 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/71aa916247c449fea718b35869a6df20
I20240502 08:16:45.358652 24181 ts_tablet_manager.cc:1909] T 71aa916247c449fea718b35869a6df20 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:45.359462 24181 tablet_replica.cc:330] T 643ce929e74c499da835fbe9b63b8880 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:45.359616 24181 raft_consensus.cc:2229] T 643ce929e74c499da835fbe9b63b8880 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:45.359748 24181 raft_consensus.cc:2258] T 643ce929e74c499da835fbe9b63b8880 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:45.360229 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d07efb303b64ce19d993c4c76a47dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.360324 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_134e3d6f761d4656acd584556c172e39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:45.360388 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_4ee153a309b04ab68b03ab96694a1952"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:45.360870 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28a947b7e7484120b3f4069f6eef5520. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:45.360996 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ee153a309b04ab68b03ab96694a1952 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.361114 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b8e0659186349e1a02afc4f25ae9cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.361485 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c234ea188e104d87a951bc195e968c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.361392 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_93bd02b663b8448aa250379544e73dea" } modify_external_catalogs: true
I20240502 08:16:45.361652 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_93bd02b663b8448aa250379544e73dea" } modify_external_catalogs: true
I20240502 08:16:45.362123 24181 ts_tablet_manager.cc:1875] T 643ce929e74c499da835fbe9b63b8880 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.362228 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8040b4c1a36844658c4cdf62e9ae61de" } new_table_name: "default.Table_0870fd03a99f4f35b03dcd76a38b4d78" modify_external_catalogs: true
I20240502 08:16:45.362455 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_4114c8f606d84b01963b9e68bb1571f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:45.362838 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 71aa916247c449fea718b35869a6df20 (table default.Table_9a9569245562493dafbeb639c6492826 [id=5cee878b3a31426c879a1bf95045bb09]) successfully deleted
W20240502 08:16:45.363073 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4114c8f606d84b01963b9e68bb1571f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.363746 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b3278c5d6d0b4593ac33bc9977f5c3b1" } modify_external_catalogs: true
I20240502 08:16:45.363981 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b3278c5d6d0b4593ac33bc9977f5c3b1" } modify_external_catalogs: true
W20240502 08:16:45.364403 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_134e3d6f761d4656acd584556c172e39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.360415 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_61f78001376f4d549a6da83157648884"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.365129 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61f78001376f4d549a6da83157648884 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.368870 24033 tablet_service.cc:1467] Processing CreateTablet for tablet ae2b20ae3a014c85af33d5ee51b704e1 (DEFAULT_TABLE table=default.Table_5bff1e299251446f91a962b909f49a42 [id=07b4ec205e8a47d2a5d3b46ce99e8a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.369161 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 22240da6da2241bd93b6547a0aae917e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:45.370905 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae2b20ae3a014c85af33d5ee51b704e1. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:45.371511 23723 consensus_peers.cc:485] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ce5ccfd29724dc49afd0892b99f3169. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:45.372339 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 594) took 1233 ms. Trace:
I20240502 08:16:45.372704 24180 rpcz_store.cc:276] 0502 08:16:44.138897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.140330 (+ 1433us) service_pool.cc:225] Handling call
0502 08:16:45.370415 (+1230085us) ts_tablet_manager.cc:1120] Deleting tablet 9b780a7870044d94bab14f0d9c978a0c
0502 08:16:45.370428 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.370558 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1229951}
I20240502 08:16:45.372998 23608 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_af464bf1ebe54d7281005831aa79b8bf [id=47682c26a85842c7b8991bf3828a9336] alter complete (version 2)
I20240502 08:16:45.373394 23608 master_service.cc:946] ReplaceTablet: received request to replace tablet 955e7bd2216646c6aef5acce8d5b0d77 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:45.373682 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet a17fb4f2f02e43dc9f7cfdb7c73f7774 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a23c06cb3704a7b83143a02efe2a8b3) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:45.374924 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 595) took 1235 ms. Trace:
I20240502 08:16:45.375131 24180 rpcz_store.cc:276] 0502 08:16:44.138961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.140475 (+ 1514us) service_pool.cc:225] Handling call
0502 08:16:45.374234 (+1233759us) ts_tablet_manager.cc:1120] Deleting tablet b6d8410ba7cc4b03bb64a138b1f9bb14
0502 08:16:45.374247 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.374906 (+ 659us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1233643}
I20240502 08:16:45.375469 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 596) took 1236 ms. Trace:
I20240502 08:16:45.375643 24180 rpcz_store.cc:276] 0502 08:16:44.139038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.140609 (+ 1571us) service_pool.cc:225] Handling call
0502 08:16:45.375362 (+1234753us) ts_tablet_manager.cc:1120] Deleting tablet fddac683eda14ec1a0140926380f2c07
0502 08:16:45.375371 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.375453 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1222516}
I20240502 08:16:45.375984 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 597) took 1236 ms. Trace:
I20240502 08:16:45.376057 24180 rpcz_store.cc:276] 0502 08:16:44.139127 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.140688 (+ 1561us) service_pool.cc:225] Handling call
0502 08:16:45.375859 (+1235171us) ts_tablet_manager.cc:1120] Deleting tablet a5a5a410b6314818b10927e35032f95d
0502 08:16:45.375868 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.375967 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1222856}
I20240502 08:16:45.377622 24179 ts_tablet_manager.cc:1888] T e0af5f4daf3149738647a08219efb245 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.377836 24179 log.cc:1196] T e0af5f4daf3149738647a08219efb245 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/e0af5f4daf3149738647a08219efb245
I20240502 08:16:45.378188 24179 ts_tablet_manager.cc:1909] T e0af5f4daf3149738647a08219efb245 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:45.381276 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 605) took 1185 ms. Trace:
I20240502 08:16:45.382652 24180 rpcz_store.cc:276] 0502 08:16:44.196113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.196180 (+ 67us) service_pool.cc:225] Handling call
0502 08:16:45.376253 (+1180073us) ts_tablet_manager.cc:1120] Deleting tablet 4645143a00d440a7b6d56c5e30d249f1
0502 08:16:45.376262 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.381249 (+ 4987us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1167500}
I20240502 08:16:45.384619 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet f36aadfa1d264f23bcafa91744aed36d (table default.Table_1136fadacc774d899c0461e8a0579376 [id=e18870bff56747b192b4698f1ca3a8ba]) successfully deleted
W20240502 08:16:45.386046 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 23ba091d8efb44858bf90a104c0f442f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23ba091d8efb44858bf90a104c0f442f already in progress: creating tablet
I20240502 08:16:45.386121 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 611) took 1159 ms. Trace:
I20240502 08:16:45.386205 24180 rpcz_store.cc:276] 0502 08:16:44.226869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.227469 (+ 600us) service_pool.cc:225] Handling call
0502 08:16:45.385989 (+1158520us) ts_tablet_manager.cc:1120] Deleting tablet fbcf17a3bcc844b1b9676580f0749c9c
0502 08:16:45.385999 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.386103 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1156822}
W20240502 08:16:45.386794 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 92adebc0c69e4a52b975253656132495 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92adebc0c69e4a52b975253656132495 already in progress: creating tablet
W20240502 08:16:45.387287 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 592eeecbf5514215b6464bd030081e4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 592eeecbf5514215b6464bd030081e4b already in progress: creating tablet
I20240502 08:16:45.387943 24180 tablet_replica.cc:330] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:45.388532 24180 raft_consensus.cc:2229] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:45.382427 24174 tablet_bootstrap.cc:492] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:45.391942 24174 ts_tablet_manager.cc:1367] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.000s
I20240502 08:16:45.392849 24174 raft_consensus.cc:348] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.394352 24174 raft_consensus.cc:374] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.394707 24174 raft_consensus.cc:729] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.393752 24180 raft_consensus.cc:2258] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:45.395879 24180 ts_tablet_manager.cc:1875] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:45.397063 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 48ae3bad6d3c48a6a067f16be24e224c consensus state has no leader
I20240502 08:16:45.398715 23608 master_service.cc:946] ReplaceTablet: received request to replace tablet ef03faf0a13348d7acad954256b5a1d2 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:45.409293 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5bff1e299251446f91a962b909f49a42" } new_table_name: "default.Table_af07f004e79840bebc030f84c14bb080" modify_external_catalogs: true
I20240502 08:16:45.411279 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_61f78001376f4d549a6da83157648884" } new_table_name: "default.Table_41a2401b3f4c4748b1d3eb964d8053d5" modify_external_catalogs: true
W20240502 08:16:45.413705 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet ef03faf0a13348d7acad954256b5a1d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef03faf0a13348d7acad954256b5a1d2 already in progress: creating tablet
W20240502 08:16:45.415521 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 91646e7af3824520a1fe28e180eeb0a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91646e7af3824520a1fe28e180eeb0a7 already in progress: creating tablet
I20240502 08:16:45.416317 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet d8f7b5f018d84680ac5533bf99102d31 (table default.Table_498b372c15a04b70b87bd67b543501cf [id=38e955d6bd644d24a7c45172ab60282b]) successfully deleted
I20240502 08:16:45.416702 23608 master_service.cc:946] ReplaceTablet: received request to replace tablet 7dadd4f432554022bc2fdc3c775ac9d1 from {username='slave'} at 127.0.0.1:36668
W20240502 08:16:45.417398 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet f5ccd43e78574f3288b199fa14bcc0e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5ccd43e78574f3288b199fa14bcc0e8 already in progress: creating tablet
W20240502 08:16:45.418148 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet d47b9c1aa6da4ca4975d863d44946e32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d47b9c1aa6da4ca4975d863d44946e32 already in progress: creating tablet
W20240502 08:16:45.418993 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet ddcb9f236489402593d59ba801615103 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddcb9f236489402593d59ba801615103 already in progress: creating tablet
W20240502 08:16:45.419658 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet d9fcc7accb1f45ad9cf55b8f64d21b91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9fcc7accb1f45ad9cf55b8f64d21b91 already in progress: creating tablet
I20240502 08:16:45.425259 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 4a0684db8dd947d79343afa5a2090614 (DEFAULT_TABLE table=default.Table_b5aef32596c94f51bb917536c2bc7341 [id=4a7bc331d5f840019bcc43dfd11f1fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.425801 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a0684db8dd947d79343afa5a2090614. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.397284 24174 consensus_queue.cc:260] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.431085 24174 ts_tablet_manager.cc:1398] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20240502 08:16:45.431430 24174 tablet_bootstrap.cc:492] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:45.433177 24174 tablet_bootstrap.cc:654] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.437076 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 28a947b7e7484120b3f4069f6eef5520 (DEFAULT_TABLE table=default.Table_cda7da2a134f43a8a3c79a6609c3e52e [id=07050bd496e84a97a86fdc4f0b7811ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.437150 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 7d07efb303b64ce19d993c4c76a47dfc (DEFAULT_TABLE table=default.Table_6e14e5ad32c946d286dc2416e1a16f61 [id=fb99cdfbcdc74ed588f620e61a0dc24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.437587 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28a947b7e7484120b3f4069f6eef5520. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.437907 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet 9700ea9dc58442e3ab1b7835a8c1e22c with delete_type TABLET_DATA_DELETED (replaced by tablet 5d04312bd81f46ed8c49c42b1af1aec9) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:45.438421 23887 tablet_service.cc:1467] Processing CreateTablet for tablet ae2b20ae3a014c85af33d5ee51b704e1 (DEFAULT_TABLE table=default.Table_5bff1e299251446f91a962b909f49a42 [id=07b4ec205e8a47d2a5d3b46ce99e8a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.438796 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet 22240da6da2241bd93b6547a0aae917e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:45.439112 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet a17fb4f2f02e43dc9f7cfdb7c73f7774 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a23c06cb3704a7b83143a02efe2a8b3) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:45.439353 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 9b780a7870044d94bab14f0d9c978a0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b780a7870044d94bab14f0d9c978a0c already in progress: creating tablet
W20240502 08:16:45.439754 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet b6d8410ba7cc4b03bb64a138b1f9bb14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6d8410ba7cc4b03bb64a138b1f9bb14 already in progress: creating tablet
W20240502 08:16:45.440110 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet fddac683eda14ec1a0140926380f2c07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fddac683eda14ec1a0140926380f2c07 already in progress: creating tablet
I20240502 08:16:45.441699 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet 8810b6f61db14fed8b12d36fed413dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:45.441852 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8e0659186349e1a02afc4f25ae9cff (DEFAULT_TABLE table=default.Table_f26e93806daa446280d6312001630c82 [id=306fe7ee33a747298ba42f338459ef6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.442189 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet e0af5f4daf3149738647a08219efb245 (table default.Table_d4bd37f8c6e54ecf82cd247846aa06d0 [id=57722a7341c541b19efa213bcf48eb53]) successfully deleted
I20240502 08:16:45.437603 23899 tablet_service.cc:1467] Processing CreateTablet for tablet c234ea188e104d87a951bc195e968c0a (DEFAULT_TABLE table=default.Table_0692aa66a8ad444eab3af22ad78c527e [id=bebf071aca394bf2932b9e89ff110ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.443444 24833 tablet.cc:1717] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2: Skipping requested alter to schema version 4, tablet already version 4
I20240502 08:16:45.445632 24798 tablet.cc:1717] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b: Skipping requested alter to schema version 4, tablet already version 4
I20240502 08:16:45.446081 24843 tablet.cc:1722] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:45.446197 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae2b20ae3a014c85af33d5ee51b704e1. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:45.450356 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet a5a5a410b6314818b10927e35032f95d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5a5a410b6314818b10927e35032f95d already in progress: creating tablet
W20240502 08:16:45.450860 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 4645143a00d440a7b6d56c5e30d249f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4645143a00d440a7b6d56c5e30d249f1 already in progress: creating tablet
W20240502 08:16:45.451139 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet fbcf17a3bcc844b1b9676580f0749c9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbcf17a3bcc844b1b9676580f0749c9c already in progress: creating tablet
I20240502 08:16:45.451654 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b8e0659186349e1a02afc4f25ae9cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.455389 24179 tablet_replica.cc:330] T 9700ea9dc58442e3ab1b7835a8c1e22c P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:45.455587 24179 raft_consensus.cc:2229] T 9700ea9dc58442e3ab1b7835a8c1e22c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:45.455726 24179 raft_consensus.cc:2258] T 9700ea9dc58442e3ab1b7835a8c1e22c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:45.456338 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c234ea188e104d87a951bc195e968c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.457620 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 848a0865eb2b4828a2594a17d96869d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.457904 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 22240da6da2241bd93b6547a0aae917e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.458313 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet a17fb4f2f02e43dc9f7cfdb7c73f7774 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a23c06cb3704a7b83143a02efe2a8b3) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.458451 23756 tablet_service.cc:1467] Processing CreateTablet for tablet c234ea188e104d87a951bc195e968c0a (DEFAULT_TABLE table=default.Table_0692aa66a8ad444eab3af22ad78c527e [id=bebf071aca394bf2932b9e89ff110ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.458896 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c234ea188e104d87a951bc195e968c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.461138 23753 tablet_service.cc:1467] Processing CreateTablet for tablet ae2b20ae3a014c85af33d5ee51b704e1 (DEFAULT_TABLE table=default.Table_5bff1e299251446f91a962b909f49a42 [id=07b4ec205e8a47d2a5d3b46ce99e8a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.461921 24179 ts_tablet_manager.cc:1875] T 9700ea9dc58442e3ab1b7835a8c1e22c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.461934 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae2b20ae3a014c85af33d5ee51b704e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.467247 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e6b05d7b8b474e1bb7d208fd747bca47" } modify_external_catalogs: true
I20240502 08:16:45.467535 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e6b05d7b8b474e1bb7d208fd747bca47" } modify_external_catalogs: true
I20240502 08:16:45.467769 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b532c18aef184e139289a03b8f0d3c48" } new_table_name: "default.Table_3ef466d7a87644e99070934d65e5c851" modify_external_catalogs: true
I20240502 08:16:45.468240 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_36e587dae1614924b2696665c53a1e6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.468802 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36e587dae1614924b2696665c53a1e6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.469502 24179 ts_tablet_manager.cc:1888] T 9700ea9dc58442e3ab1b7835a8c1e22c P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.469627 24179 log.cc:1196] T 9700ea9dc58442e3ab1b7835a8c1e22c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/9700ea9dc58442e3ab1b7835a8c1e22c
I20240502 08:16:45.469903 24179 ts_tablet_manager.cc:1909] T 9700ea9dc58442e3ab1b7835a8c1e22c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:45.470264 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_990c64d081c148d79dae73c2d06b1f11" } new_table_name: "default.Table_377dcfeddb374dfb8b977d526c8c77e4" modify_external_catalogs: true
I20240502 08:16:45.470788 24181 ts_tablet_manager.cc:1888] T 643ce929e74c499da835fbe9b63b8880 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.470894 24181 log.cc:1196] T 643ce929e74c499da835fbe9b63b8880 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/643ce929e74c499da835fbe9b63b8880
I20240502 08:16:45.471051 24179 tablet_replica.cc:330] T 8810b6f61db14fed8b12d36fed413dc8 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:45.471192 24179 raft_consensus.cc:2229] T 8810b6f61db14fed8b12d36fed413dc8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:45.471300 24179 raft_consensus.cc:2258] T 8810b6f61db14fed8b12d36fed413dc8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:45.472088 24179 ts_tablet_manager.cc:1875] T 8810b6f61db14fed8b12d36fed413dc8 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.473899 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d07efb303b64ce19d993c4c76a47dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.473901 24181 ts_tablet_manager.cc:1909] T 643ce929e74c499da835fbe9b63b8880 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:45.478246 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 9700ea9dc58442e3ab1b7835a8c1e22c (table default.Table_9741459a96e649fb8151cc28b642523e [id=2b9169f285754bfda0f4f8dc7f5cb653]) successfully deleted
W20240502 08:16:45.478797 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 22240da6da2241bd93b6547a0aae917e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22240da6da2241bd93b6547a0aae917e already in progress: creating tablet
I20240502 08:16:45.479018 24181 tablet_replica.cc:330] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:45.479204 24181 raft_consensus.cc:2229] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:45.479360 24181 raft_consensus.cc:2258] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:45.479631 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet a17fb4f2f02e43dc9f7cfdb7c73f7774 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a17fb4f2f02e43dc9f7cfdb7c73f7774 already in progress: creating tablet
I20240502 08:16:45.479823 24181 ts_tablet_manager.cc:1875] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.483886 24715 webserver.cc:460] Webserver started at http://127.22.225.125:41449/ using document root <none> and password file <none>
I20240502 08:16:45.486119 24715 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:45.487020 24843 tablet.cc:1722] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240502 08:16:45.487700 24179 ts_tablet_manager.cc:1888] T 8810b6f61db14fed8b12d36fed413dc8 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.477463 24180 ts_tablet_manager.cc:1888] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:16:45.487823 24179 log.cc:1196] T 8810b6f61db14fed8b12d36fed413dc8 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/8810b6f61db14fed8b12d36fed413dc8
I20240502 08:16:45.488082 24179 ts_tablet_manager.cc:1909] T 8810b6f61db14fed8b12d36fed413dc8 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:45.488283 24180 log.cc:1196] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/1ce5ccfd29724dc49afd0892b99f3169
I20240502 08:16:45.488870 24180 ts_tablet_manager.cc:1909] T 1ce5ccfd29724dc49afd0892b99f3169 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:45.489010 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 8810b6f61db14fed8b12d36fed413dc8 (table default.Table_ce1b3e3032d84f72b573bf1049c30237 [id=4c18f80dd28145b399b53670c182821c]) successfully deleted
I20240502 08:16:45.490353 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 612) took 1250 ms. Trace:
I20240502 08:16:45.490721 24715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:45.491067 24175 tablet_bootstrap.cc:492] T 01675f637a5a4645949abfd87b3d09a2 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:45.491238 24175 ts_tablet_manager.cc:1367] T 01675f637a5a4645949abfd87b3d09a2 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.147s user 0.010s sys 0.006s
I20240502 08:16:45.491994 24175 raft_consensus.cc:348] T 01675f637a5a4645949abfd87b3d09a2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:45.496315 24180 rpcz_store.cc:276] 0502 08:16:44.240238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.241443 (+ 1205us) service_pool.cc:225] Handling call
0502 08:16:45.387874 (+1146431us) ts_tablet_manager.cc:1120] Deleting tablet 1ce5ccfd29724dc49afd0892b99f3169
0502 08:16:45.387883 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.445183 (+ 57300us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:45.477381 (+ 32198us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:45.490335 (+ 12954us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1139684}
I20240502 08:16:45.497411 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 1ce5ccfd29724dc49afd0892b99f3169 (table default.Table_efda0c8c5d1d4e138081bfe146a06fa4 [id=3a9d7e8fe2c84cb4942fee9053a012b0]) successfully deleted
I20240502 08:16:45.498319 24174 tablet_bootstrap.cc:492] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:45.498512 24174 ts_tablet_manager.cc:1367] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.001s
I20240502 08:16:45.499241 24174 raft_consensus.cc:348] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.499478 24174 raft_consensus.cc:374] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.499568 24174 raft_consensus.cc:729] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.499847 24174 consensus_queue.cc:260] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.500365 24174 ts_tablet_manager.cc:1398] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:45.500662 24174 tablet_bootstrap.cc:492] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:45.500993 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 614) took 1248 ms. Trace:
I20240502 08:16:45.501103 24180 rpcz_store.cc:276] 0502 08:16:44.252833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.252885 (+ 52us) service_pool.cc:225] Handling call
0502 08:16:45.500849 (+1247964us) ts_tablet_manager.cc:1120] Deleting tablet cb7b37f776df47da9aa6e7fba82a8999
0502 08:16:45.500862 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.500978 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1244310}
W20240502 08:16:45.501417 23993 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 199974us
W20240502 08:16:45.501978 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet cb7b37f776df47da9aa6e7fba82a8999 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb7b37f776df47da9aa6e7fba82a8999 already in progress: creating tablet
I20240502 08:16:45.502315 24180 tablet_replica.cc:330] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:45.502557 24180 raft_consensus.cc:2229] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:45.502875 24180 pending_rounds.cc:62] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:16:45.502982 24180 pending_rounds.cc:68] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7023156450144067584 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "49b816ea1d474f2db94119441146ba78" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_20df1b504848439c869ae3739ac54185" dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" new_extra_config { } }
I20240502 08:16:45.505234 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 643ce929e74c499da835fbe9b63b8880 (table default.Table_a0ebd3d8bd13425ea80ffd0c428e20da [id=1c9043e3bf434bae83ea36aaef230b6f]) successfully deleted
I20240502 08:16:45.505589 24833 tablet.cc:1722] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:45.513351 24174 tablet_bootstrap.cc:654] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.512657 24181 ts_tablet_manager.cc:1888] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240502 08:16:45.503407 24180 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43544: Aborted: Op aborted
I20240502 08:16:45.516824 24180 raft_consensus.cc:2258] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:45.517901 24180 ts_tablet_manager.cc:1875] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.518349 24175 raft_consensus.cc:374] T 01675f637a5a4645949abfd87b3d09a2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.518607 24175 raft_consensus.cc:729] T 01675f637a5a4645949abfd87b3d09a2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.519475 24175 consensus_queue.cc:260] T 01675f637a5a4645949abfd87b3d09a2 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
W20240502 08:16:45.530236 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 571886fdbc2c43e19c790b2b7830ec27 has no consensus state
I20240502 08:16:45.533030 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5ad32437311f45d3a50a7fe4b134adb9" } modify_external_catalogs: true
I20240502 08:16:45.533237 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5ad32437311f45d3a50a7fe4b134adb9" } modify_external_catalogs: true
W20240502 08:16:45.533790 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ae2b20ae3a014c85af33d5ee51b704e1 consensus state has no leader
I20240502 08:16:45.534169 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ff8f3ce7d85c49fabe0e3ef763bd7517"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:45.534518 24174 tablet_bootstrap.cc:492] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:45.534641 24174 ts_tablet_manager.cc:1367] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.003s
W20240502 08:16:45.534705 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff8f3ce7d85c49fabe0e3ef763bd7517 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.535404 24174 raft_consensus.cc:348] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.535578 24174 raft_consensus.cc:374] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.535629 24174 raft_consensus.cc:729] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
W20240502 08:16:45.535948 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e36be367f9b4c568ce0f3278b6297d9 consensus state has no leader
I20240502 08:16:45.535817 24174 consensus_queue.cc:260] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.536315 24174 ts_tablet_manager.cc:1398] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:45.536288 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_f0a7174ad23742efad1bbd0a285e2731"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:45.536300 24175 ts_tablet_manager.cc:1398] T 01675f637a5a4645949abfd87b3d09a2 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.045s user 0.002s sys 0.000s
W20240502 08:16:45.536741 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0a7174ad23742efad1bbd0a285e2731 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.537101 24174 tablet_bootstrap.cc:492] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:45.538051 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8d3e812b72644192aa5a83f9754ff725"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.538434 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d3e812b72644192aa5a83f9754ff725 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:45.540266 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 793d364d9dec429484660a86036f5fb9 consensus state has no leader
I20240502 08:16:45.540616 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2cc3c92f41994b1ab2144237110b1d88" } modify_external_catalogs: true
I20240502 08:16:45.540786 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2cc3c92f41994b1ab2144237110b1d88" } modify_external_catalogs: true
I20240502 08:16:45.540827 24180 ts_tablet_manager.cc:1888] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:16:45.540915 24180 log.cc:1196] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/49b816ea1d474f2db94119441146ba78
I20240502 08:16:45.541285 24180 ts_tablet_manager.cc:1909] T 49b816ea1d474f2db94119441146ba78 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:45.542080 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 613) took 1294 ms. Trace:
I20240502 08:16:45.542601 24174 tablet_bootstrap.cc:654] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.542655 24180 rpcz_store.cc:276] 0502 08:16:44.252244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.252295 (+ 51us) service_pool.cc:225] Handling call
0502 08:16:45.502266 (+1249971us) ts_tablet_manager.cc:1120] Deleting tablet 49b816ea1d474f2db94119441146ba78
0502 08:16:45.502276 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.525726 (+ 23450us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:45.540721 (+ 14995us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:45.542061 (+ 1340us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1245625}
I20240502 08:16:45.544025 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 49b816ea1d474f2db94119441146ba78 (table default.Table_20df1b504848439c869ae3739ac54185 [id=a0606a4053d04dd3814f09419a4a70c2]) successfully deleted
I20240502 08:16:45.544479 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 625) took 1142 ms. Trace:
I20240502 08:16:45.544608 24180 rpcz_store.cc:276] 0502 08:16:44.402122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.426612 (+ 24490us) service_pool.cc:225] Handling call
0502 08:16:45.544358 (+1117746us) ts_tablet_manager.cc:1120] Deleting tablet 29144c1d42f0408f9b2987c9c66ed8dc
0502 08:16:45.544367 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.544461 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1094644}
W20240502 08:16:45.545157 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 29144c1d42f0408f9b2987c9c66ed8dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29144c1d42f0408f9b2987c9c66ed8dc already in progress: creating tablet
I20240502 08:16:45.545516 24180 tablet_replica.cc:330] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:45.545960 24180 raft_consensus.cc:2229] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:45.546206 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 01675f637a5a4645949abfd87b3d09a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:45.546434 24179 tablet_replica.cc:330] T 01675f637a5a4645949abfd87b3d09a2 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:45.546594 24179 raft_consensus.cc:2229] T 01675f637a5a4645949abfd87b3d09a2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:45.546684 24175 tablet_bootstrap.cc:492] T 5453cc0b93ac4cbb90a0588bb2100d60 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:45.546731 24179 raft_consensus.cc:2258] T 01675f637a5a4645949abfd87b3d09a2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:45.547343 24180 raft_consensus.cc:2258] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:45.547500 24179 ts_tablet_manager.cc:1875] T 01675f637a5a4645949abfd87b3d09a2 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.549595 23607 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet d1be2c29d3154d9189af3266191f0b3e (table default.Table_db86cd2cf86e48ffbc6a513b5c2bd7ab [id=6103f133bcdd4f768b92b37245bf5356]). Expected version 1 got 0
I20240502 08:16:45.550922 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 5b627347a1684a83bc53af603c88e908 (DEFAULT_TABLE table=default.Table_4179387181314b89ba7db097b4ee2318 [id=20819a7fcd7f4d0ca0146009ca7b72f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.551250 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b627347a1684a83bc53af603c88e908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.554049 24179 ts_tablet_manager.cc:1888] T 01675f637a5a4645949abfd87b3d09a2 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.554140 24179 log.cc:1196] T 01675f637a5a4645949abfd87b3d09a2 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/01675f637a5a4645949abfd87b3d09a2
I20240502 08:16:45.559283 24179 ts_tablet_manager.cc:1909] T 01675f637a5a4645949abfd87b3d09a2 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:45.554296 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 5b627347a1684a83bc53af603c88e908 (DEFAULT_TABLE table=default.Table_4179387181314b89ba7db097b4ee2318 [id=20819a7fcd7f4d0ca0146009ca7b72f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.560429 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b627347a1684a83bc53af603c88e908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.560832 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 01675f637a5a4645949abfd87b3d09a2 (table default.Table_c5d06caccb424872b965a751c171a0a6 [id=4411ea4d183f4694b16b6255567ab62b]) successfully deleted
I20240502 08:16:45.554484 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 49b816ea1d474f2db94119441146ba78 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.558780 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dff0d73a933427fb1f2686bac33eba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.559319 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 171406810543435e832a47ad5b3c80a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e5de793e74247dd9e73b7b1d58027d9) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.564646 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 7dadd4f432554022bc2fdc3c775ac9d1
I20240502 08:16:45.564764 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 058e13ba3a474af3aabebc93c5a88124
I20240502 08:16:45.565131 23608 catalog_manager.cc:6554] ReplaceTablet: tablet 7dadd4f432554022bc2fdc3c775ac9d1 deleted and replaced by tablet cb978e43ec794352bc0ed7f7aad4391b
I20240502 08:16:45.565783 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 3abfc0c885d64c39808a6e7c815afd7e
I20240502 08:16:45.566164 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 058e13ba3a474af3aabebc93c5a88124 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.566341 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dadd4f432554022bc2fdc3c775ac9d1 with delete_type TABLET_DATA_DELETED (replaced by tablet cb978e43ec794352bc0ed7f7aad4391b) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.517233 24843 tablet.cc:1717] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5: Skipping requested alter to schema version 4, tablet already version 4
I20240502 08:16:45.566846 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 3abfc0c885d64c39808a6e7c815afd7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.567260 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 058e13ba3a474af3aabebc93c5a88124 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:45.567461 23886 tablet_service.cc:1514] Processing DeleteTablet for tablet 3abfc0c885d64c39808a6e7c815afd7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:45.567116 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dadd4f432554022bc2fdc3c775ac9d1 with delete_type TABLET_DATA_DELETED (replaced by tablet cb978e43ec794352bc0ed7f7aad4391b) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:45.568086 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 058e13ba3a474af3aabebc93c5a88124 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:45.568126 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 058e13ba3a474af3aabebc93c5a88124 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 058e13ba3a474af3aabebc93c5a88124 already in progress: creating tablet
I20240502 08:16:45.568349 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 3abfc0c885d64c39808a6e7c815afd7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:45.568712 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 3abfc0c885d64c39808a6e7c815afd7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3abfc0c885d64c39808a6e7c815afd7e already in progress: creating tablet
W20240502 08:16:45.569283 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 7dadd4f432554022bc2fdc3c775ac9d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7dadd4f432554022bc2fdc3c775ac9d1 already in progress: creating tablet
I20240502 08:16:45.569465 24843 tablet.cc:1717] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5: Skipping requested alter to schema version 4, tablet already version 4
I20240502 08:16:45.570593 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc85cce715742cf999ad41ebc1deebd (DEFAULT_TABLE table=default.Table_e61ff7f9f4074074885941b862adf04d [id=4e64d5519d1e44a69480416eaf1081ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.570684 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc85cce715742cf999ad41ebc1deebd (DEFAULT_TABLE table=default.Table_e61ff7f9f4074074885941b862adf04d [id=4e64d5519d1e44a69480416eaf1081ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.570948 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc85cce715742cf999ad41ebc1deebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.571043 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc85cce715742cf999ad41ebc1deebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.571805 24843 tablet.cc:1717] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5: Skipping requested alter to schema version 4, tablet already version 4
I20240502 08:16:45.562150 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 5b627347a1684a83bc53af603c88e908 (DEFAULT_TABLE table=default.Table_4179387181314b89ba7db097b4ee2318 [id=20819a7fcd7f4d0ca0146009ca7b72f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.573087 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet d4b03d759a354a39aea44fd5d4eeb089
I20240502 08:16:45.573241 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b627347a1684a83bc53af603c88e908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.573827 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet d7e2da9cff594d90819f6b216dbed552
I20240502 08:16:45.574548 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b03d759a354a39aea44fd5d4eeb089 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.574803 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b03d759a354a39aea44fd5d4eeb089 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:45.574894 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 22bc17792cf040748ab672cfc6451e3c (DEFAULT_TABLE table=default.Table_5964ac143b9f461287f7b52b806195d0 [id=92a39fb7d6c34df188da4862f7175d82]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.575196 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22bc17792cf040748ab672cfc6451e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.575130 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 22bc17792cf040748ab672cfc6451e3c (DEFAULT_TABLE table=default.Table_5964ac143b9f461287f7b52b806195d0 [id=92a39fb7d6c34df188da4862f7175d82]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.575445 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22bc17792cf040748ab672cfc6451e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.576342 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b03d759a354a39aea44fd5d4eeb089 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:45.576683 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 22bc17792cf040748ab672cfc6451e3c (DEFAULT_TABLE table=default.Table_5964ac143b9f461287f7b52b806195d0 [id=92a39fb7d6c34df188da4862f7175d82]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.576969 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22bc17792cf040748ab672cfc6451e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.565524 24173 consensus_queue.cc:260] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.577813 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 0082d71f92424d4d8e55f8dfd739889e (DEFAULT_TABLE table=default.Table_32c806f7aa664335b36da222d5fd5618 [id=7a435da11737408ca3a0df4e9c5d5db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.578091 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0082d71f92424d4d8e55f8dfd739889e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.579627 24181 log.cc:1196] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/a3043c5ed98c401b82c6e967ca3f3613
I20240502 08:16:45.579933 24181 ts_tablet_manager.cc:1909] T a3043c5ed98c401b82c6e967ca3f3613 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:45.581005 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 31775513f3a446b08c0aac25ae0cab20 (DEFAULT_TABLE table=default.Table_8c1e23fe00844168a9f1755bedd3625a [id=5add458d17a549c59d462ee206c4b072]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.581297 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31775513f3a446b08c0aac25ae0cab20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.581851 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 31775513f3a446b08c0aac25ae0cab20 (DEFAULT_TABLE table=default.Table_8c1e23fe00844168a9f1755bedd3625a [id=5add458d17a549c59d462ee206c4b072]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.582150 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31775513f3a446b08c0aac25ae0cab20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.585641 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet d7e2da9cff594d90819f6b216dbed552 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:45.585808 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc85cce715742cf999ad41ebc1deebd (DEFAULT_TABLE table=default.Table_e61ff7f9f4074074885941b862adf04d [id=4e64d5519d1e44a69480416eaf1081ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.586103 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc85cce715742cf999ad41ebc1deebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.586586 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 642e26c0d0f445adadb65854d0513f77 (DEFAULT_TABLE table=default.Table_1991a52161ad40d0bbccd98a692364a7 [id=7cf5e1868a7c4f229d9e477c763dea0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.586874 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642e26c0d0f445adadb65854d0513f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.587024 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 0082d71f92424d4d8e55f8dfd739889e (DEFAULT_TABLE table=default.Table_32c806f7aa664335b36da222d5fd5618 [id=7a435da11737408ca3a0df4e9c5d5db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.587335 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0082d71f92424d4d8e55f8dfd739889e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.570036 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 642e26c0d0f445adadb65854d0513f77 (DEFAULT_TABLE table=default.Table_1991a52161ad40d0bbccd98a692364a7 [id=7cf5e1868a7c4f229d9e477c763dea0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.589321 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642e26c0d0f445adadb65854d0513f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.590550 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 9850e84117044d2f9dc82b6314dac2cc (DEFAULT_TABLE table=default.Table_8f820382d8484e16834f76af57d0ab45 [id=30a680241cb24e15ab8bad3b6d6b93ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.590715 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 0082d71f92424d4d8e55f8dfd739889e (DEFAULT_TABLE table=default.Table_32c806f7aa664335b36da222d5fd5618 [id=7a435da11737408ca3a0df4e9c5d5db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.590994 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9850e84117044d2f9dc82b6314dac2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.591032 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0082d71f92424d4d8e55f8dfd739889e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.593807 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 31775513f3a446b08c0aac25ae0cab20 (DEFAULT_TABLE table=default.Table_8c1e23fe00844168a9f1755bedd3625a [id=5add458d17a549c59d462ee206c4b072]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.594149 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31775513f3a446b08c0aac25ae0cab20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.596660 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 642e26c0d0f445adadb65854d0513f77 (DEFAULT_TABLE table=default.Table_1991a52161ad40d0bbccd98a692364a7 [id=7cf5e1868a7c4f229d9e477c763dea0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.596952 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642e26c0d0f445adadb65854d0513f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.597683 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet d7e2da9cff594d90819f6b216dbed552 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.599874 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2439a8f2e30f4bdf8b18d7ac811ab586"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.600440 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2439a8f2e30f4bdf8b18d7ac811ab586 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.600993 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b4cedcada1f94d4296418229ee7512a0" } new_table_name: "default.Table_43ea406acdfc40b7b657b4f6e1c6db50" modify_external_catalogs: true
I20240502 08:16:45.601819 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0751e871f6e14c0697044070629e6d26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.602280 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0751e871f6e14c0697044070629e6d26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.603086 24778 raft_consensus.cc:482] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:45.603202 24778 raft_consensus.cc:504] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.603745 24173 ts_tablet_manager.cc:1398] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.585s user 0.018s sys 0.005s
I20240502 08:16:45.603837 24778 leader_election.cc:288] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:45.603838 24173 ts_tablet_manager.cc:1430] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5: Tablet startup took 584ms
I20240502 08:16:45.605078 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_212251b2938346e19add6081bc09d254"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:45.605118 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_213f4bea9fad44c4a1aa726aed361575"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.605525 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_212251b2938346e19add6081bc09d254 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:45.605667 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_213f4bea9fad44c4a1aa726aed361575 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:45.603906 24173 ts_tablet_manager.cc:1432] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5: Trace:
0502 08:16:36.908441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:36.942167 (+ 33726us) service_pool.cc:225] Handling call
0502 08:16:36.944125 (+ 1958us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:36.944154 (+ 29us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:36.958885 (+ 14731us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:36.968697 (+ 9812us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:36.968832 (+ 135us) inbound_call.cc:173] Queueing success response
0502 08:16:44.901851 (+7933019us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0502 08:16:45.019222 (+117371us) ts_tablet_manager.cc:1399] Starting tablet replica
0502 08:16:45.019444 (+ 222us) tablet_replica.cc:259] Starting instrumentation
0502 08:16:45.019649 (+ 205us) tablet_replica.cc:278] Starting consensus
Metrics: {"raft.queue_time_us":24377,"raft.run_cpu_time_us":60,"raft.run_wall_time_us":61,"tablet-open.queue_time_us":7933046}
I20240502 08:16:45.607168 24173 tablet_bootstrap.cc:492] T 2bfb532c9d4c41afb7acaa4831841306 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:45.608290 24181 tablet_replica.cc:330] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:45.609521 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d28098dcbc4761a34ee65769d6fd7c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:45.610518 23723 leader_election.cc:341] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:45.610497 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_469c024c43d64e2fb98eff5ff16e988b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:45.610369 24174 tablet_bootstrap.cc:492] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:45.611019 24174 ts_tablet_manager.cc:1367] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.000s
W20240502 08:16:45.611055 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_469c024c43d64e2fb98eff5ff16e988b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.612008 24174 raft_consensus.cc:348] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.612262 24174 raft_consensus.cc:374] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.612370 24174 raft_consensus.cc:729] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.612262 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_309cba3acefd434b84e5dabeaeca55e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:45.612706 24778 raft_consensus.cc:482] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:16:45.612831 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_309cba3acefd434b84e5dabeaeca55e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.612820 24778 raft_consensus.cc:504] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.613435 24778 leader_election.cc:288] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:45.614043 24174 consensus_queue.cc:260] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.615983 24174 ts_tablet_manager.cc:1398] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:16:45.616552 24180 ts_tablet_manager.cc:1875] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.617020 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c08228fcd1304528b9446003851f6c70" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:45.617509 23723 leader_election.cc:341] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:45.618146 24181 raft_consensus.cc:2229] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:45.618407 24181 raft_consensus.cc:2258] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:45.621417 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet a3043c5ed98c401b82c6e967ca3f3613 (table default.Table_3e0123590a92473dabaacc9c158859e7 [id=d1a317371a5b43f185eb6453861e112a]) successfully deleted
W20240502 08:16:45.621995 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet d4b03d759a354a39aea44fd5d4eeb089 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4b03d759a354a39aea44fd5d4eeb089 already in progress: creating tablet
W20240502 08:16:45.622287 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet d7e2da9cff594d90819f6b216dbed552 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7e2da9cff594d90819f6b216dbed552 already in progress: creating tablet
I20240502 08:16:45.622812 24181 ts_tablet_manager.cc:1875] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.618212 24021 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dadd4f432554022bc2fdc3c775ac9d1 with delete_type TABLET_DATA_DELETED (replaced by tablet cb978e43ec794352bc0ed7f7aad4391b) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:45.621600 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 9850e84117044d2f9dc82b6314dac2cc (DEFAULT_TABLE table=default.Table_8f820382d8484e16834f76af57d0ab45 [id=30a680241cb24e15ab8bad3b6d6b93ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.623229 24039 tablet_service.cc:1467] Processing CreateTablet for tablet d36e7cd33f9a4edabe690e75602590b9 (DEFAULT_TABLE table=default.Table_ee8e3c4f16e74576b385099a280d5312 [id=19bbdbb47e2e49b286e1cd28a67b46af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.623502 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9850e84117044d2f9dc82b6314dac2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.623692 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d36e7cd33f9a4edabe690e75602590b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.624298 24174 tablet_bootstrap.cc:492] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:45.622315 24173 tablet_bootstrap.cc:654] T 2bfb532c9d4c41afb7acaa4831841306 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.625746 24174 tablet_bootstrap.cc:654] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:45.628664 23723 leader_election.cc:341] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: c08228fcd1304528b9446003851f6c70
I20240502 08:16:45.628762 23723 leader_election.cc:302] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
W20240502 08:16:45.629307 23723 leader_election.cc:341] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: c5d28098dcbc4761a34ee65769d6fd7c
I20240502 08:16:45.629410 23723 leader_election.cc:302] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:45.629643 24774 raft_consensus.cc:2726] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:45.629792 24774 raft_consensus.cc:2726] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:45.631318 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 9850e84117044d2f9dc82b6314dac2cc (DEFAULT_TABLE table=default.Table_8f820382d8484e16834f76af57d0ab45 [id=30a680241cb24e15ab8bad3b6d6b93ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.631673 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9850e84117044d2f9dc82b6314dac2cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.632236 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.633818 23887 tablet_service.cc:1467] Processing CreateTablet for tablet d36e7cd33f9a4edabe690e75602590b9 (DEFAULT_TABLE table=default.Table_ee8e3c4f16e74576b385099a280d5312 [id=19bbdbb47e2e49b286e1cd28a67b46af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.634135 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d36e7cd33f9a4edabe690e75602590b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.639019 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet d7e2da9cff594d90819f6b216dbed552 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:45.644078 24174 tablet_bootstrap.cc:492] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:45.644218 24174 ts_tablet_manager.cc:1367] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.002s
I20240502 08:16:45.644919 24174 raft_consensus.cc:348] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.645126 24174 raft_consensus.cc:374] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.645197 24174 raft_consensus.cc:729] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.645447 24174 consensus_queue.cc:260] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.651494 24175 tablet_bootstrap.cc:654] T 5453cc0b93ac4cbb90a0588bb2100d60 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.639165 24181 ts_tablet_manager.cc:1888] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.653976 24181 log.cc:1196] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/bcfacc302ff04178bfffc48111cd7a3f
I20240502 08:16:45.654320 24181 ts_tablet_manager.cc:1909] T bcfacc302ff04178bfffc48111cd7a3f P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
W20240502 08:16:45.657011 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7cbf1c0a37454beb966aa61b4f8caf72 consensus state has no leader
I20240502 08:16:45.657142 24174 ts_tablet_manager.cc:1398] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 08:16:45.661113 24174 tablet_bootstrap.cc:492] T bcfacc302ff04178bfffc48111cd7a3f P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:45.663002 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c374f42f353497fa1d745164e7fbf61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.663216 24174 tablet_bootstrap.cc:654] T bcfacc302ff04178bfffc48111cd7a3f P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.667825 23756 tablet_service.cc:1467] Processing CreateTablet for tablet d36e7cd33f9a4edabe690e75602590b9 (DEFAULT_TABLE table=default.Table_ee8e3c4f16e74576b385099a280d5312 [id=19bbdbb47e2e49b286e1cd28a67b46af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:45.668249 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d36e7cd33f9a4edabe690e75602590b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:45.673434 24174 tablet_bootstrap.cc:492] T bcfacc302ff04178bfffc48111cd7a3f P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:45.673602 24174 ts_tablet_manager.cc:1367] T bcfacc302ff04178bfffc48111cd7a3f P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.001s
I20240502 08:16:45.674376 24174 raft_consensus.cc:348] T bcfacc302ff04178bfffc48111cd7a3f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.674553 24174 raft_consensus.cc:374] T bcfacc302ff04178bfffc48111cd7a3f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.674603 24174 raft_consensus.cc:729] T bcfacc302ff04178bfffc48111cd7a3f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.675082 24774 raft_consensus.cc:482] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:45.675184 24774 raft_consensus.cc:504] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.675901 24774 leader_election.cc:288] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:45.676584 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fac7df16fc41d7a3297055a7258270" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:45.674847 24174 consensus_queue.cc:260] T bcfacc302ff04178bfffc48111cd7a3f P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.677451 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet 57df4bde56a44809ad43d563baeefecb from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:45.681629 24175 tablet_bootstrap.cc:492] T 5453cc0b93ac4cbb90a0588bb2100d60 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:45.681764 24175 ts_tablet_manager.cc:1367] T 5453cc0b93ac4cbb90a0588bb2100d60 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.136s user 0.003s sys 0.001s
I20240502 08:16:45.681919 24181 tablet_replica.cc:330] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:45.682139 24181 raft_consensus.cc:2229] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:45.682278 24181 raft_consensus.cc:2258] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:45.682276 23723 leader_election.cc:341] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:45.682490 24175 raft_consensus.cc:348] T 5453cc0b93ac4cbb90a0588bb2100d60 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.682673 24175 raft_consensus.cc:374] T 5453cc0b93ac4cbb90a0588bb2100d60 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.682723 24175 raft_consensus.cc:729] T 5453cc0b93ac4cbb90a0588bb2100d60 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.682804 24181 ts_tablet_manager.cc:1875] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.682946 24175 consensus_queue.cc:260] T 5453cc0b93ac4cbb90a0588bb2100d60 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:45.683079 23723 leader_election.cc:341] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: b1fac7df16fc41d7a3297055a7258270
I20240502 08:16:45.683180 23723 leader_election.cc:302] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:45.684114 24774 raft_consensus.cc:2726] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:45.663110 24180 ts_tablet_manager.cc:1888] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.684645 24180 log.cc:1196] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/e0af5f4daf3149738647a08219efb245
I20240502 08:16:45.684996 24175 ts_tablet_manager.cc:1398] T 5453cc0b93ac4cbb90a0588bb2100d60 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:45.685009 24180 ts_tablet_manager.cc:1909] T e0af5f4daf3149738647a08219efb245 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:45.685762 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 626) took 1203 ms. Trace:
I20240502 08:16:45.685851 24180 rpcz_store.cc:276] 0502 08:16:44.482643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.482809 (+ 166us) service_pool.cc:225] Handling call
0502 08:16:45.545468 (+1062659us) ts_tablet_manager.cc:1120] Deleting tablet e0af5f4daf3149738647a08219efb245
0502 08:16:45.545477 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.629014 (+ 83537us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:45.662990 (+ 33976us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:45.685743 (+ 22753us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1062411}
W20240502 08:16:45.686226 23595 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet e54b09eac68f423086620617dd4a6ffc (table default.Table_16cc91d0fe274b37a9114f57e8d1cc8c [id=b13a6a084a8d45728e3932be14219d5a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e54b09eac68f423086620617dd4a6ffc consensus state has no leader
I20240502 08:16:45.686321 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet e0af5f4daf3149738647a08219efb245 (table default.Table_d4bd37f8c6e54ecf82cd247846aa06d0 [id=57722a7341c541b19efa213bcf48eb53]) successfully deleted
I20240502 08:16:45.688328 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 627) took 1199 ms. Trace:
I20240502 08:16:45.688423 24180 rpcz_store.cc:276] 0502 08:16:44.488710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.488770 (+ 60us) service_pool.cc:225] Handling call
0502 08:16:45.687067 (+1198297us) ts_tablet_manager.cc:1120] Deleting tablet 843edd7092464981adad317b48871645
0502 08:16:45.687075 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.688309 (+ 1234us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1191818}
I20240502 08:16:45.689088 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 630) took 1014 ms. Trace:
I20240502 08:16:45.687767 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet bcfacc302ff04178bfffc48111cd7a3f (table default.Table_d416b40fe3d24dafa8db5ee61d13acc0 [id=fa5100e581a7430ab8f0c19b2daae994]) successfully deleted
W20240502 08:16:45.688874 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 843edd7092464981adad317b48871645 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 843edd7092464981adad317b48871645 already in progress: creating tablet
I20240502 08:16:45.689186 24180 rpcz_store.cc:276] 0502 08:16:44.675143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.675201 (+ 58us) service_pool.cc:225] Handling call
0502 08:16:45.688983 (+1013782us) ts_tablet_manager.cc:1120] Deleting tablet 890fd44075ad4b5699f2fcf0a1ab7804
0502 08:16:45.688993 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.689071 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":999575}
I20240502 08:16:45.689972 24175 tablet_bootstrap.cc:492] T c08228fcd1304528b9446003851f6c70 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:45.690290 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 890fd44075ad4b5699f2fcf0a1ab7804 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 890fd44075ad4b5699f2fcf0a1ab7804 already in progress: creating tablet
W20240502 08:16:45.690954 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 324716a1b3c44be2b415695f821de8fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 324716a1b3c44be2b415695f821de8fc already in progress: creating tablet
I20240502 08:16:45.691490 24175 tablet_bootstrap.cc:654] T c08228fcd1304528b9446003851f6c70 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.692303 24174 ts_tablet_manager.cc:1398] T bcfacc302ff04178bfffc48111cd7a3f P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.019s user 0.001s sys 0.000s
I20240502 08:16:45.698143 24174 tablet_bootstrap.cc:492] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
W20240502 08:16:45.698833 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 23ba091d8efb44858bf90a104c0f442f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23ba091d8efb44858bf90a104c0f442f already in progress: creating tablet
I20240502 08:16:45.699343 24180 tablet_replica.cc:330] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:45.699592 24180 raft_consensus.cc:2229] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:45.700021 24175 tablet_bootstrap.cc:492] T c08228fcd1304528b9446003851f6c70 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:45.700139 24175 ts_tablet_manager.cc:1367] T c08228fcd1304528b9446003851f6c70 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20240502 08:16:45.700886 24175 raft_consensus.cc:348] T c08228fcd1304528b9446003851f6c70 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.701077 24175 raft_consensus.cc:374] T c08228fcd1304528b9446003851f6c70 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.701126 24175 raft_consensus.cc:729] T c08228fcd1304528b9446003851f6c70 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.701320 24175 consensus_queue.cc:260] T c08228fcd1304528b9446003851f6c70 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.703332 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet a91e2b50321e4241847d97a558b419f2
I20240502 08:16:45.704999 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 57df4bde56a44809ad43d563baeefecb
I20240502 08:16:45.705283 24175 ts_tablet_manager.cc:1398] T c08228fcd1304528b9446003851f6c70 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:16:45.706669 23602 catalog_manager.cc:6554] ReplaceTablet: tablet 57df4bde56a44809ad43d563baeefecb deleted and replaced by tablet 56d6373938824840934211f259e14d4f
I20240502 08:16:45.707242 24180 raft_consensus.cc:2258] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:45.701467 24174 tablet_bootstrap.cc:654] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.704411 24774 raft_consensus.cc:482] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:45.708237 24774 raft_consensus.cc:504] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.708725 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 5453cc0b93ac4cbb90a0588bb2100d60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:45.708873 24774 leader_election.cc:288] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:45.704454 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet a91e2b50321e4241847d97a558b419f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.708931 24179 tablet_replica.cc:330] T 5453cc0b93ac4cbb90a0588bb2100d60 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:45.705751 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet bcfacc302ff04178bfffc48111cd7a3f with delete_type TABLET_DATA_DELETED (replaced by tablet b0cf1a6de3d649b984c50f45393ec0f1) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.709147 24179 raft_consensus.cc:2229] T 5453cc0b93ac4cbb90a0588bb2100d60 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:45.709304 24179 raft_consensus.cc:2258] T 5453cc0b93ac4cbb90a0588bb2100d60 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:45.706264 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 57df4bde56a44809ad43d563baeefecb with delete_type TABLET_DATA_DELETED (replaced by tablet 56d6373938824840934211f259e14d4f) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.709625 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1be2c29d3154d9189af3266191f0b3e" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:45.709993 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet a91e2b50321e4241847d97a558b419f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:45.710004 24026 tablet_service.cc:1514] Processing DeleteTablet for tablet a91e2b50321e4241847d97a558b419f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:45.710063 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 57df4bde56a44809ad43d563baeefecb with delete_type TABLET_DATA_DELETED (replaced by tablet 56d6373938824840934211f259e14d4f) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:45.710350 23723 leader_election.cc:341] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:45.709754 24056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1be2c29d3154d9189af3266191f0b3e" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:45.710511 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet c08228fcd1304528b9446003851f6c70 with delete_type TABLET_DATA_DELETED (replaced by tablet b59c2f93d6514c9cbf358ab96f1a06b5) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:45.710670 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 57df4bde56a44809ad43d563baeefecb with delete_type TABLET_DATA_DELETED (replaced by tablet 56d6373938824840934211f259e14d4f) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:45.710680 24056 raft_consensus.cc:2454] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:45.710896 24175 tablet_bootstrap.cc:492] T d1be2c29d3154d9189af3266191f0b3e P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:45.711058 23723 leader_election.cc:302] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:45.711489 24774 raft_consensus.cc:2781] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:45.711596 24774 raft_consensus.cc:482] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:45.711665 24774 raft_consensus.cc:3037] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:45.712329 24175 tablet_bootstrap.cc:654] T d1be2c29d3154d9189af3266191f0b3e P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.712762 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet d7e2da9cff594d90819f6b216dbed552 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:45.713519 24179 ts_tablet_manager.cc:1875] T 5453cc0b93ac4cbb90a0588bb2100d60 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.714488 24774 raft_consensus.cc:504] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.716317 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1be2c29d3154d9189af3266191f0b3e" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:45.716534 24774 leader_election.cc:288] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:45.716734 23723 leader_election.cc:341] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:45.717073 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1be2c29d3154d9189af3266191f0b3e" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:45.717228 24057 raft_consensus.cc:3037] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:45.719626 24057 raft_consensus.cc:2454] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:45.719676 24179 ts_tablet_manager.cc:1888] T 5453cc0b93ac4cbb90a0588bb2100d60 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.719857 24179 log.cc:1196] T 5453cc0b93ac4cbb90a0588bb2100d60 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/5453cc0b93ac4cbb90a0588bb2100d60
I20240502 08:16:45.720000 23723 leader_election.cc:302] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:45.720250 24179 ts_tablet_manager.cc:1909] T 5453cc0b93ac4cbb90a0588bb2100d60 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:45.720544 24774 raft_consensus.cc:2781] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:45.721341 24179 tablet_replica.cc:330] T c08228fcd1304528b9446003851f6c70 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:45.721331 24774 raft_consensus.cc:686] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:45.721526 24179 raft_consensus.cc:2229] T c08228fcd1304528b9446003851f6c70 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:45.721705 24179 raft_consensus.cc:2258] T c08228fcd1304528b9446003851f6c70 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:45.721669 24774 consensus_queue.cc:237] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.722127 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 5453cc0b93ac4cbb90a0588bb2100d60 (table default.Table_a92f0fc8bc664448a6acb94dec9daa14 [id=8c859fafcd4b49f0b4e9fc9d8961c9ad]) successfully deleted
W20240502 08:16:45.722486 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet a91e2b50321e4241847d97a558b419f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a91e2b50321e4241847d97a558b419f2 already in progress: creating tablet
I20240502 08:16:45.723410 24179 ts_tablet_manager.cc:1875] T c08228fcd1304528b9446003851f6c70 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.731704 24179 ts_tablet_manager.cc:1888] T c08228fcd1304528b9446003851f6c70 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.732306 24173 tablet_bootstrap.cc:492] T 2bfb532c9d4c41afb7acaa4831841306 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:45.732455 24173 ts_tablet_manager.cc:1367] T 2bfb532c9d4c41afb7acaa4831841306 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.125s user 0.005s sys 0.000s
I20240502 08:16:45.733150 24173 raft_consensus.cc:348] T 2bfb532c9d4c41afb7acaa4831841306 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.733383 24173 raft_consensus.cc:374] T 2bfb532c9d4c41afb7acaa4831841306 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.733451 24173 raft_consensus.cc:729] T 2bfb532c9d4c41afb7acaa4831841306 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.733670 24173 consensus_queue.cc:260] T 2bfb532c9d4c41afb7acaa4831841306 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.736831 24173 ts_tablet_manager.cc:1398] T 2bfb532c9d4c41afb7acaa4831841306 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:16:45.738447 23605 catalog_manager.cc:5484] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:45.739085 23605 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet d1be2c29d3154d9189af3266191f0b3e (table default.Table_db86cd2cf86e48ffbc6a513b5c2bd7ab [id=6103f133bcdd4f768b92b37245bf5356]). Expected version 1 got 0
I20240502 08:16:45.739704 24173 tablet_bootstrap.cc:492] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:45.741672 24173 tablet_bootstrap.cc:654] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.742683 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bfb532c9d4c41afb7acaa4831841306 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:45.744030 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_31d76df4cb2f4950970e912e0a8797b9" } new_table_name: "default.Table_57eccbf54577471a93890cd6fc133102" modify_external_catalogs: true
I20240502 08:16:45.744285 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8cb458b151584d4a8916e34bf519d16b" } new_table_name: "default.Table_fd4958dab0ee4fb09aa67dc98d2e8317" modify_external_catalogs: true
I20240502 08:16:45.745220 24180 ts_tablet_manager.cc:1875] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.749111 24174 tablet_bootstrap.cc:492] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:45.749253 24174 ts_tablet_manager.cc:1367] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.051s user 0.013s sys 0.003s
I20240502 08:16:45.750054 24174 raft_consensus.cc:348] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.750241 24174 raft_consensus.cc:374] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.750311 24174 raft_consensus.cc:729] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
W20240502 08:16:45.750236 23434 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 59907.9us
I20240502 08:16:45.750501 24174 consensus_queue.cc:260] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:45.752750 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5704dff6114142d3b27c365ab99e2277 consensus state has no leader
I20240502 08:16:45.754887 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cf112903d1344a4484d86fb9785c0f17" } new_table_name: "default.Table_79bb10e705c047138e1c4363673a65e6" modify_external_catalogs: true
I20240502 08:16:45.755182 24174 ts_tablet_manager.cc:1398] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:45.755478 24174 tablet_bootstrap.cc:492] T 0882b7509a74475780eb700818aab398 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:45.756850 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d8a5bcad2db34ac7bc3a55ce30fee5da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:45.757297 24174 tablet_bootstrap.cc:654] T 0882b7509a74475780eb700818aab398 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:45.757404 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8a5bcad2db34ac7bc3a55ce30fee5da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.759075 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a52d298f258f4463a634e594b1254755" } new_table_name: "default.Table_bde9aea224d14088aa02e33c198dc63e" modify_external_catalogs: true
I20240502 08:16:45.761940 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet a3043c5ed98c401b82c6e967ca3f3613 with delete_type TABLET_DATA_DELETED (replaced by tablet 0281c8123eab4c48bddc377f5695c6e0) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.767341 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a1f81503886a490bad183964519aee17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.767918 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1f81503886a490bad183964519aee17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.770287 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c0fffcc987024f0b876f772a57cb68b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:45.770900 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2f5e51d6ee144f3cbda8aabaee177251" } modify_external_catalogs: true
I20240502 08:16:45.771129 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2f5e51d6ee144f3cbda8aabaee177251" } modify_external_catalogs: true
W20240502 08:16:45.771348 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f8f43935536048a3a4a4da36d0d86ba5 consensus state has no leader
I20240502 08:16:45.772557 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d790560c82534057ab7061352d8c71a2" } modify_external_catalogs: true
I20240502 08:16:45.772698 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d790560c82534057ab7061352d8c71a2" } modify_external_catalogs: true
I20240502 08:16:45.773212 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_bbd16cb34af94cf4a869e14b72fc9f81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.773679 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbd16cb34af94cf4a869e14b72fc9f81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.773717 24179 log.cc:1196] T c08228fcd1304528b9446003851f6c70 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/c08228fcd1304528b9446003851f6c70
I20240502 08:16:45.774080 24179 ts_tablet_manager.cc:1909] T c08228fcd1304528b9446003851f6c70 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:45.760583 24175 tablet_bootstrap.cc:492] T d1be2c29d3154d9189af3266191f0b3e P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:45.774405 24175 ts_tablet_manager.cc:1367] T d1be2c29d3154d9189af3266191f0b3e P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.064s user 0.008s sys 0.014s
I20240502 08:16:45.774438 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_94efd28ada5b4024ab65e4d5feda0b38" } modify_external_catalogs: true
I20240502 08:16:45.774603 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_94efd28ada5b4024ab65e4d5feda0b38" } modify_external_catalogs: true
W20240502 08:16:45.774976 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0fffcc987024f0b876f772a57cb68b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.775319 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_50a6c708582c422c89aef5f2ca35a88a" } modify_external_catalogs: true
I20240502 08:16:45.775506 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_50a6c708582c422c89aef5f2ca35a88a" } modify_external_catalogs: true
I20240502 08:16:45.775741 24175 raft_consensus.cc:348] T d1be2c29d3154d9189af3266191f0b3e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.775931 24175 raft_consensus.cc:374] T d1be2c29d3154d9189af3266191f0b3e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.775983 24175 raft_consensus.cc:729] T d1be2c29d3154d9189af3266191f0b3e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
W20240502 08:16:45.776265 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92b830e128ac439eba286f29dbb20ee4 consensus state has no leader
I20240502 08:16:45.776167 24175 consensus_queue.cc:260] T d1be2c29d3154d9189af3266191f0b3e P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.779233 24174 tablet_bootstrap.cc:492] T 0882b7509a74475780eb700818aab398 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:45.779418 24174 ts_tablet_manager.cc:1367] T 0882b7509a74475780eb700818aab398 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.006s
I20240502 08:16:45.779448 24181 ts_tablet_manager.cc:1888] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.779567 24181 log.cc:1196] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/0882b7509a74475780eb700818aab398
I20240502 08:16:45.779870 24181 ts_tablet_manager.cc:1909] T 0882b7509a74475780eb700818aab398 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:45.780416 24174 raft_consensus.cc:348] T 0882b7509a74475780eb700818aab398 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.780660 24174 raft_consensus.cc:374] T 0882b7509a74475780eb700818aab398 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.780764 24174 raft_consensus.cc:729] T 0882b7509a74475780eb700818aab398 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.781045 24174 consensus_queue.cc:260] T 0882b7509a74475780eb700818aab398 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.781553 24174 ts_tablet_manager.cc:1398] T 0882b7509a74475780eb700818aab398 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:16:45.781689 24175 ts_tablet_manager.cc:1398] T d1be2c29d3154d9189af3266191f0b3e P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 08:16:45.781862 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 0882b7509a74475780eb700818aab398 (table default.Table_bc1e9d346aa846f8abca7504185f7cdb [id=8d27878c6c8f4234908a2c78cb8bbd29]) successfully deleted
I20240502 08:16:45.782132 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet c08228fcd1304528b9446003851f6c70 (table default.Table_bbf9d7694fe942de8aa8bd4ed61336a5 [id=f1d6eb661b0143ee82e797cde61391c0]) successfully deleted
W20240502 08:16:45.783464 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 57df4bde56a44809ad43d563baeefecb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57df4bde56a44809ad43d563baeefecb already in progress: creating tablet
I20240502 08:16:45.785212 24175 tablet_bootstrap.cc:492] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:45.786731 24175 tablet_bootstrap.cc:654] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.786748 24715 fs_manager.cc:709] Time spent opening directory manager: real 0.293s user 0.007s sys 0.000s
I20240502 08:16:45.788488 24174 tablet_bootstrap.cc:492] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:45.790113 24174 tablet_bootstrap.cc:654] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.793385 24173 tablet_bootstrap.cc:492] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:45.793582 24173 ts_tablet_manager.cc:1367] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.002s
I20240502 08:16:45.794471 24173 raft_consensus.cc:348] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.794728 24173 raft_consensus.cc:374] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.794827 24173 raft_consensus.cc:729] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.795073 24173 consensus_queue.cc:260] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.797091 24173 ts_tablet_manager.cc:1398] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:16:45.797366 24173 tablet_bootstrap.cc:492] T e686db13bb7541c69ae88f0fb7e27061 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:45.797485 24175 tablet_bootstrap.cc:492] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:45.797643 24175 ts_tablet_manager.cc:1367] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 08:16:45.798318 24175 raft_consensus.cc:348] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.798499 24175 raft_consensus.cc:374] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.798559 24175 raft_consensus.cc:729] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.798785 24175 consensus_queue.cc:260] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.804508 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 0882b7509a74475780eb700818aab398 with delete_type TABLET_DATA_DELETED (replaced by tablet eb96ad2289754e428a1c109b2d59c53b) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:45.805186 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 92adebc0c69e4a52b975253656132495 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92adebc0c69e4a52b975253656132495 already in progress: creating tablet
I20240502 08:16:45.808219 24173 tablet_bootstrap.cc:654] T e686db13bb7541c69ae88f0fb7e27061 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.810267 24174 tablet_bootstrap.cc:492] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:45.810397 24174 ts_tablet_manager.cc:1367] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.006s
I20240502 08:16:45.811090 24174 raft_consensus.cc:348] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.811262 24174 raft_consensus.cc:374] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.811300 24174 raft_consensus.cc:729] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.811517 24174 consensus_queue.cc:260] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.815783 24174 ts_tablet_manager.cc:1398] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:16:45.816134 24174 tablet_bootstrap.cc:492] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:45.817821 24774 raft_consensus.cc:482] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:45.817921 24174 tablet_bootstrap.cc:654] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.817943 24774 raft_consensus.cc:504] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.818521 24774 leader_election.cc:288] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:45.819037 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171406810543435e832a47ad5b3c80a4" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:45.819649 23723 leader_election.cc:341] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:45.822705 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 91646e7af3824520a1fe28e180eeb0a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91646e7af3824520a1fe28e180eeb0a7 already in progress: creating tablet
I20240502 08:16:45.823012 23607 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet f4b291bdb81a4efbac7f6691e5577bbe (table default.Table_2d6b281d29d14441ac6abe9705633102 [id=73559755c4594bcfa591d054964154ae]). Expected version 2 got 0
I20240502 08:16:45.825309 23607 master_service.cc:946] ReplaceTablet: received request to replace tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:45.825868 24181 tablet_replica.cc:330] T 2bfb532c9d4c41afb7acaa4831841306 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:45.826067 24181 raft_consensus.cc:2229] T 2bfb532c9d4c41afb7acaa4831841306 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:45.826192 24181 raft_consensus.cc:2258] T 2bfb532c9d4c41afb7acaa4831841306 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:45.826742 23723 leader_election.cc:341] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 171406810543435e832a47ad5b3c80a4
I20240502 08:16:45.826848 23723 leader_election.cc:302] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:45.827248 24774 raft_consensus.cc:2726] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:45.823038 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet d47b9c1aa6da4ca4975d863d44946e32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d47b9c1aa6da4ca4975d863d44946e32 already in progress: creating tablet
W20240502 08:16:45.828584 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet f5ccd43e78574f3288b199fa14bcc0e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5ccd43e78574f3288b199fa14bcc0e8 already in progress: creating tablet
W20240502 08:16:45.828792 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet ddcb9f236489402593d59ba801615103 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddcb9f236489402593d59ba801615103 already in progress: creating tablet
W20240502 08:16:45.828985 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet d9fcc7accb1f45ad9cf55b8f64d21b91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9fcc7accb1f45ad9cf55b8f64d21b91 already in progress: creating tablet
I20240502 08:16:45.829016 24181 ts_tablet_manager.cc:1875] T 2bfb532c9d4c41afb7acaa4831841306 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:45.829142 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 22240da6da2241bd93b6547a0aae917e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22240da6da2241bd93b6547a0aae917e already in progress: creating tablet
W20240502 08:16:45.829322 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet a17fb4f2f02e43dc9f7cfdb7c73f7774 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a17fb4f2f02e43dc9f7cfdb7c73f7774 already in progress: creating tablet
W20240502 08:16:45.829463 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 058e13ba3a474af3aabebc93c5a88124 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 058e13ba3a474af3aabebc93c5a88124 already in progress: creating tablet
W20240502 08:16:45.829628 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 3abfc0c885d64c39808a6e7c815afd7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3abfc0c885d64c39808a6e7c815afd7e already in progress: creating tablet
W20240502 08:16:45.829782 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet d4b03d759a354a39aea44fd5d4eeb089 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4b03d759a354a39aea44fd5d4eeb089 already in progress: creating tablet
W20240502 08:16:45.829934 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 7dadd4f432554022bc2fdc3c775ac9d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7dadd4f432554022bc2fdc3c775ac9d1 already in progress: creating tablet
W20240502 08:16:45.830083 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet d7e2da9cff594d90819f6b216dbed552 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7e2da9cff594d90819f6b216dbed552 already in progress: creating tablet
W20240502 08:16:45.830236 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet a91e2b50321e4241847d97a558b419f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a91e2b50321e4241847d97a558b419f2 already in progress: creating tablet
W20240502 08:16:45.830408 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 57df4bde56a44809ad43d563baeefecb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57df4bde56a44809ad43d563baeefecb already in progress: creating tablet
I20240502 08:16:45.831514 24173 tablet_bootstrap.cc:492] T e686db13bb7541c69ae88f0fb7e27061 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:45.831647 24173 ts_tablet_manager.cc:1367] T e686db13bb7541c69ae88f0fb7e27061 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20240502 08:16:45.832289 24173 raft_consensus.cc:348] T e686db13bb7541c69ae88f0fb7e27061 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.832790 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet abdcdf7db3e34e77980d4bcc07bcd090 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:45.835302 23607 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet f4b291bdb81a4efbac7f6691e5577bbe (table default.Table_2d6b281d29d14441ac6abe9705633102 [id=73559755c4594bcfa591d054964154ae]). Expected version 2 got 0
I20240502 08:16:45.837136 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e0eb928812a24d1cb82e00b549b36516
W20240502 08:16:45.823076 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4f80e0255d34711841d8c3174e5c69d consensus state has no leader
I20240502 08:16:45.839229 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet e0eb928812a24d1cb82e00b549b36516 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:45.839920 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e0eb928812a24d1cb82e00b549b36516 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0eb928812a24d1cb82e00b549b36516 already in progress: creating tablet
I20240502 08:16:45.840694 24180 ts_tablet_manager.cc:1888] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 08:16:45.840854 24180 log.cc:1196] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/68bdb761ff36426c96949e97279e2756
I20240502 08:16:45.841315 24180 ts_tablet_manager.cc:1909] T 68bdb761ff36426c96949e97279e2756 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:45.842311 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 642) took 1004 ms. Trace:
I20240502 08:16:45.842417 24180 rpcz_store.cc:276] 0502 08:16:44.838253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.853156 (+ 14903us) service_pool.cc:225] Handling call
0502 08:16:45.699154 (+845998us) ts_tablet_manager.cc:1120] Deleting tablet 68bdb761ff36426c96949e97279e2756
0502 08:16:45.699165 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.779571 (+ 80406us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:45.834033 (+ 54462us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:45.842283 (+ 8250us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":845824}
I20240502 08:16:45.832459 24173 raft_consensus.cc:374] T e686db13bb7541c69ae88f0fb7e27061 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.839895 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet e0eb928812a24d1cb82e00b549b36516 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:45.844238 24173 raft_consensus.cc:729] T e686db13bb7541c69ae88f0fb7e27061 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.844470 24173 consensus_queue.cc:260] T e686db13bb7541c69ae88f0fb7e27061 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.849926 24173 ts_tablet_manager.cc:1398] T e686db13bb7541c69ae88f0fb7e27061 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240502 08:16:45.850237 24173 tablet_bootstrap.cc:492] T 4f1736b496354e2dadd844242d3867f8 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:45.851810 24173 tablet_bootstrap.cc:654] T 4f1736b496354e2dadd844242d3867f8 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.852197 24874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:45.853065 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 644) took 1014 ms. Trace:
I20240502 08:16:45.853158 24180 rpcz_store.cc:276] 0502 08:16:44.838419 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.853368 (+ 14949us) service_pool.cc:225] Handling call
0502 08:16:45.852896 (+999528us) ts_tablet_manager.cc:1120] Deleting tablet 92adebc0c69e4a52b975253656132495
0502 08:16:45.852909 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.853040 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":999335}
I20240502 08:16:45.853487 24180 tablet_replica.cc:330] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:45.853683 24180 raft_consensus.cc:2229] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:45.853866 24180 raft_consensus.cc:2258] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:45.854384 24181 ts_tablet_manager.cc:1888] T 2bfb532c9d4c41afb7acaa4831841306 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.856940 24181 log.cc:1196] T 2bfb532c9d4c41afb7acaa4831841306 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/2bfb532c9d4c41afb7acaa4831841306
I20240502 08:16:45.857491 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet e0eb928812a24d1cb82e00b549b36516 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.857882 24180 ts_tablet_manager.cc:1875] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.858000 24181 ts_tablet_manager.cc:1909] T 2bfb532c9d4c41afb7acaa4831841306 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:45.859037 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 2bfb532c9d4c41afb7acaa4831841306 (table default.Table_f48b4cdee51644e0b3c904d772f01404 [id=35259dda872249a7bdddd93783103db8]) successfully deleted
I20240502 08:16:45.860265 23607 master_service.cc:946] ReplaceTablet: received request to replace tablet c5c377e037894728ac084b6500ad20de from {username='slave'} at 127.0.0.1:36668
W20240502 08:16:45.862442 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e0eb928812a24d1cb82e00b549b36516 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0eb928812a24d1cb82e00b549b36516 already in progress: creating tablet
I20240502 08:16:45.864643 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 68bdb761ff36426c96949e97279e2756 (table default.Table_de4adf089f7347969051330f6f43e162 [id=eca4080e8ebf46e9b58c85ee1d8675ca]) successfully deleted
I20240502 08:16:45.864841 24175 ts_tablet_manager.cc:1398] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.067s user 0.015s sys 0.002s
W20240502 08:16:45.865032 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 92adebc0c69e4a52b975253656132495 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92adebc0c69e4a52b975253656132495 already in progress: creating tablet
I20240502 08:16:45.865778 24715 fs_manager.cc:725] Time spent opening block manager: real 0.062s user 0.007s sys 0.001s
I20240502 08:16:45.867727 24175 tablet_bootstrap.cc:492] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:45.869180 24175 tablet_bootstrap.cc:654] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.869691 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_eb2c2e999a154f41bfe1d08fbf5502d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:45.869756 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_989d6e2cbf2547d3859e400799f166c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.870252 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_989d6e2cbf2547d3859e400799f166c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:45.870252 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb2c2e999a154f41bfe1d08fbf5502d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.872435 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2d1d85e9244d4f45b19e7961c5aba776"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.872851 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d1d85e9244d4f45b19e7961c5aba776 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.873883 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ee25478ce0e04c4091efb71e8277beff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.874301 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee25478ce0e04c4091efb71e8277beff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.876645 24175 tablet_bootstrap.cc:492] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:45.876753 24175 ts_tablet_manager.cc:1367] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240502 08:16:45.876988 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet e686db13bb7541c69ae88f0fb7e27061 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:45.877204 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e712916274144e5e86661561646c3521
I20240502 08:16:45.877224 24181 tablet_replica.cc:330] T e686db13bb7541c69ae88f0fb7e27061 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:45.877300 24175 raft_consensus.cc:348] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.877393 24181 raft_consensus.cc:2229] T e686db13bb7541c69ae88f0fb7e27061 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:45.877453 24175 raft_consensus.cc:374] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.877493 24175 raft_consensus.cc:729] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.877540 24181 raft_consensus.cc:2258] T e686db13bb7541c69ae88f0fb7e27061 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:45.877714 24175 consensus_queue.cc:260] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.878186 24181 ts_tablet_manager.cc:1875] T e686db13bb7541c69ae88f0fb7e27061 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.878404 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 71aa916247c449fea718b35869a6df20 with delete_type TABLET_DATA_DELETED (replaced by tablet 0501575a84ec454780f48574bf3155fe) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.879025 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet e712916274144e5e86661561646c3521 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:45.882102 24174 tablet_bootstrap.cc:492] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:45.882475 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet e712916274144e5e86661561646c3521 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:45.883311 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e712916274144e5e86661561646c3521 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e712916274144e5e86661561646c3521 already in progress: creating tablet
I20240502 08:16:45.883989 24175 ts_tablet_manager.cc:1398] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:16:45.884296 24174 ts_tablet_manager.cc:1367] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.068s user 0.007s sys 0.005s
I20240502 08:16:45.885118 24174 raft_consensus.cc:348] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.885298 24174 raft_consensus.cc:374] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.885347 24174 raft_consensus.cc:729] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.885555 24174 consensus_queue.cc:260] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.886082 24174 ts_tablet_manager.cc:1398] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:45.887125 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet e712916274144e5e86661561646c3521 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.887368 24174 tablet_bootstrap.cc:492] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:45.887744 24175 tablet_bootstrap.cc:492] T 0dff0d73a933427fb1f2686bac33eba1 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:45.887120 24180 ts_tablet_manager.cc:1888] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.888166 24180 log.cc:1196] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/8810b6f61db14fed8b12d36fed413dc8
I20240502 08:16:45.888453 24180 ts_tablet_manager.cc:1909] T 8810b6f61db14fed8b12d36fed413dc8 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:45.889199 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 631) took 1052 ms. Trace:
I20240502 08:16:45.889269 24180 rpcz_store.cc:276] 0502 08:16:44.837226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.864637 (+ 27411us) service_pool.cc:225] Handling call
0502 08:16:45.853444 (+988807us) ts_tablet_manager.cc:1120] Deleting tablet 8810b6f61db14fed8b12d36fed413dc8
0502 08:16:45.853452 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.878110 (+ 24658us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:45.887024 (+ 8914us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:45.889184 (+ 2160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":988594}
I20240502 08:16:45.889653 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 646) took 1007 ms. Trace:
I20240502 08:16:45.889734 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 8810b6f61db14fed8b12d36fed413dc8 (table default.Table_ce1b3e3032d84f72b573bf1049c30237 [id=4c18f80dd28145b399b53670c182821c]) successfully deleted
I20240502 08:16:45.889732 24180 rpcz_store.cc:276] 0502 08:16:44.881933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.882204 (+ 271us) service_pool.cc:225] Handling call
0502 08:16:45.889517 (+1007313us) ts_tablet_manager.cc:1120] Deleting tablet 592eeecbf5514215b6464bd030081e4b
0502 08:16:45.889526 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.889638 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1007099}
I20240502 08:16:45.890545 24175 tablet_bootstrap.cc:654] T 0dff0d73a933427fb1f2686bac33eba1 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:45.891109 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 592eeecbf5514215b6464bd030081e4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 592eeecbf5514215b6464bd030081e4b already in progress: creating tablet
I20240502 08:16:45.892549 24181 ts_tablet_manager.cc:1888] T e686db13bb7541c69ae88f0fb7e27061 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.892661 24181 log.cc:1196] T e686db13bb7541c69ae88f0fb7e27061 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/e686db13bb7541c69ae88f0fb7e27061
I20240502 08:16:45.892697 24174 tablet_bootstrap.cc:654] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.893496 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_792c73b104fe4271b1fe9ab5d3445a8c" } new_table_name: "default.Table_0a76b688246e4aaaabcbc5ca97aaa72e" modify_external_catalogs: true
I20240502 08:16:45.893548 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_19bc9c458bb643f7ae59fcd36e6d3b27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.894111 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19bc9c458bb643f7ae59fcd36e6d3b27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.896227 24181 ts_tablet_manager.cc:1909] T e686db13bb7541c69ae88f0fb7e27061 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:45.896531 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 649) took 1012 ms. Trace:
I20240502 08:16:45.896772 24180 rpcz_store.cc:276] 0502 08:16:44.883756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.883819 (+ 63us) service_pool.cc:225] Handling call
0502 08:16:45.896223 (+1012404us) ts_tablet_manager.cc:1120] Deleting tablet 91646e7af3824520a1fe28e180eeb0a7
0502 08:16:45.896232 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.896514 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1012175}
I20240502 08:16:45.897467 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet e686db13bb7541c69ae88f0fb7e27061 (table default.Table_178527d31d7044fc98fa41886d651049 [id=6dc275aed42b42118a70099d3eae3c21]) successfully deleted
W20240502 08:16:45.897511 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 91646e7af3824520a1fe28e180eeb0a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91646e7af3824520a1fe28e180eeb0a7 already in progress: creating tablet
I20240502 08:16:45.897943 24180 tablet_replica.cc:330] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:45.898135 24180 raft_consensus.cc:2229] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:45.898288 24180 raft_consensus.cc:2258] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:45.898756 24180 ts_tablet_manager.cc:1875] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:45.899001 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e712916274144e5e86661561646c3521 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e712916274144e5e86661561646c3521 already in progress: creating tablet
I20240502 08:16:45.905210 24174 tablet_bootstrap.cc:492] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:45.905345 24174 ts_tablet_manager.cc:1367] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20240502 08:16:45.906097 24174 raft_consensus.cc:348] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.906288 24174 raft_consensus.cc:374] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.906347 24174 raft_consensus.cc:729] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.906558 24174 consensus_queue.cc:260] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.907063 24174 ts_tablet_manager.cc:1398] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:45.907380 24174 tablet_bootstrap.cc:492] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:45.909233 24175 tablet_bootstrap.cc:492] T 0dff0d73a933427fb1f2686bac33eba1 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:45.909440 24175 ts_tablet_manager.cc:1367] T 0dff0d73a933427fb1f2686bac33eba1 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.012s
I20240502 08:16:45.910181 24175 raft_consensus.cc:348] T 0dff0d73a933427fb1f2686bac33eba1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.910432 24175 raft_consensus.cc:374] T 0dff0d73a933427fb1f2686bac33eba1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.910545 24175 raft_consensus.cc:729] T 0dff0d73a933427fb1f2686bac33eba1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.910818 24175 consensus_queue.cc:260] T 0dff0d73a933427fb1f2686bac33eba1 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.911340 24175 ts_tablet_manager.cc:1398] T 0dff0d73a933427fb1f2686bac33eba1 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:45.911818 24175 tablet_bootstrap.cc:492] T 171406810543435e832a47ad5b3c80a4 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:45.913097 24175 tablet_bootstrap.cc:654] T 171406810543435e832a47ad5b3c80a4 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.917061 24174 tablet_bootstrap.cc:654] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.920132 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_309cba3acefd434b84e5dabeaeca55e2" } modify_external_catalogs: true
I20240502 08:16:45.920351 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_309cba3acefd434b84e5dabeaeca55e2" } modify_external_catalogs: true
I20240502 08:16:45.922011 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_91d4de88c52a461d8d4006baab592636"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.922721 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91d4de88c52a461d8d4006baab592636 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.926244 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet e686db13bb7541c69ae88f0fb7e27061 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:45.927292 23597 catalog_manager.cc:4797] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e686db13bb7541c69ae88f0fb7e27061 because the tablet was not found. No further retry: Not found: Tablet not found: e686db13bb7541c69ae88f0fb7e27061
I20240502 08:16:45.927803 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bfb532c9d4c41afb7acaa4831841306 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.929018 24174 tablet_bootstrap.cc:492] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:45.867996 24715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/data,/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/wal
uuid: "84f3c07a31414c4f8fb61c44dc9f16e1"
format_stamp: "Formatted at 2024-05-02 08:16:30 on dist-test-slave-m44h"
I20240502 08:16:45.930900 24174 ts_tablet_manager.cc:1367] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240502 08:16:45.931798 24174 raft_consensus.cc:348] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.931993 24174 raft_consensus.cc:374] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.932065 24174 raft_consensus.cc:729] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.932305 24174 consensus_queue.cc:260] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.933287 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 643ce929e74c499da835fbe9b63b8880 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:45.935024 24790 raft_consensus.cc:482] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:45.935212 24790 raft_consensus.cc:504] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.936085 24790 leader_election.cc:288] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:45.930327 24180 ts_tablet_manager.cc:1888] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.936733 24180 log.cc:1196] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/9700ea9dc58442e3ab1b7835a8c1e22c
I20240502 08:16:45.937062 24180 ts_tablet_manager.cc:1909] T 9700ea9dc58442e3ab1b7835a8c1e22c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
W20240502 08:16:45.937469 23861 leader_election.cc:341] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 848a0865eb2b4828a2594a17d96869d0
I20240502 08:16:45.937855 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 651) took 1028 ms. Trace:
I20240502 08:16:45.937945 24180 rpcz_store.cc:276] 0502 08:16:44.909478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.909534 (+ 56us) service_pool.cc:225] Handling call
0502 08:16:45.897901 (+988367us) ts_tablet_manager.cc:1120] Deleting tablet 9700ea9dc58442e3ab1b7835a8c1e22c
0502 08:16:45.897910 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.918318 (+ 20408us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:45.930223 (+ 11905us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:45.937834 (+ 7611us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":988081}
I20240502 08:16:45.938373 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 9700ea9dc58442e3ab1b7835a8c1e22c (table default.Table_9741459a96e649fb8151cc28b642523e [id=2b9169f285754bfda0f4f8dc7f5cb653]) successfully deleted
I20240502 08:16:45.938594 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848a0865eb2b4828a2594a17d96869d0" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:45.938820 23788 raft_consensus.cc:2454] T 848a0865eb2b4828a2594a17d96869d0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:45.932832 24174 ts_tablet_manager.cc:1398] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:45.940205 24174 tablet_bootstrap.cc:492] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:45.940682 23858 leader_election.cc:302] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:45.941218 24790 raft_consensus.cc:2781] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:45.941322 24790 raft_consensus.cc:482] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:45.941394 24790 raft_consensus.cc:3037] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:45.942057 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 650) took 1032 ms. Trace:
I20240502 08:16:45.942134 24180 rpcz_store.cc:276] 0502 08:16:44.909324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:44.922393 (+ 13069us) service_pool.cc:225] Handling call
0502 08:16:45.941930 (+1019537us) ts_tablet_manager.cc:1120] Deleting tablet ef03faf0a13348d7acad954256b5a1d2
0502 08:16:45.941937 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:45.942040 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1019005}
W20240502 08:16:45.942545 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet ef03faf0a13348d7acad954256b5a1d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef03faf0a13348d7acad954256b5a1d2 already in progress: creating tablet
I20240502 08:16:45.943441 24180 tablet_replica.cc:330] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:45.943665 24180 raft_consensus.cc:2229] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:45.943805 24180 raft_consensus.cc:2258] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:45.944319 24180 ts_tablet_manager.cc:1875] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.944775 24790 raft_consensus.cc:504] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.946009 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848a0865eb2b4828a2594a17d96869d0" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:45.946223 23788 raft_consensus.cc:3037] T 848a0865eb2b4828a2594a17d96869d0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:45.946870 24174 tablet_bootstrap.cc:654] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.949337 23788 raft_consensus.cc:2454] T 848a0865eb2b4828a2594a17d96869d0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
W20240502 08:16:45.950788 23861 leader_election.cc:341] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 848a0865eb2b4828a2594a17d96869d0
I20240502 08:16:45.951668 23858 leader_election.cc:302] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:45.945324 24715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:45.952687 24173 tablet_bootstrap.cc:492] T 4f1736b496354e2dadd844242d3867f8 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:45.952764 24790 leader_election.cc:288] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:45.952836 24173 ts_tablet_manager.cc:1367] T 4f1736b496354e2dadd844242d3867f8 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.103s user 0.007s sys 0.013s
I20240502 08:16:45.953202 24790 raft_consensus.cc:2781] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:45.953449 24173 raft_consensus.cc:348] T 4f1736b496354e2dadd844242d3867f8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.953738 24173 raft_consensus.cc:374] T 4f1736b496354e2dadd844242d3867f8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.953796 24173 raft_consensus.cc:729] T 4f1736b496354e2dadd844242d3867f8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.954082 24790 raft_consensus.cc:686] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:16:45.954026 24173 consensus_queue.cc:260] T 4f1736b496354e2dadd844242d3867f8 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.954511 24173 ts_tablet_manager.cc:1398] T 4f1736b496354e2dadd844242d3867f8 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:45.954746 24173 tablet_bootstrap.cc:492] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:45.956413 24174 tablet_bootstrap.cc:492] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:45.956418 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ac9fe60ab8574ba9b6c57d9ffea15fb2" } new_table_name: "default.Table_d3d08ffbf0a743a1ac0642fa87cf6f7e" modify_external_catalogs: true
I20240502 08:16:45.956588 24174 ts_tablet_manager.cc:1367] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240502 08:16:45.957260 24174 raft_consensus.cc:348] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.957486 24174 raft_consensus.cc:374] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.958313 24790 consensus_queue.cc:237] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.958974 24173 tablet_bootstrap.cc:654] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.957841 24774 raft_consensus.cc:482] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:45.961480 24774 raft_consensus.cc:504] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.962278 24774 leader_election.cc:288] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:45.957556 24174 raft_consensus.cc:729] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.963201 24174 consensus_queue.cc:260] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:45.963876 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41f2e98ef2884b05994d02114e992e80 consensus state has no leader
I20240502 08:16:45.964016 24174 ts_tablet_manager.cc:1398] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 08:16:45.964298 24174 tablet_bootstrap.cc:492] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
W20240502 08:16:45.967401 23723 leader_election.cc:341] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 2bfb532c9d4c41afb7acaa4831841306
I20240502 08:16:45.968075 24174 tablet_bootstrap.cc:654] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.969882 24180 ts_tablet_manager.cc:1888] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.970041 24180 log.cc:1196] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/b1fac7df16fc41d7a3297055a7258270
I20240502 08:16:45.970343 24180 ts_tablet_manager.cc:1909] T b1fac7df16fc41d7a3297055a7258270 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:45.971622 24174 tablet_bootstrap.cc:492] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:45.971755 24174 ts_tablet_manager.cc:1367] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240502 08:16:45.972368 24174 raft_consensus.cc:348] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.972544 24174 raft_consensus.cc:374] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.972611 24174 raft_consensus.cc:729] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.972816 24174 consensus_queue.cc:260] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.973227 24173 tablet_bootstrap.cc:492] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:45.973357 24173 ts_tablet_manager.cc:1367] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20240502 08:16:45.973212 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7b3b1098fbed4d2792dadff76cbc948b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:45.973855 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b3b1098fbed4d2792dadff76cbc948b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:45.974190 24173 raft_consensus.cc:348] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.974385 24173 raft_consensus.cc:374] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:45.974431 24173 raft_consensus.cc:729] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:45.974637 24173 consensus_queue.cc:260] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.975229 24173 ts_tablet_manager.cc:1398] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:45.975522 24173 tablet_bootstrap.cc:492] T 4a57ed188315493198e95cefb6907ab6 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:45.976032 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1736b496354e2dadd844242d3867f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 6746c25111d04f20a6f30ce0838d71f5) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:45.976447 24181 tablet_replica.cc:330] T 4f1736b496354e2dadd844242d3867f8 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:45.976609 24181 raft_consensus.cc:2229] T 4f1736b496354e2dadd844242d3867f8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:45.976743 24181 raft_consensus.cc:2258] T 4f1736b496354e2dadd844242d3867f8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:45.977643 24173 tablet_bootstrap.cc:654] T 4a57ed188315493198e95cefb6907ab6 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.977826 24778 raft_consensus.cc:482] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:45.977955 24778 raft_consensus.cc:504] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:45.978143 24174 ts_tablet_manager.cc:1398] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:16:45.978567 24778 leader_election.cc:288] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:45.979681 23723 leader_election.cc:341] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: a3043c5ed98c401b82c6e967ca3f3613
I20240502 08:16:45.980448 24181 ts_tablet_manager.cc:1875] T 4f1736b496354e2dadd844242d3867f8 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:45.980947 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet b1fac7df16fc41d7a3297055a7258270 (table default.Table_bcd1014009b046a7ba72d1d757e69e26 [id=e6823667e67a470e870f4ed70fc58a85]) successfully deleted
I20240502 08:16:45.986598 24180 tablet_replica.cc:330] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:45.986843 24180 raft_consensus.cc:2229] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:45.987056 24180 raft_consensus.cc:2258] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:45.987685 24180 ts_tablet_manager.cc:1875] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:45.990067 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet d47b9c1aa6da4ca4975d863d44946e32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d47b9c1aa6da4ca4975d863d44946e32 already in progress: creating tablet
I20240502 08:16:45.991127 24174 tablet_bootstrap.cc:492] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:45.992161 24181 ts_tablet_manager.cc:1888] T 4f1736b496354e2dadd844242d3867f8 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.992285 24181 log.cc:1196] T 4f1736b496354e2dadd844242d3867f8 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/4f1736b496354e2dadd844242d3867f8
I20240502 08:16:45.992594 24181 ts_tablet_manager.cc:1909] T 4f1736b496354e2dadd844242d3867f8 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:45.994103 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 4f1736b496354e2dadd844242d3867f8 (table default.Table_02f448a8a9844def8ecda8716fdc352e [id=79a03d22b7e34f76bb3242c640129a13]) successfully deleted
I20240502 08:16:45.995978 24174 tablet_bootstrap.cc:654] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:45.998502 24180 ts_tablet_manager.cc:1888] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:45.999925 24180 log.cc:1196] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/c08228fcd1304528b9446003851f6c70
I20240502 08:16:46.000563 24180 ts_tablet_manager.cc:1909] T c08228fcd1304528b9446003851f6c70 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:46.002352 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet c08228fcd1304528b9446003851f6c70 (table default.Table_bbf9d7694fe942de8aa8bd4ed61336a5 [id=f1d6eb661b0143ee82e797cde61391c0]) successfully deleted
W20240502 08:16:46.005060 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet f5ccd43e78574f3288b199fa14bcc0e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5ccd43e78574f3288b199fa14bcc0e8 already in progress: creating tablet
I20240502 08:16:46.006062 24180 tablet_replica.cc:330] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:46.006844 24180 raft_consensus.cc:2229] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.007652 24180 raft_consensus.cc:2258] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.008780 24180 ts_tablet_manager.cc:1875] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:46.009852 23858 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 58210.3us
I20240502 08:16:46.010496 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bfb532c9d4c41afb7acaa4831841306" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:46.011281 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3043c5ed98c401b82c6e967ca3f3613" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:46.012543 23723 leader_election.cc:341] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.012679 23723 leader_election.cc:302] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
W20240502 08:16:46.013118 23723 leader_election.cc:341] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.013226 23723 leader_election.cc:302] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:46.013813 24778 raft_consensus.cc:2726] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:46.013998 24778 raft_consensus.cc:2726] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:46.026831 24174 tablet_bootstrap.cc:492] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:46.027808 24174 ts_tablet_manager.cc:1367] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.003s
I20240502 08:16:46.029609 24174 raft_consensus.cc:348] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.031186 24842 raft_consensus.cc:482] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.031395 24842 raft_consensus.cc:504] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.032264 24842 leader_election.cc:288] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:46.032965 23608 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet b811b81ddf2b4dc9b5e5f80331bff294 (table default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418]). Expected version 2 got 0
I20240502 08:16:46.032897 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b291bdb81a4efbac7f6691e5577bbe" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:46.033464 23608 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet b811b81ddf2b4dc9b5e5f80331bff294 (table default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet b811b81ddf2b4dc9b5e5f80331bff294 consensus state has no leader
I20240502 08:16:46.033828 24174 raft_consensus.cc:374] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.034086 24174 raft_consensus.cc:729] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.034368 24174 consensus_queue.cc:260] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.035238 23610 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet d1be2c29d3154d9189af3266191f0b3e (table default.Table_db86cd2cf86e48ffbc6a513b5c2bd7ab [id=6103f133bcdd4f768b92b37245bf5356]). Expected version 1 got 0
W20240502 08:16:46.035764 23996 leader_election.cc:341] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.036090 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b291bdb81a4efbac7f6691e5577bbe" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:46.036283 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c0fffcc987024f0b876f772a57cb68b6" } new_table_name: "default.Table_03bc1e423ebc4377b041699c570137b1" modify_external_catalogs: true
W20240502 08:16:46.036491 23993 leader_election.cc:341] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.036576 23993 leader_election.cc:302] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:46.036901 24842 raft_consensus.cc:2726] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:46.037612 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_20e4d00ccdf240fd922450cb02438505" } new_table_name: "default.Table_7e325050ee86487aa23e9b0edfbd2e1d" modify_external_catalogs: true
I20240502 08:16:46.038002 23610 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet b811b81ddf2b4dc9b5e5f80331bff294 (table default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418]). Expected version 2 got 0
I20240502 08:16:46.039052 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet e686db13bb7541c69ae88f0fb7e27061 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:46.039570 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_22be413d6296475eb6266315d70808d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:46.040040 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22be413d6296475eb6266315d70808d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:46.042183 24174 ts_tablet_manager.cc:1398] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240502 08:16:46.042528 24174 tablet_bootstrap.cc:492] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:46.043123 23900 tablet_service.cc:1467] Processing CreateTablet for tablet df87b8b1ff504031bf007880792192dd (DEFAULT_TABLE table=default.Table_213f4bea9fad44c4a1aa726aed361575 [id=c0b7b1ec67d84ac19f50cd7be15e3154]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.043521 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df87b8b1ff504031bf007880792192dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.050046 24174 tablet_bootstrap.cc:654] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.051337 24180 ts_tablet_manager.cc:1888] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:46.051467 24180 log.cc:1196] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/5453cc0b93ac4cbb90a0588bb2100d60
I20240502 08:16:46.051841 24180 ts_tablet_manager.cc:1909] T 5453cc0b93ac4cbb90a0588bb2100d60 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:46.051868 24039 tablet_service.cc:1467] Processing CreateTablet for tablet df87b8b1ff504031bf007880792192dd (DEFAULT_TABLE table=default.Table_213f4bea9fad44c4a1aa726aed361575 [id=c0b7b1ec67d84ac19f50cd7be15e3154]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.052254 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df87b8b1ff504031bf007880792192dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.052958 24180 tablet_replica.cc:330] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:46.053141 24180 raft_consensus.cc:2229] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.053277 24180 raft_consensus.cc:2258] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.053783 24180 ts_tablet_manager.cc:1875] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.054734 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 5453cc0b93ac4cbb90a0588bb2100d60 (table default.Table_a92f0fc8bc664448a6acb94dec9daa14 [id=8c859fafcd4b49f0b4e9fc9d8961c9ad]) successfully deleted
W20240502 08:16:46.055050 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet d9fcc7accb1f45ad9cf55b8f64d21b91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9fcc7accb1f45ad9cf55b8f64d21b91 already in progress: creating tablet
W20240502 08:16:46.055263 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet ddcb9f236489402593d59ba801615103 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddcb9f236489402593d59ba801615103 already in progress: creating tablet
I20240502 08:16:46.061522 24778 raft_consensus.cc:482] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.061691 23761 tablet_service.cc:1467] Processing CreateTablet for tablet df87b8b1ff504031bf007880792192dd (DEFAULT_TABLE table=default.Table_213f4bea9fad44c4a1aa726aed361575 [id=c0b7b1ec67d84ac19f50cd7be15e3154]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.061738 24778 raft_consensus.cc:504] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.062091 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df87b8b1ff504031bf007880792192dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.062357 24778 leader_election.cc:288] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:46.066515 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dff0d73a933427fb1f2686bac33eba1" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:46.066776 23925 raft_consensus.cc:2454] T 0dff0d73a933427fb1f2686bac33eba1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
W20240502 08:16:46.067590 23723 leader_election.cc:341] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 0dff0d73a933427fb1f2686bac33eba1
I20240502 08:16:46.068162 23723 leader_election.cc:302] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:46.068518 24778 raft_consensus.cc:2781] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:46.068603 24778 raft_consensus.cc:482] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:46.068706 24778 raft_consensus.cc:3037] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.071084 24778 raft_consensus.cc:504] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.071722 24778 leader_election.cc:288] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:46.071702 24885 raft_consensus.cc:482] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.072700 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dff0d73a933427fb1f2686bac33eba1" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:46.072916 23925 raft_consensus.cc:3037] T 0dff0d73a933427fb1f2686bac33eba1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.074307 24885 raft_consensus.cc:504] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:46.074532 23723 leader_election.cc:341] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 0dff0d73a933427fb1f2686bac33eba1
I20240502 08:16:46.076161 23925 raft_consensus.cc:2454] T 0dff0d73a933427fb1f2686bac33eba1 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
W20240502 08:16:46.076231 23861 leader_election.cc:341] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 9c03a4353c4e46e3bcdf3a36aac04bf5
I20240502 08:16:46.078716 24885 leader_election.cc:288] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:46.079399 23723 leader_election.cc:302] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:46.080209 24180 ts_tablet_manager.cc:1888] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:46.080377 24180 log.cc:1196] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/c5d28098dcbc4761a34ee65769d6fd7c
I20240502 08:16:46.080582 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c03a4353c4e46e3bcdf3a36aac04bf5" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:46.080755 24180 ts_tablet_manager.cc:1909] T c5d28098dcbc4761a34ee65769d6fd7c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:46.080848 23788 raft_consensus.cc:2454] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:46.081403 24778 raft_consensus.cc:2781] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:46.081806 24778 raft_consensus.cc:686] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:46.082167 24778 consensus_queue.cc:237] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.082469 23607 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 798c8bb81d9f44c797ad1093d7790336
I20240502 08:16:46.084035 23858 leader_election.cc:302] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:46.084648 24885 raft_consensus.cc:2781] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:46.084746 24885 raft_consensus.cc:482] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:46.084841 24885 raft_consensus.cc:3037] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.085616 24891 raft_consensus.cc:482] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.085762 24891 raft_consensus.cc:504] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.086386 24891 leader_election.cc:288] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:46.087671 23723 leader_election.cc:341] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 71aa916247c449fea718b35869a6df20
I20240502 08:16:46.089890 24180 tablet_replica.cc:330] T 848a0865eb2b4828a2594a17d96869d0 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:46.090523 24180 raft_consensus.cc:2229] T 848a0865eb2b4828a2594a17d96869d0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.091060 24885 raft_consensus.cc:504] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.091276 23607 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet d1be2c29d3154d9189af3266191f0b3e (table default.Table_db86cd2cf86e48ffbc6a513b5c2bd7ab [id=6103f133bcdd4f768b92b37245bf5356]). Expected version 1 got 0
I20240502 08:16:46.091361 24180 raft_consensus.cc:2258] T 848a0865eb2b4828a2594a17d96869d0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.091936 24180 ts_tablet_manager.cc:1875] T 848a0865eb2b4828a2594a17d96869d0 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.092497 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c03a4353c4e46e3bcdf3a36aac04bf5" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:46.092289 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet c5d28098dcbc4761a34ee65769d6fd7c (table default.Table_81d96be059774b5bbaf8c8d21b2935ce [id=dd9c1fd15bf446fd87bd3647cb7f041d]) successfully deleted
I20240502 08:16:46.092753 23788 raft_consensus.cc:3037] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.095528 23788 raft_consensus.cc:2454] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:46.095997 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71aa916247c449fea718b35869a6df20" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:46.097066 23858 leader_election.cc:302] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:46.097625 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dff0d73a933427fb1f2686bac33eba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:46.097832 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 848a0865eb2b4828a2594a17d96869d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:46.097986 24179 tablet_replica.cc:330] T 0dff0d73a933427fb1f2686bac33eba1 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:46.098134 24179 raft_consensus.cc:2229] T 0dff0d73a933427fb1f2686bac33eba1 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.098251 24179 raft_consensus.cc:2258] T 0dff0d73a933427fb1f2686bac33eba1 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:46.098542 23723 leader_election.cc:341] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.098701 23723 leader_election.cc:302] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:46.099053 24179 ts_tablet_manager.cc:1875] T 0dff0d73a933427fb1f2686bac33eba1 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.099308 24778 raft_consensus.cc:2726] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:46.099581 24790 raft_consensus.cc:2781] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:46.102058 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_377dcfeddb374dfb8b977d526c8c77e4" } modify_external_catalogs: true
I20240502 08:16:46.102391 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:46.102772 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1736b496354e2dadd844242d3867f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 6746c25111d04f20a6f30ce0838d71f5) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:46.105538 24179 ts_tablet_manager.cc:1888] T 0dff0d73a933427fb1f2686bac33eba1 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:46.105698 24179 log.cc:1196] T 0dff0d73a933427fb1f2686bac33eba1 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/0dff0d73a933427fb1f2686bac33eba1
I20240502 08:16:46.106003 24179 ts_tablet_manager.cc:1909] T 0dff0d73a933427fb1f2686bac33eba1 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:46.106827 24179 tablet_replica.cc:330] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:46.107013 24179 raft_consensus.cc:2229] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:46.107295 24179 pending_rounds.cc:62] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b: Trying to abort 1 pending ops.
I20240502 08:16:46.107350 24179 pending_rounds.cc:68] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156453277249536 op_type: NO_OP noop_request { }
I20240502 08:16:46.107468 24179 raft_consensus.cc:2866] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:46.107546 24179 raft_consensus.cc:2258] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.108983 24179 ts_tablet_manager.cc:1875] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.110970 24180 ts_tablet_manager.cc:1888] T 848a0865eb2b4828a2594a17d96869d0 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:46.112036 24180 log.cc:1196] T 848a0865eb2b4828a2594a17d96869d0 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/848a0865eb2b4828a2594a17d96869d0
I20240502 08:16:46.113237 24180 ts_tablet_manager.cc:1909] T 848a0865eb2b4828a2594a17d96869d0 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:46.118531 24175 tablet_bootstrap.cc:492] T 171406810543435e832a47ad5b3c80a4 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:46.118687 24175 ts_tablet_manager.cc:1367] T 171406810543435e832a47ad5b3c80a4 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.207s user 0.057s sys 0.087s
I20240502 08:16:46.119382 24175 raft_consensus.cc:348] T 171406810543435e832a47ad5b3c80a4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.119601 24175 raft_consensus.cc:374] T 171406810543435e832a47ad5b3c80a4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.119657 24175 raft_consensus.cc:729] T 171406810543435e832a47ad5b3c80a4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.119987 24180 tablet_replica.cc:330] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:46.119877 24175 consensus_queue.cc:260] T 171406810543435e832a47ad5b3c80a4 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.106583 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 0dff0d73a933427fb1f2686bac33eba1 (table default.Table_6ccfa5bc466242b2a325aa1ce6c85e9c [id=9076329a97254502a8c10b167c7f5b41]) successfully deleted
I20240502 08:16:46.120752 24180 raft_consensus.cc:2229] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:46.121034 24180 pending_rounds.cc:62] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:16:46.121119 24180 pending_rounds.cc:68] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156453715210240 op_type: NO_OP noop_request { }
I20240502 08:16:46.121273 24180 raft_consensus.cc:2866] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:46.121395 24180 raft_consensus.cc:2258] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:46.121449 23861 leader_election.cc:341] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 9c03a4353c4e46e3bcdf3a36aac04bf5
I20240502 08:16:46.121817 24885 leader_election.cc:288] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:46.122617 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 6c49f172372448c2be4909ae36371267 (DEFAULT_TABLE table=default.Table_ff8f3ce7d85c49fabe0e3ef763bd7517 [id=ff9209b28981436aaadedaf4f29c7afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.122881 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 14f54d6ded85434da35c1dfcff52763a (DEFAULT_TABLE table=default.Table_0751e871f6e14c0697044070629e6d26 [id=186aa7acd94149988b8ad7484947c02b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.123550 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 765fdb66f12941f0ac16223e931e576e (DEFAULT_TABLE table=default.Table_2439a8f2e30f4bdf8b18d7ac811ab586 [id=d4a0f9e9a1024be8802a911a7c95caa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.124086 24037 tablet_service.cc:1467] Processing CreateTablet for tablet c7e48620d7434dde909bf6b4a6a21f16 (DEFAULT_TABLE table=default.Table_8d3e812b72644192aa5a83f9754ff725 [id=3a2e125794b4483dbd01ab2c66e375ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.124542 24025 tablet_service.cc:1467] Processing CreateTablet for tablet becb61dbe43340cd8aab25d5e5a77bf7 (DEFAULT_TABLE table=default.Table_36e587dae1614924b2696665c53a1e6b [id=6e904a5e38354dafa98cbc3ea5d8f442]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.125156 24031 tablet_service.cc:1467] Processing CreateTablet for tablet da1f3f79cbe94fafa747db1d747be8df (DEFAULT_TABLE table=default.Table_5c021393c18a4030b8894b73c1b8acff [id=07875553d04b46c9baba26b68a83e6cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.125607 24180 ts_tablet_manager.cc:1875] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.115273 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 848a0865eb2b4828a2594a17d96869d0 (table default.Table_7bae786c8c2e44bd9e893ff90696d216 [id=7ffacb15a8a249258fb42c4312f87d69]) successfully deleted
I20240502 08:16:46.126080 24790 raft_consensus.cc:686] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:16:46.126401 24790 consensus_queue.cc:237] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:46.126910 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 22240da6da2241bd93b6547a0aae917e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22240da6da2241bd93b6547a0aae917e already in progress: creating tablet
W20240502 08:16:46.127261 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet a17fb4f2f02e43dc9f7cfdb7c73f7774 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a17fb4f2f02e43dc9f7cfdb7c73f7774 already in progress: creating tablet
W20240502 08:16:46.127496 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 49b816ea1d474f2db94119441146ba78 because the tablet was not found. No further retry: Not found: Tablet not found: 49b816ea1d474f2db94119441146ba78
I20240502 08:16:46.127663 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 14f54d6ded85434da35c1dfcff52763a (DEFAULT_TABLE table=default.Table_0751e871f6e14c0697044070629e6d26 [id=186aa7acd94149988b8ad7484947c02b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.128060 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f54d6ded85434da35c1dfcff52763a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.130424 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 14f54d6ded85434da35c1dfcff52763a (DEFAULT_TABLE table=default.Table_0751e871f6e14c0697044070629e6d26 [id=186aa7acd94149988b8ad7484947c02b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.130796 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f54d6ded85434da35c1dfcff52763a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.130849 24179 ts_tablet_manager.cc:1888] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:46.130931 24179 log.cc:1196] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/848a0865eb2b4828a2594a17d96869d0
I20240502 08:16:46.130975 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 6c49f172372448c2be4909ae36371267 (DEFAULT_TABLE table=default.Table_ff8f3ce7d85c49fabe0e3ef763bd7517 [id=ff9209b28981436aaadedaf4f29c7afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.131239 24179 ts_tablet_manager.cc:1909] T 848a0865eb2b4828a2594a17d96869d0 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:46.131327 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c49f172372448c2be4909ae36371267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.131618 24179 tablet_replica.cc:330] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:46.132215 24175 ts_tablet_manager.cc:1398] T 171406810543435e832a47ad5b3c80a4 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.013s user 0.001s sys 0.002s
I20240502 08:16:46.132458 24175 tablet_bootstrap.cc:492] T 7c374f42f353497fa1d745164e7fbf61 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:46.133687 24175 tablet_bootstrap.cc:654] T 7c374f42f353497fa1d745164e7fbf61 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.133818 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 765fdb66f12941f0ac16223e931e576e (DEFAULT_TABLE table=default.Table_2439a8f2e30f4bdf8b18d7ac811ab586 [id=d4a0f9e9a1024be8802a911a7c95caa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.133846 23893 tablet_service.cc:1467] Processing CreateTablet for tablet c7e48620d7434dde909bf6b4a6a21f16 (DEFAULT_TABLE table=default.Table_8d3e812b72644192aa5a83f9754ff725 [id=3a2e125794b4483dbd01ab2c66e375ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.133891 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 765fdb66f12941f0ac16223e931e576e (DEFAULT_TABLE table=default.Table_2439a8f2e30f4bdf8b18d7ac811ab586 [id=d4a0f9e9a1024be8802a911a7c95caa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.134115 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 765fdb66f12941f0ac16223e931e576e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.134229 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 765fdb66f12941f0ac16223e931e576e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.134361 23886 tablet_service.cc:1467] Processing CreateTablet for tablet becb61dbe43340cd8aab25d5e5a77bf7 (DEFAULT_TABLE table=default.Table_36e587dae1614924b2696665c53a1e6b [id=6e904a5e38354dafa98cbc3ea5d8f442]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.134642 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet becb61dbe43340cd8aab25d5e5a77bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.137818 23896 tablet_service.cc:1467] Processing CreateTablet for tablet da1f3f79cbe94fafa747db1d747be8df (DEFAULT_TABLE table=default.Table_5c021393c18a4030b8894b73c1b8acff [id=07875553d04b46c9baba26b68a83e6cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.137846 24778 raft_consensus.cc:482] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.138137 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da1f3f79cbe94fafa747db1d747be8df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.138078 23767 tablet_service.cc:1467] Processing CreateTablet for tablet c7e48620d7434dde909bf6b4a6a21f16 (DEFAULT_TABLE table=default.Table_8d3e812b72644192aa5a83f9754ff725 [id=3a2e125794b4483dbd01ab2c66e375ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.138046 24778 raft_consensus.cc:504] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.138423 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e48620d7434dde909bf6b4a6a21f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.138664 24778 leader_election.cc:288] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:46.137871 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 6c49f172372448c2be4909ae36371267 (DEFAULT_TABLE table=default.Table_ff8f3ce7d85c49fabe0e3ef763bd7517 [id=ff9209b28981436aaadedaf4f29c7afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.139106 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c49f172372448c2be4909ae36371267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.141829 23764 tablet_service.cc:1467] Processing CreateTablet for tablet becb61dbe43340cd8aab25d5e5a77bf7 (DEFAULT_TABLE table=default.Table_36e587dae1614924b2696665c53a1e6b [id=6e904a5e38354dafa98cbc3ea5d8f442]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.141943 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e48620d7434dde909bf6b4a6a21f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.142135 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet becb61dbe43340cd8aab25d5e5a77bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.123028 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c49f172372448c2be4909ae36371267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.145881 23766 tablet_service.cc:1467] Processing CreateTablet for tablet da1f3f79cbe94fafa747db1d747be8df (DEFAULT_TABLE table=default.Table_5c021393c18a4030b8894b73c1b8acff [id=07875553d04b46c9baba26b68a83e6cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.146224 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da1f3f79cbe94fafa747db1d747be8df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.149708 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e48620d7434dde909bf6b4a6a21f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.153157 24175 tablet_bootstrap.cc:492] T 7c374f42f353497fa1d745164e7fbf61 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:46.153268 24175 ts_tablet_manager.cc:1367] T 7c374f42f353497fa1d745164e7fbf61 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.004s
I20240502 08:16:46.153627 24179 raft_consensus.cc:2229] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:46.153978 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet becb61dbe43340cd8aab25d5e5a77bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.154155 24179 pending_rounds.cc:62] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b: Trying to abort 1 pending ops.
I20240502 08:16:46.154208 24179 pending_rounds.cc:68] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156453897752576 op_type: NO_OP noop_request { }
I20240502 08:16:46.154316 24179 raft_consensus.cc:2866] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:46.154388 24179 raft_consensus.cc:2258] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.154516 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da1f3f79cbe94fafa747db1d747be8df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.154949 24175 raft_consensus.cc:348] T 7c374f42f353497fa1d745164e7fbf61 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.155109 24175 raft_consensus.cc:374] T 7c374f42f353497fa1d745164e7fbf61 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.155156 24175 raft_consensus.cc:729] T 7c374f42f353497fa1d745164e7fbf61 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.155314 24175 consensus_queue.cc:260] T 7c374f42f353497fa1d745164e7fbf61 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.155802 24175 ts_tablet_manager.cc:1398] T 7c374f42f353497fa1d745164e7fbf61 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:46.156100 24175 tablet_bootstrap.cc:492] T c5d28098dcbc4761a34ee65769d6fd7c P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:46.157274 24175 tablet_bootstrap.cc:654] T c5d28098dcbc4761a34ee65769d6fd7c P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.157727 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 765fdb66f12941f0ac16223e931e576e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.158862 24179 ts_tablet_manager.cc:1875] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.162068 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f54d6ded85434da35c1dfcff52763a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.163205 23761 tablet_service.cc:1467] Processing CreateTablet for tablet df266081ed8345dda989b3050f605685 (DEFAULT_TABLE table=default.Table_212251b2938346e19add6081bc09d254 [id=1b94a6dd42d74dd58ccb3cef790ac3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.163578 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df266081ed8345dda989b3050f605685. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.165757 24778 raft_consensus.cc:482] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.165920 24778 raft_consensus.cc:504] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.166508 24778 leader_election.cc:288] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:46.167047 24179 ts_tablet_manager.cc:1888] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:46.167173 24179 log.cc:1196] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/9c03a4353c4e46e3bcdf3a36aac04bf5
I20240502 08:16:46.167541 24179 ts_tablet_manager.cc:1909] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:46.167483 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27346a7eae6a488fa3415646262a4b88" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:46.167675 24057 raft_consensus.cc:2454] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
W20240502 08:16:46.174113 23723 leader_election.cc:341] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 7c374f42f353497fa1d745164e7fbf61
I20240502 08:16:46.174327 23723 leader_election.cc:302] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:46.174676 24778 raft_consensus.cc:2781] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:46.174750 24778 raft_consensus.cc:482] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:46.174821 24778 raft_consensus.cc:3037] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.177148 24778 raft_consensus.cc:504] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.179191 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27346a7eae6a488fa3415646262a4b88" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:46.180148 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 848a0865eb2b4828a2594a17d96869d0 (table default.Table_7bae786c8c2e44bd9e893ff90696d216 [id=7ffacb15a8a249258fb42c4312f87d69]) successfully deleted
I20240502 08:16:46.180696 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 (table default.Table_9663c413d5be4a049eb7a02ececc1276 [id=ad4f0c8adc524a019beaf37e153b33a7]) successfully deleted
W20240502 08:16:46.184432 23723 leader_election.cc:341] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.185992 24175 tablet_bootstrap.cc:492] T c5d28098dcbc4761a34ee65769d6fd7c P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:46.186161 24175 ts_tablet_manager.cc:1367] T c5d28098dcbc4761a34ee65769d6fd7c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20240502 08:16:46.186791 24175 raft_consensus.cc:348] T c5d28098dcbc4761a34ee65769d6fd7c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.186973 24175 raft_consensus.cc:374] T c5d28098dcbc4761a34ee65769d6fd7c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.187024 24175 raft_consensus.cc:729] T c5d28098dcbc4761a34ee65769d6fd7c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.187253 24175 consensus_queue.cc:260] T c5d28098dcbc4761a34ee65769d6fd7c P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.187748 24175 ts_tablet_manager.cc:1398] T c5d28098dcbc4761a34ee65769d6fd7c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:46.188031 24175 tablet_bootstrap.cc:492] T b1fac7df16fc41d7a3297055a7258270 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:46.189399 24175 tablet_bootstrap.cc:654] T b1fac7df16fc41d7a3297055a7258270 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.194074 24175 tablet_bootstrap.cc:492] T b1fac7df16fc41d7a3297055a7258270 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:46.194211 24175 ts_tablet_manager.cc:1367] T b1fac7df16fc41d7a3297055a7258270 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:16:46.194850 24175 raft_consensus.cc:348] T b1fac7df16fc41d7a3297055a7258270 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.195047 24175 raft_consensus.cc:374] T b1fac7df16fc41d7a3297055a7258270 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.195109 24175 raft_consensus.cc:729] T b1fac7df16fc41d7a3297055a7258270 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.195322 24175 consensus_queue.cc:260] T b1fac7df16fc41d7a3297055a7258270 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.197476 24778 leader_election.cc:288] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:46.198192 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27346a7eae6a488fa3415646262a4b88" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:46.198383 24057 raft_consensus.cc:3037] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.198606 24175 ts_tablet_manager.cc:1398] T b1fac7df16fc41d7a3297055a7258270 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:46.200796 24057 raft_consensus.cc:2454] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:46.201493 24175 tablet_bootstrap.cc:492] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:46.201779 23723 leader_election.cc:302] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:46.202702 24180 ts_tablet_manager.cc:1888] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:46.202817 24180 log.cc:1196] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/0dff0d73a933427fb1f2686bac33eba1
I20240502 08:16:46.202919 24175 tablet_bootstrap.cc:654] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.203194 24180 ts_tablet_manager.cc:1909] T 0dff0d73a933427fb1f2686bac33eba1 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:46.207419 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 0dff0d73a933427fb1f2686bac33eba1 (table default.Table_6ccfa5bc466242b2a325aa1ce6c85e9c [id=9076329a97254502a8c10b167c7f5b41]) successfully deleted
I20240502 08:16:46.209013 24180 tablet_replica.cc:330] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:46.209208 24180 raft_consensus.cc:2229] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.210343 24180 raft_consensus.cc:2258] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.210886 24180 ts_tablet_manager.cc:1875] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.215020 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27346a7eae6a488fa3415646262a4b88" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:46.215785 23723 leader_election.cc:341] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.218706 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c374f42f353497fa1d745164e7fbf61" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:46.219023 23925 raft_consensus.cc:2454] T 7c374f42f353497fa1d745164e7fbf61 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:46.219388 24778 raft_consensus.cc:2781] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:46.219941 24778 raft_consensus.cc:686] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:46.220585 24778 consensus_queue.cc:237] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.222780 23900 tablet_service.cc:1467] Processing CreateTablet for tablet df266081ed8345dda989b3050f605685 (DEFAULT_TABLE table=default.Table_212251b2938346e19add6081bc09d254 [id=1b94a6dd42d74dd58ccb3cef790ac3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.223299 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df266081ed8345dda989b3050f605685. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.226557 24039 tablet_service.cc:1467] Processing CreateTablet for tablet df266081ed8345dda989b3050f605685 (DEFAULT_TABLE table=default.Table_212251b2938346e19add6081bc09d254 [id=1b94a6dd42d74dd58ccb3cef790ac3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.226817 24180 ts_tablet_manager.cc:1888] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:46.227074 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df266081ed8345dda989b3050f605685. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.230499 24180 log.cc:1196] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/171406810543435e832a47ad5b3c80a4
I20240502 08:16:46.230631 23723 leader_election.cc:302] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:46.231107 24891 raft_consensus.cc:2781] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:46.231181 24891 raft_consensus.cc:482] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:46.231232 24891 raft_consensus.cc:3037] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.232160 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_377dcfeddb374dfb8b977d526c8c77e4" } modify_external_catalogs: true
I20240502 08:16:46.233814 24891 raft_consensus.cc:504] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.234403 24891 leader_election.cc:288] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:46.235913 23723 leader_election.cc:341] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 7c374f42f353497fa1d745164e7fbf61
I20240502 08:16:46.236486 24180 ts_tablet_manager.cc:1909] T 171406810543435e832a47ad5b3c80a4 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:46.237172 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c374f42f353497fa1d745164e7fbf61" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:46.237639 23925 raft_consensus.cc:3037] T 7c374f42f353497fa1d745164e7fbf61 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.238607 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 171406810543435e832a47ad5b3c80a4 (table default.Table_a286e67e441e43a1b4738cd219dbea98 [id=b56e3c0686244ad6b49afdd589c5ed34]) successfully deleted
I20240502 08:16:46.240691 23925 raft_consensus.cc:2454] T 7c374f42f353497fa1d745164e7fbf61 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
W20240502 08:16:46.241065 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 058e13ba3a474af3aabebc93c5a88124 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 058e13ba3a474af3aabebc93c5a88124 already in progress: creating tablet
I20240502 08:16:46.241274 23723 leader_election.cc:302] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:46.241866 24891 raft_consensus.cc:2781] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:46.241997 24891 raft_consensus.cc:686] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
W20240502 08:16:46.242182 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 7dadd4f432554022bc2fdc3c775ac9d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7dadd4f432554022bc2fdc3c775ac9d1 already in progress: creating tablet
I20240502 08:16:46.242251 24891 consensus_queue.cc:237] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:46.243185 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 3abfc0c885d64c39808a6e7c815afd7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3abfc0c885d64c39808a6e7c815afd7e already in progress: creating tablet
W20240502 08:16:46.245185 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet d4b03d759a354a39aea44fd5d4eeb089 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4b03d759a354a39aea44fd5d4eeb089 already in progress: creating tablet
W20240502 08:16:46.246876 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet d7e2da9cff594d90819f6b216dbed552 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7e2da9cff594d90819f6b216dbed552 already in progress: creating tablet
I20240502 08:16:46.247258 24180 tablet_replica.cc:330] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:46.247876 24180 raft_consensus.cc:2229] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.248363 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_52e9e35e99234042ae5e823f448c5cb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:46.249063 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52e9e35e99234042ae5e823f448c5cb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:46.249068 24180 raft_consensus.cc:2258] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.249650 24180 ts_tablet_manager.cc:1875] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:46.259999 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 56d6373938824840934211f259e14d4f has no consensus state
I20240502 08:16:46.260113 24180 ts_tablet_manager.cc:1888] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:46.260205 24180 log.cc:1196] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/9c03a4353c4e46e3bcdf3a36aac04bf5
I20240502 08:16:46.260500 24180 ts_tablet_manager.cc:1909] T 9c03a4353c4e46e3bcdf3a36aac04bf5 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:46.266880 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_06da23829ca541b7ae4be7b15db26c34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:46.267653 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06da23829ca541b7ae4be7b15db26c34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:46.271656 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 (table default.Table_9663c413d5be4a049eb7a02ececc1276 [id=ad4f0c8adc524a019beaf37e153b33a7]) successfully deleted
I20240502 08:16:46.272480 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet 171406810543435e832a47ad5b3c80a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e5de793e74247dd9e73b7b1d58027d9) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:46.272711 24180 tablet_replica.cc:330] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:46.272817 24179 tablet_replica.cc:330] T 171406810543435e832a47ad5b3c80a4 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:46.272931 24180 raft_consensus.cc:2229] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:46.272903 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:46.273006 24179 raft_consensus.cc:2229] T 171406810543435e832a47ad5b3c80a4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.273142 24179 raft_consensus.cc:2258] T 171406810543435e832a47ad5b3c80a4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.273161 24180 pending_rounds.cc:62] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:16:46.273207 24180 pending_rounds.cc:68] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156454370983936 op_type: NO_OP noop_request { }
I20240502 08:16:46.273308 24180 raft_consensus.cc:2866] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:46.273375 24180 raft_consensus.cc:2258] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.273903 24179 ts_tablet_manager.cc:1875] T 171406810543435e832a47ad5b3c80a4 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.277532 24173 tablet_bootstrap.cc:492] T 4a57ed188315493198e95cefb6907ab6 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:46.277745 24173 ts_tablet_manager.cc:1367] T 4a57ed188315493198e95cefb6907ab6 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.302s user 0.024s sys 0.038s
I20240502 08:16:46.278645 24173 raft_consensus.cc:348] T 4a57ed188315493198e95cefb6907ab6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.278872 24173 raft_consensus.cc:374] T 4a57ed188315493198e95cefb6907ab6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.278934 24173 raft_consensus.cc:729] T 4a57ed188315493198e95cefb6907ab6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.279171 24173 consensus_queue.cc:260] T 4a57ed188315493198e95cefb6907ab6 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.279824 24173 ts_tablet_manager.cc:1398] T 4a57ed188315493198e95cefb6907ab6 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:46.281512 24179 ts_tablet_manager.cc:1888] T 171406810543435e832a47ad5b3c80a4 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:46.281713 24179 log.cc:1196] T 171406810543435e832a47ad5b3c80a4 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/171406810543435e832a47ad5b3c80a4
I20240502 08:16:46.282040 24179 ts_tablet_manager.cc:1909] T 171406810543435e832a47ad5b3c80a4 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:46.287323 24173 tablet_bootstrap.cc:492] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:46.288552 24180 ts_tablet_manager.cc:1875] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.290369 24173 tablet_bootstrap.cc:654] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.297461 24180 ts_tablet_manager.cc:1888] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:46.300159 24180 log.cc:1196] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/7c374f42f353497fa1d745164e7fbf61
I20240502 08:16:46.300221 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 171406810543435e832a47ad5b3c80a4 (table default.Table_a286e67e441e43a1b4738cd219dbea98 [id=b56e3c0686244ad6b49afdd589c5ed34]) successfully deleted
W20240502 08:16:46.300552 23597 catalog_manager.cc:4797] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 9c03a4353c4e46e3bcdf3a36aac04bf5 because the tablet was not found. No further retry: Not found: Tablet not found: 9c03a4353c4e46e3bcdf3a36aac04bf5
I20240502 08:16:46.300752 24180 ts_tablet_manager.cc:1909] T 7c374f42f353497fa1d745164e7fbf61 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:46.310822 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 7c374f42f353497fa1d745164e7fbf61 (table default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]) successfully deleted
W20240502 08:16:46.311890 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet a91e2b50321e4241847d97a558b419f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a91e2b50321e4241847d97a558b419f2 already in progress: creating tablet
I20240502 08:16:46.311946 23886 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fac7df16fc41d7a3297055a7258270 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:46.312201 24179 tablet_replica.cc:330] T b1fac7df16fc41d7a3297055a7258270 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:46.312227 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a57ed188315493198e95cefb6907ab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:46.312390 24179 raft_consensus.cc:2229] T b1fac7df16fc41d7a3297055a7258270 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.312463 24181 tablet_replica.cc:330] T 4a57ed188315493198e95cefb6907ab6 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:46.312582 24179 raft_consensus.cc:2258] T b1fac7df16fc41d7a3297055a7258270 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.312685 24181 raft_consensus.cc:2229] T 4a57ed188315493198e95cefb6907ab6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.312861 24181 raft_consensus.cc:2258] T 4a57ed188315493198e95cefb6907ab6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.313329 24179 ts_tablet_manager.cc:1875] T b1fac7df16fc41d7a3297055a7258270 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.313971 24181 ts_tablet_manager.cc:1875] T 4a57ed188315493198e95cefb6907ab6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.314149 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d28098dcbc4761a34ee65769d6fd7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:46.314608 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c374f42f353497fa1d745164e7fbf61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:46.314843 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 171406810543435e832a47ad5b3c80a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e5de793e74247dd9e73b7b1d58027d9) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:46.315461 23602 catalog_manager.cc:5484] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:46.319458 24179 ts_tablet_manager.cc:1888] T b1fac7df16fc41d7a3297055a7258270 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:46.319554 24179 log.cc:1196] T b1fac7df16fc41d7a3297055a7258270 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/b1fac7df16fc41d7a3297055a7258270
I20240502 08:16:46.319823 24179 ts_tablet_manager.cc:1909] T b1fac7df16fc41d7a3297055a7258270 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:46.320051 24181 ts_tablet_manager.cc:1888] T 4a57ed188315493198e95cefb6907ab6 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:46.320173 24181 log.cc:1196] T 4a57ed188315493198e95cefb6907ab6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/4a57ed188315493198e95cefb6907ab6
I20240502 08:16:46.320647 24181 ts_tablet_manager.cc:1909] T 4a57ed188315493198e95cefb6907ab6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:46.321308 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet b1fac7df16fc41d7a3297055a7258270 (table default.Table_bcd1014009b046a7ba72d1d757e69e26 [id=e6823667e67a470e870f4ed70fc58a85]) successfully deleted
I20240502 08:16:46.322276 24179 tablet_replica.cc:330] T c5d28098dcbc4761a34ee65769d6fd7c P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:46.322613 24180 tablet_replica.cc:330] T bcfacc302ff04178bfffc48111cd7a3f P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:46.322667 24179 raft_consensus.cc:2229] T c5d28098dcbc4761a34ee65769d6fd7c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.322805 24179 raft_consensus.cc:2258] T c5d28098dcbc4761a34ee65769d6fd7c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.323539 24179 ts_tablet_manager.cc:1875] T c5d28098dcbc4761a34ee65769d6fd7c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.323892 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 42e11d405a2e4e518a10bf383e9ae6c4
I20240502 08:16:46.325080 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 42e11d405a2e4e518a10bf383e9ae6c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:46.325505 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 42e11d405a2e4e518a10bf383e9ae6c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:46.325632 24180 raft_consensus.cc:2229] T bcfacc302ff04178bfffc48111cd7a3f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.325781 24180 raft_consensus.cc:2258] T bcfacc302ff04178bfffc48111cd7a3f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.326296 24180 ts_tablet_manager.cc:1875] T bcfacc302ff04178bfffc48111cd7a3f P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.327351 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 42e11d405a2e4e518a10bf383e9ae6c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:46.329033 24173 tablet_bootstrap.cc:492] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:46.329182 24173 ts_tablet_manager.cc:1367] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.001s
I20240502 08:16:46.329989 24173 raft_consensus.cc:348] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.330178 24173 raft_consensus.cc:374] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.330228 24173 raft_consensus.cc:729] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.330442 24173 consensus_queue.cc:260] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.335289 24179 ts_tablet_manager.cc:1888] T c5d28098dcbc4761a34ee65769d6fd7c P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:46.335395 24179 log.cc:1196] T c5d28098dcbc4761a34ee65769d6fd7c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/c5d28098dcbc4761a34ee65769d6fd7c
I20240502 08:16:46.335712 24179 ts_tablet_manager.cc:1909] T c5d28098dcbc4761a34ee65769d6fd7c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:46.336601 24179 tablet_replica.cc:330] T 7c374f42f353497fa1d745164e7fbf61 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:46.336681 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet c5d28098dcbc4761a34ee65769d6fd7c (table default.Table_81d96be059774b5bbaf8c8d21b2935ce [id=dd9c1fd15bf446fd87bd3647cb7f041d]) successfully deleted
I20240502 08:16:46.336772 24179 raft_consensus.cc:2229] T 7c374f42f353497fa1d745164e7fbf61 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.336899 24179 raft_consensus.cc:2258] T 7c374f42f353497fa1d745164e7fbf61 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.337684 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 4a57ed188315493198e95cefb6907ab6 (table default.Table_8c4dfc5c68e84b6ab815e9f333143678 [id=58ae4f5729f44a9c8be960be8a1a13a1]) successfully deleted
I20240502 08:16:46.338680 24173 ts_tablet_manager.cc:1398] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
W20240502 08:16:46.338753 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 42e11d405a2e4e518a10bf383e9ae6c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42e11d405a2e4e518a10bf383e9ae6c4 already in progress: creating tablet
I20240502 08:16:46.339295 24173 tablet_bootstrap.cc:492] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:46.340893 24173 tablet_bootstrap.cc:654] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.344350 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5fb99cde55ed4b33a109255d52e75384" } new_table_name: "default.Table_5a6711c7a8e74b05af6d0c8fb3cfa33f" modify_external_catalogs: true
I20240502 08:16:46.344401 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a57ed188315493198e95cefb6907ab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:46.344992 23597 catalog_manager.cc:4797] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 4a57ed188315493198e95cefb6907ab6 because the tablet was not found. No further retry: Not found: Tablet not found: 4a57ed188315493198e95cefb6907ab6
I20240502 08:16:46.346769 24180 ts_tablet_manager.cc:1888] T bcfacc302ff04178bfffc48111cd7a3f P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:46.347743 24173 tablet_bootstrap.cc:492] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:46.347954 24173 ts_tablet_manager.cc:1367] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 08:16:46.348737 24173 raft_consensus.cc:348] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.349004 24173 raft_consensus.cc:374] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.349107 24173 raft_consensus.cc:729] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.349371 24173 consensus_queue.cc:260] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.350700 24173 ts_tablet_manager.cc:1398] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:46.351011 24173 tablet_bootstrap.cc:492] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:46.351117 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 5e56af7b42cb481a8363d60f49ee3f61
I20240502 08:16:46.352612 24173 tablet_bootstrap.cc:654] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.353430 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e56af7b42cb481a8363d60f49ee3f61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:46.353266 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e56af7b42cb481a8363d60f49ee3f61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:46.354183 24180 log.cc:1196] T bcfacc302ff04178bfffc48111cd7a3f P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/bcfacc302ff04178bfffc48111cd7a3f
I20240502 08:16:46.354723 24180 ts_tablet_manager.cc:1909] T bcfacc302ff04178bfffc48111cd7a3f P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:46.355899 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet bcfacc302ff04178bfffc48111cd7a3f (table default.Table_d416b40fe3d24dafa8db5ee61d13acc0 [id=fa5100e581a7430ab8f0c19b2daae994]) successfully deleted
W20240502 08:16:46.356040 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 5e56af7b42cb481a8363d60f49ee3f61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e56af7b42cb481a8363d60f49ee3f61 already in progress: creating tablet
W20240502 08:16:46.356693 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 57df4bde56a44809ad43d563baeefecb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57df4bde56a44809ad43d563baeefecb already in progress: creating tablet
I20240502 08:16:46.357023 24180 tablet_replica.cc:330] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:46.357106 24179 ts_tablet_manager.cc:1875] T 7c374f42f353497fa1d745164e7fbf61 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.357333 24180 raft_consensus.cc:2229] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.357581 24180 raft_consensus.cc:2258] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.358793 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e56af7b42cb481a8363d60f49ee3f61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:46.361886 23893 tablet_service.cc:1467] Processing CreateTablet for tablet d5cf3e4cb0a9411da55bd24d9a6b60ea (DEFAULT_TABLE table=default.Table_4114c8f606d84b01963b9e68bb1571f0 [id=e42548a13eeb41ae9d069f660b69f417]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.362280 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5cf3e4cb0a9411da55bd24d9a6b60ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.358359 24174 tablet_bootstrap.cc:492] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:46.360872 23767 tablet_service.cc:1467] Processing CreateTablet for tablet e860d70d30c74112a5e4cdb447d752af (DEFAULT_TABLE table=default.Table_3d50b45ce5014fafae3e3a7dbe1d25ec [id=8ee644501bbd4b15874507f0adc6e070]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.363426 24174 ts_tablet_manager.cc:1367] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.321s user 0.033s sys 0.025s
I20240502 08:16:46.363602 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e860d70d30c74112a5e4cdb447d752af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.364251 24174 raft_consensus.cc:348] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.364445 24174 raft_consensus.cc:374] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.364500 24174 raft_consensus.cc:729] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.364701 24174 consensus_queue.cc:260] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.365157 24174 ts_tablet_manager.cc:1398] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:46.365314 24179 ts_tablet_manager.cc:1888] T 7c374f42f353497fa1d745164e7fbf61 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:46.365425 24174 tablet_bootstrap.cc:492] T 98a96292b28a4b0aad88fe0a96730a68 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:46.365442 24179 log.cc:1196] T 7c374f42f353497fa1d745164e7fbf61 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/7c374f42f353497fa1d745164e7fbf61
I20240502 08:16:46.365824 24179 ts_tablet_manager.cc:1909] T 7c374f42f353497fa1d745164e7fbf61 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:46.367341 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 7c374f42f353497fa1d745164e7fbf61 (table default.Table_2047aed3b97c4f4283bd8f6e80aeb92e [id=3315e636abd343eea0f081c9b3fbf4af]) successfully deleted
W20240502 08:16:46.367678 23597 catalog_manager.cc:4797] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 171406810543435e832a47ad5b3c80a4 because the tablet was not found. No further retry: Not found: Tablet not found: 171406810543435e832a47ad5b3c80a4
W20240502 08:16:46.368283 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 42e11d405a2e4e518a10bf383e9ae6c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42e11d405a2e4e518a10bf383e9ae6c4 already in progress: creating tablet
W20240502 08:16:46.368577 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 5e56af7b42cb481a8363d60f49ee3f61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e56af7b42cb481a8363d60f49ee3f61 already in progress: creating tablet
I20240502 08:16:46.370433 24174 tablet_bootstrap.cc:654] T 98a96292b28a4b0aad88fe0a96730a68 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.372936 24180 ts_tablet_manager.cc:1875] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.374262 24842 raft_consensus.cc:482] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.374419 24842 raft_consensus.cc:504] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.375061 24842 leader_election.cc:288] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:46.375696 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27093862fef54c5c85d810ebeff9819c" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:46.376273 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27093862fef54c5c85d810ebeff9819c" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:46.376807 23996 leader_election.cc:341] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:46.378077 23993 leader_election.cc:341] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.378176 23993 leader_election.cc:302] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:46.378912 24842 raft_consensus.cc:2726] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:46.381904 24039 tablet_service.cc:1467] Processing CreateTablet for tablet e860d70d30c74112a5e4cdb447d752af (DEFAULT_TABLE table=default.Table_3d50b45ce5014fafae3e3a7dbe1d25ec [id=8ee644501bbd4b15874507f0adc6e070]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.382521 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e860d70d30c74112a5e4cdb447d752af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.387122 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 77c49da1e0f0435ba241a6a66526e790 (table default.Table_8a9cdf1f53b0494694035cd549988bf0 [id=0de18f9c7e194e698b4f37603bfd4780]). Expected version 2 got 0
I20240502 08:16:46.388231 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fb665eaa632a40188077e3127c5a3e9c" } modify_external_catalogs: true
I20240502 08:16:46.388489 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fb665eaa632a40188077e3127c5a3e9c" } modify_external_catalogs: true
I20240502 08:16:46.381552 23900 tablet_service.cc:1467] Processing CreateTablet for tablet e860d70d30c74112a5e4cdb447d752af (DEFAULT_TABLE table=default.Table_3d50b45ce5014fafae3e3a7dbe1d25ec [id=8ee644501bbd4b15874507f0adc6e070]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.391602 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e860d70d30c74112a5e4cdb447d752af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.392582 24174 tablet_bootstrap.cc:492] T 98a96292b28a4b0aad88fe0a96730a68 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:46.392719 24174 ts_tablet_manager.cc:1367] T 98a96292b28a4b0aad88fe0a96730a68 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.004s
I20240502 08:16:46.393534 24174 raft_consensus.cc:348] T 98a96292b28a4b0aad88fe0a96730a68 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.393723 24174 raft_consensus.cc:374] T 98a96292b28a4b0aad88fe0a96730a68 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.393776 24174 raft_consensus.cc:729] T 98a96292b28a4b0aad88fe0a96730a68 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.394002 24174 consensus_queue.cc:260] T 98a96292b28a4b0aad88fe0a96730a68 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.394510 24174 ts_tablet_manager.cc:1398] T 98a96292b28a4b0aad88fe0a96730a68 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:46.395949 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet abdcdf7db3e34e77980d4bcc07bcd090
I20240502 08:16:46.396495 23609 catalog_manager.cc:6554] ReplaceTablet: tablet abdcdf7db3e34e77980d4bcc07bcd090 deleted and replaced by tablet 4bea89eccaec4d049c0ec3900ed600c2
I20240502 08:16:46.397794 23609 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 27093862fef54c5c85d810ebeff9819c (table default.Table_aded763332144f979b418931264d1eb5 [id=0035c98028aa49f7a7dcea576bb23351]). Expected version 1 got 0
I20240502 08:16:46.398520 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet 91c7167e29cb44b68f27f972652c6739 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:46.401225 24174 tablet_bootstrap.cc:492] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:46.404608 24174 tablet_bootstrap.cc:654] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:46.405180 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89a17261fbe0490fbe73ffe199ebb7be has no consensus state
I20240502 08:16:46.405560 24180 ts_tablet_manager.cc:1888] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:46.405673 24180 log.cc:1196] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/a3043c5ed98c401b82c6e967ca3f3613
I20240502 08:16:46.405959 24180 ts_tablet_manager.cc:1909] T a3043c5ed98c401b82c6e967ca3f3613 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
W20240502 08:16:46.406714 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 40fab016ed4f489aa3ccdeafcfbe1477 consensus state has no leader
I20240502 08:16:46.406787 24180 tablet_replica.cc:330] T 0882b7509a74475780eb700818aab398 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:46.406956 24180 raft_consensus.cc:2229] T 0882b7509a74475780eb700818aab398 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.407090 24180 raft_consensus.cc:2258] T 0882b7509a74475780eb700818aab398 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.407138 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_774c13246f3442b28645d6c755c67800"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:46.407585 24180 ts_tablet_manager.cc:1875] T 0882b7509a74475780eb700818aab398 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:46.407689 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_774c13246f3442b28645d6c755c67800 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:46.409065 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet a3043c5ed98c401b82c6e967ca3f3613 (table default.Table_3e0123590a92473dabaacc9c158859e7 [id=d1a317371a5b43f185eb6453861e112a]) successfully deleted
I20240502 08:16:46.410035 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5a39648854e8433eac74de3c8f04b5c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:46.410449 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a39648854e8433eac74de3c8f04b5c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:46.417691 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet abdcdf7db3e34e77980d4bcc07bcd090 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bea89eccaec4d049c0ec3900ed600c2) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:46.418303 24174 tablet_bootstrap.cc:492] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:46.418437 24174 ts_tablet_manager.cc:1367] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20240502 08:16:46.419168 24174 raft_consensus.cc:348] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.419333 24174 raft_consensus.cc:374] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.419386 24174 raft_consensus.cc:729] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.419576 24174 consensus_queue.cc:260] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:46.419778 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet abdcdf7db3e34e77980d4bcc07bcd090 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abdcdf7db3e34e77980d4bcc07bcd090 already in progress: creating tablet
I20240502 08:16:46.420046 24174 ts_tablet_manager.cc:1398] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:46.420306 24174 tablet_bootstrap.cc:492] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:46.422111 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 793d364d9dec429484660a86036f5fb9
I20240502 08:16:46.421535 24174 tablet_bootstrap.cc:654] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.423099 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 793d364d9dec429484660a86036f5fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:46 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:46.424160 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet 793d364d9dec429484660a86036f5fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:46 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:46.424947 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 793d364d9dec429484660a86036f5fb9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 793d364d9dec429484660a86036f5fb9 already in progress: creating tablet
W20240502 08:16:46.428495 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 793d364d9dec429484660a86036f5fb9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 793d364d9dec429484660a86036f5fb9 already in progress: creating tablet
I20240502 08:16:46.428699 24180 ts_tablet_manager.cc:1888] T 0882b7509a74475780eb700818aab398 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:46.428808 24180 log.cc:1196] T 0882b7509a74475780eb700818aab398 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/0882b7509a74475780eb700818aab398
I20240502 08:16:46.429090 24180 ts_tablet_manager.cc:1909] T 0882b7509a74475780eb700818aab398 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:46.429399 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 793d364d9dec429484660a86036f5fb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:46 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:46.430058 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet abdcdf7db3e34e77980d4bcc07bcd090 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bea89eccaec4d049c0ec3900ed600c2) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:46.430042 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 0882b7509a74475780eb700818aab398 (table default.Table_bc1e9d346aa846f8abca7504185f7cdb [id=8d27878c6c8f4234908a2c78cb8bbd29]) successfully deleted
I20240502 08:16:46.430354 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet abdcdf7db3e34e77980d4bcc07bcd090 with delete_type TABLET_DATA_DELETED (replaced by tablet 4bea89eccaec4d049c0ec3900ed600c2) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:46.431392 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet abdcdf7db3e34e77980d4bcc07bcd090 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abdcdf7db3e34e77980d4bcc07bcd090 already in progress: creating tablet
I20240502 08:16:46.432812 24025 tablet_service.cc:1467] Processing CreateTablet for tablet d5cf3e4cb0a9411da55bd24d9a6b60ea (DEFAULT_TABLE table=default.Table_4114c8f606d84b01963b9e68bb1571f0 [id=e42548a13eeb41ae9d069f660b69f417]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.433308 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5cf3e4cb0a9411da55bd24d9a6b60ea. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:46.441498 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e0eb928812a24d1cb82e00b549b36516 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0eb928812a24d1cb82e00b549b36516 already in progress: creating tablet
I20240502 08:16:46.442335 23767 tablet_service.cc:1467] Processing CreateTablet for tablet d5cf3e4cb0a9411da55bd24d9a6b60ea (DEFAULT_TABLE table=default.Table_4114c8f606d84b01963b9e68bb1571f0 [id=e42548a13eeb41ae9d069f660b69f417]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.442780 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5cf3e4cb0a9411da55bd24d9a6b60ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.444013 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 88acb3b0c2b7480e917ed68b7646b47d (DEFAULT_TABLE table=default.Table_20b409a694114049bf0d966535b376a8 [id=82ec88b54caf435bb5a8b2eba9a16e91]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.444824 24180 tablet_replica.cc:330] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:46.445519 24180 raft_consensus.cc:2229] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.447413 24057 raft_consensus.cc:1261] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:46.448239 24778 consensus_queue.cc:1035] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:46.450314 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88acb3b0c2b7480e917ed68b7646b47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.450973 23925 raft_consensus.cc:3037] T d1be2c29d3154d9189af3266191f0b3e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.454947 23925 raft_consensus.cc:1261] T d1be2c29d3154d9189af3266191f0b3e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:46.457378 24778 consensus_queue.cc:1035] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:46.462054 24180 raft_consensus.cc:2258] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.465178 24180 ts_tablet_manager.cc:1875] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.466257 24174 tablet_bootstrap.cc:492] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:46.472391 24174 ts_tablet_manager.cc:1367] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.003s
I20240502 08:16:46.474869 24778 raft_consensus.cc:482] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.475071 24778 raft_consensus.cc:504] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.476298 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "643ce929e74c499da835fbe9b63b8880" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:46.476861 23723 leader_election.cc:341] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.477506 24778 leader_election.cc:288] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:46.474105 24174 raft_consensus.cc:348] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:46.478008 23723 leader_election.cc:341] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 643ce929e74c499da835fbe9b63b8880
I20240502 08:16:46.478454 23723 leader_election.cc:302] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:46.478811 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 88acb3b0c2b7480e917ed68b7646b47d (DEFAULT_TABLE table=default.Table_20b409a694114049bf0d966535b376a8 [id=82ec88b54caf435bb5a8b2eba9a16e91]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.478889 24778 raft_consensus.cc:2726] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:46.479097 24174 raft_consensus.cc:374] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.479171 24174 raft_consensus.cc:729] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.479540 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88acb3b0c2b7480e917ed68b7646b47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.479760 24174 consensus_queue.cc:260] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.484732 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 88acb3b0c2b7480e917ed68b7646b47d (DEFAULT_TABLE table=default.Table_20b409a694114049bf0d966535b376a8 [id=82ec88b54caf435bb5a8b2eba9a16e91]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.484853 24174 ts_tablet_manager.cc:1398] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 08:16:46.488358 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 71e570b62d8b4b23a213567e79f15e1d (DEFAULT_TABLE table=default.Table_134e3d6f761d4656acd584556c172e39 [id=6b975ab157614742bee11acde3f1f23d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.488857 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e570b62d8b4b23a213567e79f15e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.485128 24174 tablet_bootstrap.cc:492] T d9fcc7accb1f45ad9cf55b8f64d21b91 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:46.491837 24180 ts_tablet_manager.cc:1888] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:46.491943 24180 log.cc:1196] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/71aa916247c449fea718b35869a6df20
I20240502 08:16:46.492226 24180 ts_tablet_manager.cc:1909] T 71aa916247c449fea718b35869a6df20 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:46.493173 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 71aa916247c449fea718b35869a6df20 (table default.Table_9a9569245562493dafbeb639c6492826 [id=5cee878b3a31426c879a1bf95045bb09]) successfully deleted
I20240502 08:16:46.485340 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88acb3b0c2b7480e917ed68b7646b47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.495069 24174 tablet_bootstrap.cc:654] T d9fcc7accb1f45ad9cf55b8f64d21b91 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.496022 24778 raft_consensus.cc:482] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.496142 24778 raft_consensus.cc:504] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.496722 24778 leader_election.cc:288] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:46.497669 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1736b496354e2dadd844242d3867f8" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:46.498320 23723 leader_election.cc:341] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:46.505053 23723 leader_election.cc:341] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 4f1736b496354e2dadd844242d3867f8
I20240502 08:16:46.505175 23723 leader_election.cc:302] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:46.505709 24778 raft_consensus.cc:2726] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:46.508358 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 71e570b62d8b4b23a213567e79f15e1d (DEFAULT_TABLE table=default.Table_134e3d6f761d4656acd584556c172e39 [id=6b975ab157614742bee11acde3f1f23d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.508858 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e570b62d8b4b23a213567e79f15e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.512019 24919 tablet.cc:1722] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:46.515758 24174 tablet_bootstrap.cc:492] T d9fcc7accb1f45ad9cf55b8f64d21b91 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:46.515908 24174 ts_tablet_manager.cc:1367] T d9fcc7accb1f45ad9cf55b8f64d21b91 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20240502 08:16:46.516644 24174 raft_consensus.cc:348] T d9fcc7accb1f45ad9cf55b8f64d21b91 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.516841 24174 raft_consensus.cc:374] T d9fcc7accb1f45ad9cf55b8f64d21b91 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.516891 24174 raft_consensus.cc:729] T d9fcc7accb1f45ad9cf55b8f64d21b91 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.517083 24174 consensus_queue.cc:260] T d9fcc7accb1f45ad9cf55b8f64d21b91 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.518095 24920 tablet.cc:1722] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240502 08:16:46.521045 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e712916274144e5e86661561646c3521 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e712916274144e5e86661561646c3521 already in progress: creating tablet
I20240502 08:16:46.517546 24174 ts_tablet_manager.cc:1398] T d9fcc7accb1f45ad9cf55b8f64d21b91 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:46.522934 24174 tablet_bootstrap.cc:492] T b8b1fa17dad143f0a7f1751e651eb81a P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:46.523928 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 71e570b62d8b4b23a213567e79f15e1d (DEFAULT_TABLE table=default.Table_134e3d6f761d4656acd584556c172e39 [id=6b975ab157614742bee11acde3f1f23d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.524370 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e570b62d8b4b23a213567e79f15e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.524713 23767 tablet_service.cc:1467] Processing CreateTablet for tablet cb978e43ec794352bc0ed7f7aad4391b (DEFAULT_TABLE table=default.Table_5d88e7a8816a45ccb7a49f559050dad2 [id=76ef21ef044a4e8f91e7ef80e0316457]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.526669 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb978e43ec794352bc0ed7f7aad4391b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.527580 24174 tablet_bootstrap.cc:654] T b8b1fa17dad143f0a7f1751e651eb81a P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.521449 24180 tablet_replica.cc:330] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:46.534615 24891 raft_consensus.cc:482] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.534871 24891 raft_consensus.cc:504] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.536022 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e686db13bb7541c69ae88f0fb7e27061" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:46.536551 23723 leader_election.cc:341] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.536926 24891 leader_election.cc:288] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:46.537503 24180 raft_consensus.cc:2229] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240502 08:16:46.537846 23723 leader_election.cc:341] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: e686db13bb7541c69ae88f0fb7e27061
I20240502 08:16:46.540326 23896 tablet_service.cc:1467] Processing CreateTablet for tablet cb978e43ec794352bc0ed7f7aad4391b (DEFAULT_TABLE table=default.Table_5d88e7a8816a45ccb7a49f559050dad2 [id=76ef21ef044a4e8f91e7ef80e0316457]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.540441 23723 leader_election.cc:302] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:46.540728 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb978e43ec794352bc0ed7f7aad4391b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.540884 24891 raft_consensus.cc:2726] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:46.541198 24180 raft_consensus.cc:2258] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.541750 24180 ts_tablet_manager.cc:1875] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.542253 24174 tablet_bootstrap.cc:492] T b8b1fa17dad143f0a7f1751e651eb81a P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:46.543174 24174 ts_tablet_manager.cc:1367] T b8b1fa17dad143f0a7f1751e651eb81a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.004s
I20240502 08:16:46.543823 24174 raft_consensus.cc:348] T b8b1fa17dad143f0a7f1751e651eb81a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.543991 24174 raft_consensus.cc:374] T b8b1fa17dad143f0a7f1751e651eb81a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.544034 24174 raft_consensus.cc:729] T b8b1fa17dad143f0a7f1751e651eb81a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.544104 24027 tablet_service.cc:1467] Processing CreateTablet for tablet cb978e43ec794352bc0ed7f7aad4391b (DEFAULT_TABLE table=default.Table_5d88e7a8816a45ccb7a49f559050dad2 [id=76ef21ef044a4e8f91e7ef80e0316457]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.544198 24174 consensus_queue.cc:260] T b8b1fa17dad143f0a7f1751e651eb81a P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.544598 24174 ts_tablet_manager.cc:1398] T b8b1fa17dad143f0a7f1751e651eb81a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:16:46.544617 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb978e43ec794352bc0ed7f7aad4391b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.548441 24174 tablet_bootstrap.cc:492] T 1e5de793e74247dd9e73b7b1d58027d9 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:46.549904 24174 tablet_bootstrap.cc:654] T 1e5de793e74247dd9e73b7b1d58027d9 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.555608 23576 master_runner.cc:186] Error getting master registration for 127.22.225.125:36189: Network error: Client connection negotiation failed: client connection to 127.22.225.125:36189: connect: Connection refused (error 111),
I20240502 08:16:46.555819 23576 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240502 08:16:46.558063 24180 ts_tablet_manager.cc:1888] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:46.558166 24180 log.cc:1196] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/2bfb532c9d4c41afb7acaa4831841306
I20240502 08:16:46.558415 24180 ts_tablet_manager.cc:1909] T 2bfb532c9d4c41afb7acaa4831841306 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:46.559393 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 2bfb532c9d4c41afb7acaa4831841306 (table default.Table_f48b4cdee51644e0b3c904d772f01404 [id=35259dda872249a7bdddd93783103db8]) successfully deleted
I20240502 08:16:46.559738 24180 tablet_replica.cc:330] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:46.559911 24180 raft_consensus.cc:2229] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.560055 24180 raft_consensus.cc:2258] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.560505 24180 ts_tablet_manager.cc:1875] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:46.561643 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93883f7676c148d6a9cf22690b737e80 consensus state has no leader
I20240502 08:16:46.561894 23603 master_service.cc:946] ReplaceTablet: received request to replace tablet b844458ae73f413cbf2113027a5af359 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:46.563449 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 8ea07009009542b59d42b9e443990c72
I20240502 08:16:46.564273 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea07009009542b59d42b9e443990c72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:46 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:46.564507 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea07009009542b59d42b9e443990c72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:46 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:46.564905 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet 25c8a6aaabe2413d8ce88f54ce45a4c4 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:46.565429 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea07009009542b59d42b9e443990c72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:46 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:46.565886 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 8ea07009009542b59d42b9e443990c72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ea07009009542b59d42b9e443990c72 already in progress: creating tablet
I20240502 08:16:46.566079 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet 93883f7676c148d6a9cf22690b737e80 from {username='slave'} at 127.0.0.1:36668
W20240502 08:16:46.568061 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 8ea07009009542b59d42b9e443990c72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ea07009009542b59d42b9e443990c72 already in progress: creating tablet
I20240502 08:16:46.570443 24180 ts_tablet_manager.cc:1888] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:46.570600 24180 log.cc:1196] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/643ce929e74c499da835fbe9b63b8880
I20240502 08:16:46.570930 24180 ts_tablet_manager.cc:1909] T 643ce929e74c499da835fbe9b63b8880 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:46.571892 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 643ce929e74c499da835fbe9b63b8880 (table default.Table_a0ebd3d8bd13425ea80ffd0c428e20da [id=1c9043e3bf434bae83ea36aaef230b6f]) successfully deleted
I20240502 08:16:46.574363 24180 tablet_replica.cc:330] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:46.576421 24180 raft_consensus.cc:2229] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.576632 24180 raft_consensus.cc:2258] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.577742 24173 tablet_bootstrap.cc:492] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:46.577875 24173 ts_tablet_manager.cc:1367] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.227s user 0.023s sys 0.008s
I20240502 08:16:46.578505 24173 raft_consensus.cc:348] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.578670 24173 raft_consensus.cc:374] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.578708 24173 raft_consensus.cc:729] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.578886 24173 consensus_queue.cc:260] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.579363 24173 ts_tablet_manager.cc:1398] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:16:46.579641 24173 tablet_bootstrap.cc:492] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:46.579826 24175 tablet_bootstrap.cc:492] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:46.579932 24175 ts_tablet_manager.cc:1367] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.378s user 0.095s sys 0.111s
I20240502 08:16:46.580583 24175 raft_consensus.cc:348] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.580742 24175 raft_consensus.cc:374] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.580791 24175 raft_consensus.cc:729] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.581312 24173 tablet_bootstrap.cc:654] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.581740 24175 consensus_queue.cc:260] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.583133 24175 ts_tablet_manager.cc:1398] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:16:46.583452 24175 tablet_bootstrap.cc:492] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:46.585263 24175 tablet_bootstrap.cc:654] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.585793 24180 ts_tablet_manager.cc:1875] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.592648 24175 tablet_bootstrap.cc:492] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:46.592840 24175 ts_tablet_manager.cc:1367] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 08:16:46.594291 24175 raft_consensus.cc:348] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.594559 24175 raft_consensus.cc:374] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.594655 24175 raft_consensus.cc:729] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.594945 24175 consensus_queue.cc:260] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.595616 24175 ts_tablet_manager.cc:1398] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:16:46.595924 24175 tablet_bootstrap.cc:492] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:46.594637 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 571886fdbc2c43e19c790b2b7830ec27 (DEFAULT_TABLE table=default.Table_41a2401b3f4c4748b1d3eb964d8053d5 [id=959f40776ede4f2e8dfa0b1b849f3f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.596624 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 571886fdbc2c43e19c790b2b7830ec27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.597370 24175 tablet_bootstrap.cc:654] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.600133 24774 raft_consensus.cc:482] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.600313 24774 raft_consensus.cc:504] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.601035 24774 leader_election.cc:288] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:46.602294 24056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27093862fef54c5c85d810ebeff9819c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:46.602496 24056 raft_consensus.cc:2454] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:46.602929 23723 leader_election.cc:302] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:46.604884 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27093862fef54c5c85d810ebeff9819c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:46.607657 24778 raft_consensus.cc:2781] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:46.607743 24778 raft_consensus.cc:482] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:46.607815 24778 raft_consensus.cc:3037] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.610136 24778 raft_consensus.cc:504] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.610683 24778 leader_election.cc:288] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:46.611382 24056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27093862fef54c5c85d810ebeff9819c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:46.611642 24056 raft_consensus.cc:3037] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.613515 24173 tablet_bootstrap.cc:492] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:46.613740 24173 ts_tablet_manager.cc:1367] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.003s
I20240502 08:16:46.614176 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27093862fef54c5c85d810ebeff9819c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:46.614363 24173 raft_consensus.cc:348] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.614509 24173 raft_consensus.cc:374] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.614552 24173 raft_consensus.cc:729] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.614964 24056 raft_consensus.cc:2454] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:46.614894 24173 consensus_queue.cc:260] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:46.615172 23723 leader_election.cc:341] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.616246 23723 leader_election.cc:302] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:46.616763 24774 raft_consensus.cc:2781] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:46.616874 24774 raft_consensus.cc:686] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:46.617082 24774 consensus_queue.cc:237] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.617265 24173 ts_tablet_manager.cc:1398] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:16:46.617532 24173 tablet_bootstrap.cc:492] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:46.619573 24180 ts_tablet_manager.cc:1888] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:46.619678 24180 log.cc:1196] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/e686db13bb7541c69ae88f0fb7e27061
I20240502 08:16:46.619966 24180 ts_tablet_manager.cc:1909] T e686db13bb7541c69ae88f0fb7e27061 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:46.620600 24173 tablet_bootstrap.cc:654] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:46.621069 23723 leader_election.cc:341] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.622282 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet e686db13bb7541c69ae88f0fb7e27061 (table default.Table_178527d31d7044fc98fa41886d651049 [id=6dc275aed42b42118a70099d3eae3c21]) successfully deleted
I20240502 08:16:46.622982 24180 tablet_replica.cc:330] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:46.623136 24180 raft_consensus.cc:2229] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.623270 24180 raft_consensus.cc:2258] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.623744 24180 ts_tablet_manager.cc:1875] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.623754 24173 tablet_bootstrap.cc:492] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:46.623934 24173 ts_tablet_manager.cc:1367] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:16:46.624943 24173 raft_consensus.cc:348] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.625789 24173 raft_consensus.cc:374] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.625861 24173 raft_consensus.cc:729] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.626087 24173 consensus_queue.cc:260] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.627236 24173 ts_tablet_manager.cc:1398] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:16:46.627517 24173 tablet_bootstrap.cc:492] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:46.631562 24173 tablet_bootstrap.cc:654] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.632937 24180 ts_tablet_manager.cc:1888] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:46.636629 24173 tablet_bootstrap.cc:492] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:46.636816 24173 ts_tablet_manager.cc:1367] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240502 08:16:46.633049 24180 log.cc:1196] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/4f1736b496354e2dadd844242d3867f8
I20240502 08:16:46.637171 24180 ts_tablet_manager.cc:1909] T 4f1736b496354e2dadd844242d3867f8 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:46.637718 24173 raft_consensus.cc:348] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.637960 24173 raft_consensus.cc:374] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.638024 24173 raft_consensus.cc:729] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.638262 24173 consensus_queue.cc:260] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.640547 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 571886fdbc2c43e19c790b2b7830ec27 (DEFAULT_TABLE table=default.Table_41a2401b3f4c4748b1d3eb964d8053d5 [id=959f40776ede4f2e8dfa0b1b849f3f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.640918 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 571886fdbc2c43e19c790b2b7830ec27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.641665 24173 ts_tablet_manager.cc:1398] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:16:46.642702 24173 tablet_bootstrap.cc:492] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:46.645907 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 7a23c06cb3704a7b83143a02efe2a8b3 (DEFAULT_TABLE table=default.Table_c2a25d0d51cd40769e5045ff80c9c3fc [id=a1004106109d47e7a1f9c6337480d3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.646131 24173 tablet_bootstrap.cc:654] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.646303 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a23c06cb3704a7b83143a02efe2a8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.649854 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 571886fdbc2c43e19c790b2b7830ec27 (DEFAULT_TABLE table=default.Table_41a2401b3f4c4748b1d3eb964d8053d5 [id=959f40776ede4f2e8dfa0b1b849f3f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.650159 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 571886fdbc2c43e19c790b2b7830ec27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.652026 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 7a23c06cb3704a7b83143a02efe2a8b3 (DEFAULT_TABLE table=default.Table_c2a25d0d51cd40769e5045ff80c9c3fc [id=a1004106109d47e7a1f9c6337480d3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.652364 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a23c06cb3704a7b83143a02efe2a8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.658366 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 7a23c06cb3704a7b83143a02efe2a8b3 (DEFAULT_TABLE table=default.Table_c2a25d0d51cd40769e5045ff80c9c3fc [id=a1004106109d47e7a1f9c6337480d3c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.658723 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a23c06cb3704a7b83143a02efe2a8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.660451 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 4f1736b496354e2dadd844242d3867f8 (table default.Table_02f448a8a9844def8ecda8716fdc352e [id=79a03d22b7e34f76bb3242c640129a13]) successfully deleted
W20240502 08:16:46.661149 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 42e11d405a2e4e518a10bf383e9ae6c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42e11d405a2e4e518a10bf383e9ae6c4 already in progress: creating tablet
W20240502 08:16:46.662907 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 5e56af7b42cb481a8363d60f49ee3f61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e56af7b42cb481a8363d60f49ee3f61 already in progress: creating tablet
W20240502 08:16:46.663584 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 793d364d9dec429484660a86036f5fb9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 793d364d9dec429484660a86036f5fb9 already in progress: creating tablet
W20240502 08:16:46.664355 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet abdcdf7db3e34e77980d4bcc07bcd090 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abdcdf7db3e34e77980d4bcc07bcd090 already in progress: creating tablet
W20240502 08:16:46.665104 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 8ea07009009542b59d42b9e443990c72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ea07009009542b59d42b9e443990c72 already in progress: creating tablet
I20240502 08:16:46.668227 24173 tablet_bootstrap.cc:492] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:46.668727 24173 ts_tablet_manager.cc:1367] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.005s
I20240502 08:16:46.669729 24173 raft_consensus.cc:348] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.670861 24173 raft_consensus.cc:374] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.671032 24173 raft_consensus.cc:729] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.671392 24173 consensus_queue.cc:260] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.673874 24173 ts_tablet_manager.cc:1398] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:16:46.674343 24173 tablet_bootstrap.cc:492] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:46.677281 24173 tablet_bootstrap.cc:654] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.677976 24928 tablet.cc:1722] T d1be2c29d3154d9189af3266191f0b3e P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240502 08:16:46.693146 23595 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 045b0882a40f4eaba94935d9bc90da0c (table default.Table_7c20ea5847184b48b0bd45fd5751b992 [id=abb72d2b04794e4aab7acee7a3a0925b], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 045b0882a40f4eaba94935d9bc90da0c consensus state has no leader
I20240502 08:16:46.696198 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 49acc2af1a064dd2b0b60120beed69ba (DEFAULT_TABLE table=default.Table_469c024c43d64e2fb98eff5ff16e988b [id=fd6a3828cb43456fbdf93a43c42b48e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.696724 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49acc2af1a064dd2b0b60120beed69ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.701664 24774 raft_consensus.cc:482] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.701828 24774 raft_consensus.cc:504] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.702672 24774 leader_election.cc:288] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:46.704007 24056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b291bdb81a4efbac7f6691e5577bbe" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:46.704294 24056 raft_consensus.cc:2454] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:46.704794 23723 leader_election.cc:302] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:46.705473 24774 raft_consensus.cc:2781] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:46.705585 24774 raft_consensus.cc:482] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:46.705675 24774 raft_consensus.cc:3037] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.709486 24774 raft_consensus.cc:504] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.717725 24774 leader_election.cc:288] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:46.717860 24056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b291bdb81a4efbac7f6691e5577bbe" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:46.718123 24056 raft_consensus.cc:3037] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.718997 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 49acc2af1a064dd2b0b60120beed69ba (DEFAULT_TABLE table=default.Table_469c024c43d64e2fb98eff5ff16e988b [id=fd6a3828cb43456fbdf93a43c42b48e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.720983 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49acc2af1a064dd2b0b60120beed69ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.721586 24056 raft_consensus.cc:2454] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:46.722208 23723 leader_election.cc:302] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:46.722537 24774 raft_consensus.cc:2781] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:46.724225 24774 raft_consensus.cc:686] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:46.724552 24774 consensus_queue.cc:237] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.712715 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b291bdb81a4efbac7f6691e5577bbe" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:46.714334 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b291bdb81a4efbac7f6691e5577bbe" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:46.733974 23723 leader_election.cc:341] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:46.734686 23723 leader_election.cc:341] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.718497 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 49acc2af1a064dd2b0b60120beed69ba (DEFAULT_TABLE table=default.Table_469c024c43d64e2fb98eff5ff16e988b [id=fd6a3828cb43456fbdf93a43c42b48e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.735836 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49acc2af1a064dd2b0b60120beed69ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.743002 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 91c7167e29cb44b68f27f972652c6739
I20240502 08:16:46.743574 23609 catalog_manager.cc:6554] ReplaceTablet: tablet 91c7167e29cb44b68f27f972652c6739 deleted and replaced by tablet ee3d91d55ffd49ceb20d2c33408f5a5a
I20240502 08:16:46.747975 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c7167e29cb44b68f27f972652c6739 with delete_type TABLET_DATA_DELETED (replaced by tablet ee3d91d55ffd49ceb20d2c33408f5a5a) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:46.749022 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 91c7167e29cb44b68f27f972652c6739 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91c7167e29cb44b68f27f972652c6739 already in progress: creating tablet
I20240502 08:16:46.749079 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c7167e29cb44b68f27f972652c6739 with delete_type TABLET_DATA_DELETED (replaced by tablet ee3d91d55ffd49ceb20d2c33408f5a5a) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:46.749217 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2c882aad1a40484297ed2b6af34de3c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:46.749775 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 91c7167e29cb44b68f27f972652c6739 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91c7167e29cb44b68f27f972652c6739 already in progress: creating tablet
W20240502 08:16:46.750075 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c882aad1a40484297ed2b6af34de3c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:46.750157 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c7167e29cb44b68f27f972652c6739 with delete_type TABLET_DATA_DELETED (replaced by tablet ee3d91d55ffd49ceb20d2c33408f5a5a) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:46.750967 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 91c7167e29cb44b68f27f972652c6739 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91c7167e29cb44b68f27f972652c6739 already in progress: creating tablet
I20240502 08:16:46.751714 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7b367902a5d84aa5ac4aa4a0b97b48f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:46.752194 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b367902a5d84aa5ac4aa4a0b97b48f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:46.752468 23605 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet d1be2c29d3154d9189af3266191f0b3e (table default.Table_db86cd2cf86e48ffbc6a513b5c2bd7ab [id=6103f133bcdd4f768b92b37245bf5356]). Expected version 1 got 0
I20240502 08:16:46.753448 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_eb2c2e999a154f41bfe1d08fbf5502d5" } modify_external_catalogs: true
I20240502 08:16:46.756021 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_13623be380874b4d8b231ac4ed815b11" } new_table_name: "default.Table_36102dd8079049bcb0c8f32890c742fb" modify_external_catalogs: true
I20240502 08:16:46.757803 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_eb2c2e999a154f41bfe1d08fbf5502d5" } modify_external_catalogs: true
I20240502 08:16:46.763082 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e20e3a08238f428fa6c14fa84e67444a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:46.763746 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e20e3a08238f428fa6c14fa84e67444a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:46.765964 23608 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet d1be2c29d3154d9189af3266191f0b3e (table default.Table_db86cd2cf86e48ffbc6a513b5c2bd7ab [id=6103f133bcdd4f768b92b37245bf5356]). Expected version 1 got 0
I20240502 08:16:46.770571 24842 raft_consensus.cc:482] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.770747 24842 raft_consensus.cc:504] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.771546 24842 leader_election.cc:288] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:46.772878 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c49da1e0f0435ba241a6a66526e790" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:46.773649 23993 leader_election.cc:341] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.774026 23609 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 27093862fef54c5c85d810ebeff9819c (table default.Table_aded763332144f979b418931264d1eb5 [id=0035c98028aa49f7a7dcea576bb23351]). Expected version 1 got 0
I20240502 08:16:46.774593 23609 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet f4b291bdb81a4efbac7f6691e5577bbe (table default.Table_2d6b281d29d14441ac6abe9705633102 [id=73559755c4594bcfa591d054964154ae]). Expected version 2 got 0
I20240502 08:16:46.785660 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c49da1e0f0435ba241a6a66526e790" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:46.786549 23996 leader_election.cc:341] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.786710 23996 leader_election.cc:302] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:46.787468 24842 raft_consensus.cc:2726] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:46.792186 23594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.225.125:36189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:16:46.793509 23594 consensus_peers.cc:485] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 -> Peer 84f3c07a31414c4f8fb61c44dc9f16e1 (127.22.225.125:36189): Couldn't send request to peer 84f3c07a31414c4f8fb61c44dc9f16e1. Status: Network error: Client connection negotiation failed: client connection to 127.22.225.125:36189: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:16:46.795339 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet b844458ae73f413cbf2113027a5af359
I20240502 08:16:46.795939 23603 catalog_manager.cc:6554] ReplaceTablet: tablet b844458ae73f413cbf2113027a5af359 deleted and replaced by tablet 1638e78b6f544209bd8d6da6556fbce4
I20240502 08:16:46.796458 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet b844458ae73f413cbf2113027a5af359 with delete_type TABLET_DATA_DELETED (replaced by tablet 1638e78b6f544209bd8d6da6556fbce4) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:46.796810 24180 tablet_replica.cc:330] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:46.797040 24180 raft_consensus.cc:2229] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.797268 24180 raft_consensus.cc:2258] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.797663 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_74a4ec7454a14f74ad2c1dc97fab2080"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:46.797943 24180 ts_tablet_manager.cc:1875] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:46.798419 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74a4ec7454a14f74ad2c1dc97fab2080 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:46.799937 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_19bc9c458bb643f7ae59fcd36e6d3b27" } new_table_name: "default.Table_f7042786757e4da189a33bb4c3190322" modify_external_catalogs: true
I20240502 08:16:46.801759 24790 raft_consensus.cc:482] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.801954 24790 raft_consensus.cc:504] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:46.802156 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90d44be09f284f3d9042038c92f3b28d consensus state has no leader
I20240502 08:16:46.802547 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_843f461368e94ea9b6b9244506423221"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:46.802677 24790 leader_election.cc:288] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:46.803740 23861 leader_election.cc:341] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 0882b7509a74475780eb700818aab398
W20240502 08:16:46.803822 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_843f461368e94ea9b6b9244506423221 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:46.804551 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 93883f7676c148d6a9cf22690b737e80
I20240502 08:16:46.805155 23602 catalog_manager.cc:6554] ReplaceTablet: tablet 93883f7676c148d6a9cf22690b737e80 deleted and replaced by tablet 462f77a57ebf45948cbd3dc20c271650
I20240502 08:16:46.805617 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 93883f7676c148d6a9cf22690b737e80 with delete_type TABLET_DATA_DELETED (replaced by tablet 462f77a57ebf45948cbd3dc20c271650) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:46.806689 24778 raft_consensus.cc:482] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.806840 24778 raft_consensus.cc:504] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.807633 24778 leader_election.cc:288] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:46.808789 24056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b811b81ddf2b4dc9b5e5f80331bff294" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:46.809046 24056 raft_consensus.cc:2454] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
W20240502 08:16:46.808598 23858 leader_election.cc:341] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 0882b7509a74475780eb700818aab398
I20240502 08:16:46.809341 23858 leader_election.cc:302] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:46.809672 24790 raft_consensus.cc:2726] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:46.810092 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b811b81ddf2b4dc9b5e5f80331bff294" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:46.811060 23723 leader_election.cc:302] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:46.805483 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1f2e6b876bcf4bc39b3935a9a85e1219" } modify_external_catalogs: true
I20240502 08:16:46.811393 24778 raft_consensus.cc:2781] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:46.811493 24778 raft_consensus.cc:482] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:46.811568 24778 raft_consensus.cc:3037] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.811520 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1f2e6b876bcf4bc39b3935a9a85e1219" } modify_external_catalogs: true
W20240502 08:16:46.813277 23723 leader_election.cc:341] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.814868 24778 raft_consensus.cc:504] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.815850 24778 leader_election.cc:288] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:46.816982 24056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b811b81ddf2b4dc9b5e5f80331bff294" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:46.817071 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b811b81ddf2b4dc9b5e5f80331bff294" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:46.817198 24056 raft_consensus.cc:3037] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:46.817631 23723 leader_election.cc:341] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.820358 24056 raft_consensus.cc:2454] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:46.821153 23723 leader_election.cc:302] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:46.821743 24778 raft_consensus.cc:2781] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:46.822362 24180 ts_tablet_manager.cc:1888] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:16:46.822480 24180 log.cc:1196] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/b844458ae73f413cbf2113027a5af359
I20240502 08:16:46.822845 24778 raft_consensus.cc:686] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:46.823355 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 93883f7676c148d6a9cf22690b737e80 with delete_type TABLET_DATA_DELETED (replaced by tablet 462f77a57ebf45948cbd3dc20c271650) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:46.823945 24778 consensus_queue.cc:237] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:46.824667 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 93883f7676c148d6a9cf22690b737e80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93883f7676c148d6a9cf22690b737e80 already in progress: creating tablet
I20240502 08:16:46.825189 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 93883f7676c148d6a9cf22690b737e80 with delete_type TABLET_DATA_DELETED (replaced by tablet 462f77a57ebf45948cbd3dc20c271650) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:46.826136 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 93883f7676c148d6a9cf22690b737e80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93883f7676c148d6a9cf22690b737e80 already in progress: creating tablet
I20240502 08:16:46.823148 24180 ts_tablet_manager.cc:1909] T b844458ae73f413cbf2113027a5af359 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:46.828472 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet b844458ae73f413cbf2113027a5af359 (table default.Table_a86b6b9ac2d8498c97341dc60ca5ee34 [id=fd93259a378b4ea582a6661a59104abb]) successfully deleted
I20240502 08:16:46.828531 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet b844458ae73f413cbf2113027a5af359 with delete_type TABLET_DATA_DELETED (replaced by tablet 1638e78b6f544209bd8d6da6556fbce4) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:46.828794 24181 tablet_replica.cc:330] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:46.829028 24181 raft_consensus.cc:2229] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
W20240502 08:16:46.829015 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 93883f7676c148d6a9cf22690b737e80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93883f7676c148d6a9cf22690b737e80 already in progress: creating tablet
I20240502 08:16:46.829392 24181 raft_consensus.cc:2258] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.830124 24181 ts_tablet_manager.cc:1875] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.838289 24181 ts_tablet_manager.cc:1888] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:16:46.838447 24181 log.cc:1196] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/b844458ae73f413cbf2113027a5af359
I20240502 08:16:46.838809 24181 ts_tablet_manager.cc:1909] T b844458ae73f413cbf2113027a5af359 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:46.840131 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet b844458ae73f413cbf2113027a5af359 (table default.Table_a86b6b9ac2d8498c97341dc60ca5ee34 [id=fd93259a378b4ea582a6661a59104abb]) successfully deleted
I20240502 08:16:46.840178 23893 tablet_service.cc:1467] Processing CreateTablet for tablet e37d58fbcf6b476fb8c4558b3763c94f (DEFAULT_TABLE table=default.Table_4ee153a309b04ab68b03ab96694a1952 [id=719bc0cac41c4ce6bfa4cf06aca03da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.840883 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e37d58fbcf6b476fb8c4558b3763c94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.847218 24175 tablet_bootstrap.cc:492] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:46.847554 24175 ts_tablet_manager.cc:1367] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.252s user 0.036s sys 0.065s
I20240502 08:16:46.848788 24175 raft_consensus.cc:348] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.849184 24175 raft_consensus.cc:374] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.849298 24175 raft_consensus.cc:729] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.849748 24175 consensus_queue.cc:260] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.851222 23761 tablet_service.cc:1467] Processing CreateTablet for tablet e37d58fbcf6b476fb8c4558b3763c94f (DEFAULT_TABLE table=default.Table_4ee153a309b04ab68b03ab96694a1952 [id=719bc0cac41c4ce6bfa4cf06aca03da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.851449 24175 ts_tablet_manager.cc:1398] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:16:46.851742 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e37d58fbcf6b476fb8c4558b3763c94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.851804 24175 tablet_bootstrap.cc:492] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:46.853988 24175 tablet_bootstrap.cc:654] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.863221 24175 tablet_bootstrap.cc:492] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:46.863369 24175 ts_tablet_manager.cc:1367] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240502 08:16:46.864157 24175 raft_consensus.cc:348] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.864351 24175 raft_consensus.cc:374] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.864409 24175 raft_consensus.cc:729] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.864605 24175 consensus_queue.cc:260] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.864933 24027 tablet_service.cc:1467] Processing CreateTablet for tablet e37d58fbcf6b476fb8c4558b3763c94f (DEFAULT_TABLE table=default.Table_4ee153a309b04ab68b03ab96694a1952 [id=719bc0cac41c4ce6bfa4cf06aca03da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.865106 24175 ts_tablet_manager.cc:1398] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:16:46.865374 24175 tablet_bootstrap.cc:492] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:46.865897 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e37d58fbcf6b476fb8c4558b3763c94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.866883 24175 tablet_bootstrap.cc:654] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.870160 24175 tablet_bootstrap.cc:492] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:46.870371 24175 ts_tablet_manager.cc:1367] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240502 08:16:46.871264 24175 raft_consensus.cc:348] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.871567 24175 raft_consensus.cc:374] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.871677 24175 raft_consensus.cc:729] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.872149 24175 consensus_queue.cc:260] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.873804 24175 ts_tablet_manager.cc:1398] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:16:46.879742 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 586fc16fa81f40c1b8554f9894be4092 (DEFAULT_TABLE table=default.Table_f0a7174ad23742efad1bbd0a285e2731 [id=b12fbfbcb96f42df883e4fabbcc4c6cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.880767 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586fc16fa81f40c1b8554f9894be4092. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.885257 24175 tablet_bootstrap.cc:492] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:46.885850 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 586fc16fa81f40c1b8554f9894be4092 (DEFAULT_TABLE table=default.Table_f0a7174ad23742efad1bbd0a285e2731 [id=b12fbfbcb96f42df883e4fabbcc4c6cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.886472 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586fc16fa81f40c1b8554f9894be4092. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.887326 24175 tablet_bootstrap.cc:654] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.893860 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet b844458ae73f413cbf2113027a5af359 with delete_type TABLET_DATA_DELETED (replaced by tablet 1638e78b6f544209bd8d6da6556fbce4) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:46.894237 24179 tablet_replica.cc:330] T b844458ae73f413cbf2113027a5af359 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:46.894474 24179 raft_consensus.cc:2229] T b844458ae73f413cbf2113027a5af359 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:46.894706 24179 raft_consensus.cc:2258] T b844458ae73f413cbf2113027a5af359 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:46.895424 24179 ts_tablet_manager.cc:1875] T b844458ae73f413cbf2113027a5af359 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:46.897850 24842 raft_consensus.cc:482] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.898053 24842 raft_consensus.cc:504] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.898948 24842 leader_election.cc:288] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:46.899897 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98a96292b28a4b0aad88fe0a96730a68" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:46.900643 23996 leader_election.cc:341] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.901468 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98a96292b28a4b0aad88fe0a96730a68" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:46.901746 23788 raft_consensus.cc:2454] T 98a96292b28a4b0aad88fe0a96730a68 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:46.902320 23993 leader_election.cc:302] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:46.902878 24842 raft_consensus.cc:2781] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:46.902951 24842 raft_consensus.cc:482] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:46.903023 24842 raft_consensus.cc:3037] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.908037 24175 tablet_bootstrap.cc:492] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:46.908223 24175 ts_tablet_manager.cc:1367] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.004s
I20240502 08:16:46.909371 24175 raft_consensus.cc:348] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.909659 24175 raft_consensus.cc:374] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.909729 24175 raft_consensus.cc:729] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.910007 24175 consensus_queue.cc:260] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.911018 24175 ts_tablet_manager.cc:1398] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:46.911319 24175 tablet_bootstrap.cc:492] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:46.912184 24842 raft_consensus.cc:504] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.913102 24175 tablet_bootstrap.cc:654] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.913810 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98a96292b28a4b0aad88fe0a96730a68" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:46.914482 23996 leader_election.cc:341] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:46.915535 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98a96292b28a4b0aad88fe0a96730a68" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:46.915792 23788 raft_consensus.cc:3037] T 98a96292b28a4b0aad88fe0a96730a68 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.919313 23788 raft_consensus.cc:2454] T 98a96292b28a4b0aad88fe0a96730a68 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:46.920033 23993 leader_election.cc:302] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:46.921300 24842 leader_election.cc:288] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:46.921686 24842 raft_consensus.cc:2781] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:46.922160 24175 tablet_bootstrap.cc:492] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:46.922327 24175 ts_tablet_manager.cc:1367] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 08:16:46.922379 24842 raft_consensus.cc:686] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:46.923393 24842 consensus_queue.cc:237] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.923831 24175 raft_consensus.cc:348] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.924111 24175 raft_consensus.cc:374] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.924191 24175 raft_consensus.cc:729] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.924746 24175 consensus_queue.cc:260] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.925592 24175 ts_tablet_manager.cc:1398] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:46.925916 24175 tablet_bootstrap.cc:492] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:46.927799 24179 ts_tablet_manager.cc:1888] T b844458ae73f413cbf2113027a5af359 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:16:46.928180 24175 tablet_bootstrap.cc:654] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.933691 24179 log.cc:1196] T b844458ae73f413cbf2113027a5af359 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/b844458ae73f413cbf2113027a5af359
I20240502 08:16:46.940404 24179 ts_tablet_manager.cc:1909] T b844458ae73f413cbf2113027a5af359 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:46.941351 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet b844458ae73f413cbf2113027a5af359 (table default.Table_a86b6b9ac2d8498c97341dc60ca5ee34 [id=fd93259a378b4ea582a6661a59104abb]) successfully deleted
I20240502 08:16:46.941939 24175 tablet_bootstrap.cc:492] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:46.942189 24175 ts_tablet_manager.cc:1367] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20240502 08:16:46.943162 24175 raft_consensus.cc:348] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.943392 24175 raft_consensus.cc:374] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.943465 24175 raft_consensus.cc:729] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.943719 24175 consensus_queue.cc:260] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.944859 24175 ts_tablet_manager.cc:1398] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:46.945251 24175 tablet_bootstrap.cc:492] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:46.958500 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 586fc16fa81f40c1b8554f9894be4092 (DEFAULT_TABLE table=default.Table_f0a7174ad23742efad1bbd0a285e2731 [id=b12fbfbcb96f42df883e4fabbcc4c6cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.959223 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586fc16fa81f40c1b8554f9894be4092. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:46.963550 23993 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 43544.8us
I20240502 08:16:46.965451 24173 tablet_bootstrap.cc:492] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:46.965634 24173 ts_tablet_manager.cc:1367] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.291s user 0.077s sys 0.116s
I20240502 08:16:46.966612 24173 raft_consensus.cc:348] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.966804 24173 raft_consensus.cc:374] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.966856 24173 raft_consensus.cc:729] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.967056 24173 consensus_queue.cc:260] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.967622 24173 ts_tablet_manager.cc:1398] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:46.967891 24173 tablet_bootstrap.cc:492] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:46.968703 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 069ee3844b514038a6cdf9313766b702 (DEFAULT_TABLE table=default.Table_5ffe8637fd9b4cd3b5b5aa0eb542b267 [id=4a4481b21b9d42929ba223dc5f44fd15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.969060 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 069ee3844b514038a6cdf9313766b702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.969343 24173 tablet_bootstrap.cc:654] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.971212 24175 tablet_bootstrap.cc:654] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.973851 24174 tablet_bootstrap.cc:492] T 1e5de793e74247dd9e73b7b1d58027d9 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:46.974007 24174 ts_tablet_manager.cc:1367] T 1e5de793e74247dd9e73b7b1d58027d9 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.426s user 0.127s sys 0.213s
I20240502 08:16:46.974735 24174 raft_consensus.cc:348] T 1e5de793e74247dd9e73b7b1d58027d9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.974990 24174 raft_consensus.cc:374] T 1e5de793e74247dd9e73b7b1d58027d9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.975069 24174 raft_consensus.cc:729] T 1e5de793e74247dd9e73b7b1d58027d9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.977169 24174 consensus_queue.cc:260] T 1e5de793e74247dd9e73b7b1d58027d9 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:46.980399 24174 ts_tablet_manager.cc:1398] T 1e5de793e74247dd9e73b7b1d58027d9 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 08:16:46.980721 24174 tablet_bootstrap.cc:492] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:46.982477 24174 tablet_bootstrap.cc:654] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.998909 24173 tablet_bootstrap.cc:492] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:46.999075 24173 ts_tablet_manager.cc:1367] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.025s
I20240502 08:16:46.999382 24944 raft_consensus.cc:482] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.999554 24944 raft_consensus.cc:504] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.001400 24173 raft_consensus.cc:348] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.001631 24173 raft_consensus.cc:374] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.001693 24173 raft_consensus.cc:729] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.001891 24173 consensus_queue.cc:260] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.002664 24173 ts_tablet_manager.cc:1398] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:47.002971 24173 tablet_bootstrap.cc:492] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:47.003240 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9fcc7accb1f45ad9cf55b8f64d21b91" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:47.003783 23996 leader_election.cc:341] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:47.004195 24944 leader_election.cc:288] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:47.004599 24173 tablet_bootstrap.cc:654] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.005828 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9fcc7accb1f45ad9cf55b8f64d21b91" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:47.006107 23788 raft_consensus.cc:2454] T d9fcc7accb1f45ad9cf55b8f64d21b91 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:47.006785 23993 leader_election.cc:302] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:47.007846 24842 raft_consensus.cc:2781] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:47.007979 24842 raft_consensus.cc:482] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:47.008095 24842 raft_consensus.cc:3037] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:47.013971 24173 tablet_bootstrap.cc:492] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:47.014122 24173 ts_tablet_manager.cc:1367] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20240502 08:16:47.014792 24173 raft_consensus.cc:348] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.015002 24173 raft_consensus.cc:374] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.015054 24173 raft_consensus.cc:729] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.015266 24173 consensus_queue.cc:260] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.015751 24173 ts_tablet_manager.cc:1398] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:47.016032 24173 tablet_bootstrap.cc:492] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:47.017333 24173 tablet_bootstrap.cc:654] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.022053 24173 tablet_bootstrap.cc:492] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:47.022174 24173 ts_tablet_manager.cc:1367] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:16:47.022819 24173 raft_consensus.cc:348] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.022987 24173 raft_consensus.cc:374] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.023036 24173 raft_consensus.cc:729] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.023241 24173 consensus_queue.cc:260] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.023710 24173 ts_tablet_manager.cc:1398] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 08:16:47.023955 24173 tablet_bootstrap.cc:492] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:47.026721 24173 tablet_bootstrap.cc:654] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.030311 24173 tablet_bootstrap.cc:492] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:47.030429 24173 ts_tablet_manager.cc:1367] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240502 08:16:47.031325 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 069ee3844b514038a6cdf9313766b702 (DEFAULT_TABLE table=default.Table_5ffe8637fd9b4cd3b5b5aa0eb542b267 [id=4a4481b21b9d42929ba223dc5f44fd15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.031723 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 069ee3844b514038a6cdf9313766b702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.037962 24173 raft_consensus.cc:348] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.038225 24173 raft_consensus.cc:374] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.038326 24173 raft_consensus.cc:729] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.038568 24173 consensus_queue.cc:260] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.039194 24173 ts_tablet_manager.cc:1398] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 08:16:47.039180 24842 raft_consensus.cc:504] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.039549 24173 tablet_bootstrap.cc:492] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:47.041249 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9fcc7accb1f45ad9cf55b8f64d21b91" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:47.041795 23996 leader_election.cc:341] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:47.042132 24842 leader_election.cc:288] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:47.042543 24173 tablet_bootstrap.cc:654] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.043202 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9fcc7accb1f45ad9cf55b8f64d21b91" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:47.043398 23788 raft_consensus.cc:3037] T d9fcc7accb1f45ad9cf55b8f64d21b91 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:47.046952 23788 raft_consensus.cc:2454] T d9fcc7accb1f45ad9cf55b8f64d21b91 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:47.050777 23605 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 51aeffcfa23b4bb39e5966cb5820ad41
I20240502 08:16:47.050900 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 25c8a6aaabe2413d8ce88f54ce45a4c4
I20240502 08:16:47.051117 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5d5022ea23c34e4f960c366d5ac96c39" } modify_external_catalogs: true
I20240502 08:16:47.051314 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5d5022ea23c34e4f960c366d5ac96c39" } modify_external_catalogs: true
I20240502 08:16:47.051529 23601 catalog_manager.cc:6554] ReplaceTablet: tablet 25c8a6aaabe2413d8ce88f54ce45a4c4 deleted and replaced by tablet 1f42627a877c4f83a75320f1b5ab539f
I20240502 08:16:47.051913 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2df50e9ecea9421b9bfebd892ebe62bd" } new_table_name: "default.Table_f15d0ef92f374df0b6e9efb21261adfb" modify_external_catalogs: true
I20240502 08:16:47.052881 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e61ff7f9f4074074885941b862adf04d" } new_table_name: "default.Table_42e57ecead1e4463ba08536696995fec" modify_external_catalogs: true
I20240502 08:16:47.053380 23608 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_db86cd2cf86e48ffbc6a513b5c2bd7ab [id=6103f133bcdd4f768b92b37245bf5356] alter complete (version 1)
I20240502 08:16:47.056607 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 25c8a6aaabe2413d8ce88f54ce45a4c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 1f42627a877c4f83a75320f1b5ab539f) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:47.057233 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 25c8a6aaabe2413d8ce88f54ce45a4c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25c8a6aaabe2413d8ce88f54ce45a4c4 already in progress: creating tablet
I20240502 08:16:47.058669 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ed62fe1188a546ac8cf56b8f8adcdb62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.059350 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed62fe1188a546ac8cf56b8f8adcdb62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.062240 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b1fa17dad143f0a7f1751e651eb81a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:47.062440 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet d9fcc7accb1f45ad9cf55b8f64d21b91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:47.062644 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet 98a96292b28a4b0aad88fe0a96730a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:47.062680 24026 tablet_service.cc:1514] Processing DeleteTablet for tablet d9fcc7accb1f45ad9cf55b8f64d21b91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.062752 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 98a96292b28a4b0aad88fe0a96730a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.062815 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e5de793e74247dd9e73b7b1d58027d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.062871 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c072eb45e1b41eeb5aafa6d5a786944 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.062985 24180 tablet_replica.cc:330] T b8b1fa17dad143f0a7f1751e651eb81a P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:47.063206 24180 raft_consensus.cc:2229] T b8b1fa17dad143f0a7f1751e651eb81a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:47.063344 24180 raft_consensus.cc:2258] T b8b1fa17dad143f0a7f1751e651eb81a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:47.063922 24180 ts_tablet_manager.cc:1875] T b8b1fa17dad143f0a7f1751e651eb81a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:47.065721 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_86918f47dba74a9eb36be5145a017f5b" } modify_external_catalogs: true
I20240502 08:16:47.065898 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_86918f47dba74a9eb36be5145a017f5b" } modify_external_catalogs: true
I20240502 08:16:47.055773 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_db2ecaab8195464388e2ba94695fce03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.069890 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db2ecaab8195464388e2ba94695fce03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.071887 24180 ts_tablet_manager.cc:1888] T b8b1fa17dad143f0a7f1751e651eb81a P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:47.072032 24180 log.cc:1196] T b8b1fa17dad143f0a7f1751e651eb81a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/b8b1fa17dad143f0a7f1751e651eb81a
I20240502 08:16:47.072379 24180 ts_tablet_manager.cc:1909] T b8b1fa17dad143f0a7f1751e651eb81a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:47.073386 24175 tablet_bootstrap.cc:492] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:47.073406 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet b8b1fa17dad143f0a7f1751e651eb81a (table default.Table_d4bd37f8c6e54ecf82cd247846aa06d0 [id=57722a7341c541b19efa213bcf48eb53]) successfully deleted
I20240502 08:16:47.073535 24175 ts_tablet_manager.cc:1367] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.128s user 0.006s sys 0.000s
I20240502 08:16:47.073812 24180 tablet_replica.cc:330] T d9fcc7accb1f45ad9cf55b8f64d21b91 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:47.074035 24180 raft_consensus.cc:2229] T d9fcc7accb1f45ad9cf55b8f64d21b91 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:47.074227 24180 raft_consensus.cc:2258] T d9fcc7accb1f45ad9cf55b8f64d21b91 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:47.074330 24175 raft_consensus.cc:348] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.074577 24175 raft_consensus.cc:374] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.074642 24175 raft_consensus.cc:729] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.074932 24180 ts_tablet_manager.cc:1875] T d9fcc7accb1f45ad9cf55b8f64d21b91 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:47.074837 24175 consensus_queue.cc:260] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.077658 24175 ts_tablet_manager.cc:1398] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:47.078006 24181 tablet_replica.cc:330] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:47.078092 24175 tablet_bootstrap.cc:492] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:47.078258 24181 raft_consensus.cc:2229] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:47.078418 24181 raft_consensus.cc:2258] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:47.079094 24181 ts_tablet_manager.cc:1875] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:47.079898 24175 tablet_bootstrap.cc:654] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.081779 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 25c8a6aaabe2413d8ce88f54ce45a4c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 1f42627a877c4f83a75320f1b5ab539f) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.082924 24180 ts_tablet_manager.cc:1888] T d9fcc7accb1f45ad9cf55b8f64d21b91 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:47.086024 24180 log.cc:1196] T d9fcc7accb1f45ad9cf55b8f64d21b91 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/d9fcc7accb1f45ad9cf55b8f64d21b91
I20240502 08:16:47.085984 23993 leader_election.cc:302] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
W20240502 08:16:47.086205 23993 raft_consensus.cc:2670] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:16:47.086720 24180 ts_tablet_manager.cc:1909] T d9fcc7accb1f45ad9cf55b8f64d21b91 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:47.088068 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet d9fcc7accb1f45ad9cf55b8f64d21b91 (table default.Table_24d1bd1b4dcc4882884e13873cf896ff [id=9d91876e10fd42f883117eb1434d3a8f]) successfully deleted
I20240502 08:16:47.089044 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 2411259b872e4be4b8eb8832b36094c4
I20240502 08:16:47.089243 24180 tablet_replica.cc:330] T 98a96292b28a4b0aad88fe0a96730a68 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:47.089411 24180 raft_consensus.cc:2229] T 98a96292b28a4b0aad88fe0a96730a68 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240502 08:16:47.089812 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c71f7398eab4c099cb45a55e50eee51 has no consensus state
I20240502 08:16:47.089555 24180 raft_consensus.cc:2258] T 98a96292b28a4b0aad88fe0a96730a68 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:47.090591 24180 ts_tablet_manager.cc:1875] T 98a96292b28a4b0aad88fe0a96730a68 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:47.091475 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet 2411259b872e4be4b8eb8832b36094c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:46 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:47.092584 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 2411259b872e4be4b8eb8832b36094c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:46 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.092797 24944 raft_consensus.cc:482] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.093006 24944 raft_consensus.cc:504] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.094182 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2126bf2d4ab44628b52481dc2ec3a99" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:47.094283 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_56944338c8ef4b8a98aff14eef9fa611"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.094972 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56944338c8ef4b8a98aff14eef9fa611 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:47.095005 23996 leader_election.cc:341] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:47.095481 24944 leader_election.cc:288] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:47.097378 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b1fa17dad143f0a7f1751e651eb81a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.097613 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet c17e1be6ef294769a01981f4d361be21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.089421 23610 catalog_manager.cc:5484] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:47.097880 23610 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 27093862fef54c5c85d810ebeff9819c (table default.Table_aded763332144f979b418931264d1eb5 [id=0035c98028aa49f7a7dcea576bb23351]). Expected version 1 got 0
I20240502 08:16:47.098381 23610 catalog_manager.cc:5484] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:47.098865 23610 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet f4b291bdb81a4efbac7f6691e5577bbe (table default.Table_2d6b281d29d14441ac6abe9705633102 [id=73559755c4594bcfa591d054964154ae]). Expected version 2 got 0
I20240502 08:16:47.099184 23610 catalog_manager.cc:5484] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:47.099581 23610 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet b811b81ddf2b4dc9b5e5f80331bff294 (table default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418]). Expected version 2 got 0
I20240502 08:16:47.099975 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 319fabf91458438ba5e40ab68b323055 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.101471 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_41458994cfa3412b8d5c9e2948df25e3" } new_table_name: "default.Table_769de68591d64bdb9b80f7bb337da552" modify_external_catalogs: true
I20240502 08:16:47.102414 24180 ts_tablet_manager.cc:1888] T 98a96292b28a4b0aad88fe0a96730a68 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:47.103317 24180 log.cc:1196] T 98a96292b28a4b0aad88fe0a96730a68 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/98a96292b28a4b0aad88fe0a96730a68
I20240502 08:16:47.103929 24180 ts_tablet_manager.cc:1909] T 98a96292b28a4b0aad88fe0a96730a68 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:47.104799 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet df9402cf8abc4b4d893d42f9a3a99016 with delete_type TABLET_DATA_DELETED (replaced by tablet fddac683eda14ec1a0140926380f2c07) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.105978 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 98a96292b28a4b0aad88fe0a96730a68 (table default.Table_2849e243f1a94b139d3234f0c8e3c755 [id=120f59a131ed47bb8f56dbc3a4ebec3b]) successfully deleted
W20240502 08:16:47.106770 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 2411259b872e4be4b8eb8832b36094c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2411259b872e4be4b8eb8832b36094c4 already in progress: creating tablet
I20240502 08:16:47.113222 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2126bf2d4ab44628b52481dc2ec3a99" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:47.114189 23993 leader_election.cc:341] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:47.114439 23993 leader_election.cc:302] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:47.114912 24944 raft_consensus.cc:482] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.115139 24944 raft_consensus.cc:504] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.116493 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319fabf91458438ba5e40ab68b323055" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:47.118605 23996 leader_election.cc:341] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:47.132238 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fc85cce715742cf999ad41ebc1deebd consensus state has no leader
I20240502 08:16:47.132781 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1d04c3885cac4598a01f1669c65be7c5" } modify_external_catalogs: true
I20240502 08:16:47.133018 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1d04c3885cac4598a01f1669c65be7c5" } modify_external_catalogs: true
I20240502 08:16:47.137584 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_bebe57a661bf4fa1b75922c7847aa294"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.138134 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bebe57a661bf4fa1b75922c7847aa294 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.140753 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_f8c00933a68d47e7abd51a6d9b1b27af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.141430 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8c00933a68d47e7abd51a6d9b1b27af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.143481 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0f6ad9ecb09e4a03b995f7350e0a3af0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.144048 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f6ad9ecb09e4a03b995f7350e0a3af0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.148283 24958 raft_consensus.cc:2726] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:47.148370 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet bb4b4a4d6a364de8af3b54d99a0ae138
I20240502 08:16:47.149256 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet bb4b4a4d6a364de8af3b54d99a0ae138 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:47.149175 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_664d4018b9074906bd16eb6782673acc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.149679 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_664d4018b9074906bd16eb6782673acc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:47.150015 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet bb4b4a4d6a364de8af3b54d99a0ae138 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb4b4a4d6a364de8af3b54d99a0ae138 already in progress: creating tablet
I20240502 08:16:47.151101 24958 raft_consensus.cc:482] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.151772 24958 raft_consensus.cc:504] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.154826 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b1fa17dad143f0a7f1751e651eb81a" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:47.155472 23996 leader_election.cc:341] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:47.155861 24958 leader_election.cc:288] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:47.156754 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba3aa2621fdb43fd8299f81af486ef1e consensus state has no leader
I20240502 08:16:47.157269 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_96165587b33b444c8ec8a3b2df7a77d1" } new_table_name: "default.Table_3edd27da6927478281e76361718425a3" modify_external_catalogs: true
I20240502 08:16:47.158772 23607 master_service.cc:946] ReplaceTablet: received request to replace tablet ba3aa2621fdb43fd8299f81af486ef1e from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:47.161867 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e5de793e74247dd9e73b7b1d58027d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:47.162304 24180 tablet_replica.cc:330] T 1e5de793e74247dd9e73b7b1d58027d9 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:47.162508 24180 raft_consensus.cc:2229] T 1e5de793e74247dd9e73b7b1d58027d9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:47.162657 24180 raft_consensus.cc:2258] T 1e5de793e74247dd9e73b7b1d58027d9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:47.163547 24180 ts_tablet_manager.cc:1875] T 1e5de793e74247dd9e73b7b1d58027d9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:47.169245 23602 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet f4b291bdb81a4efbac7f6691e5577bbe (table default.Table_2d6b281d29d14441ac6abe9705633102 [id=73559755c4594bcfa591d054964154ae]). Expected version 2 got 0
I20240502 08:16:47.169849 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 27346a7eae6a488fa3415646262a4b88
I20240502 08:16:47.170370 24774 raft_consensus.cc:482] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.170512 24774 raft_consensus.cc:504] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.171844 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5768cb928e244d2914f10f4eda3c1cd" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:47.172063 24057 raft_consensus.cc:2454] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:47.172603 23723 leader_election.cc:302] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:47.172933 24778 raft_consensus.cc:2781] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:47.173000 24778 raft_consensus.cc:482] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:47.173055 24778 raft_consensus.cc:3037] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:47.176033 24778 raft_consensus.cc:504] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.176695 24778 leader_election.cc:288] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:47.177534 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5768cb928e244d2914f10f4eda3c1cd" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:47.177747 24057 raft_consensus.cc:3037] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:47.178939 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 48d96e42813f4169969dd5931e169923
I20240502 08:16:47.179872 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 48d96e42813f4169969dd5931e169923 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:47.180339 24057 raft_consensus.cc:2454] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:47.180693 24774 leader_election.cc:288] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:47.181025 23723 leader_election.cc:302] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:47.181730 24180 ts_tablet_manager.cc:1888] T 1e5de793e74247dd9e73b7b1d58027d9 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:47.182036 24180 log.cc:1196] T 1e5de793e74247dd9e73b7b1d58027d9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/1e5de793e74247dd9e73b7b1d58027d9
I20240502 08:16:47.182439 24180 ts_tablet_manager.cc:1909] T 1e5de793e74247dd9e73b7b1d58027d9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:47.183593 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 1e5de793e74247dd9e73b7b1d58027d9 (table default.Table_a286e67e441e43a1b4738cd219dbea98 [id=b56e3c0686244ad6b49afdd589c5ed34]) successfully deleted
I20240502 08:16:47.181924 24774 raft_consensus.cc:2781] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:47.184325 24774 raft_consensus.cc:686] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:47.184659 24774 consensus_queue.cc:237] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:47.186858 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 48d96e42813f4169969dd5931e169923 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48d96e42813f4169969dd5931e169923 already in progress: creating tablet
I20240502 08:16:47.192122 24057 raft_consensus.cc:1261] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:47.193042 24778 consensus_queue.cc:1035] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:47.197537 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet ba3aa2621fdb43fd8299f81af486ef1e
I20240502 08:16:47.198032 23607 catalog_manager.cc:6554] ReplaceTablet: tablet ba3aa2621fdb43fd8299f81af486ef1e deleted and replaced by tablet e20e126b1c9546e4a194bf9842f444a5
I20240502 08:16:47.198585 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet ba3aa2621fdb43fd8299f81af486ef1e with delete_type TABLET_DATA_DELETED (replaced by tablet e20e126b1c9546e4a194bf9842f444a5) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:47.198664 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2da8b9d3f374d8a812336057c0ce7d2 consensus state has no leader
I20240502 08:16:47.198901 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet ba3aa2621fdb43fd8299f81af486ef1e with delete_type TABLET_DATA_DELETED (replaced by tablet e20e126b1c9546e4a194bf9842f444a5) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.199278 24778 consensus_queue.cc:1035] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:47.199364 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet ba3aa2621fdb43fd8299f81af486ef1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba3aa2621fdb43fd8299f81af486ef1e already in progress: creating tablet
I20240502 08:16:47.204082 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet 27346a7eae6a488fa3415646262a4b88 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:47.208238 24180 tablet_replica.cc:330] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:47.208452 24180 raft_consensus.cc:2229] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:47.208686 24180 pending_rounds.cc:62] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:16:47.208734 24180 pending_rounds.cc:68] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156454317977600 op_type: NO_OP noop_request { }
I20240502 08:16:47.208855 24180 raft_consensus.cc:2866] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:47.208930 24180 raft_consensus.cc:2258] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:47.210078 24180 ts_tablet_manager.cc:1875] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:47.217850 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 27346a7eae6a488fa3415646262a4b88 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.227933 24944 leader_election.cc:288] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:47.245188 23603 master_service.cc:946] ReplaceTablet: received request to replace tablet 202fa612a085450ca9c1225434cb7fe4 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:47.253942 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 202fa612a085450ca9c1225434cb7fe4
I20240502 08:16:47.254637 23603 catalog_manager.cc:6554] ReplaceTablet: tablet 202fa612a085450ca9c1225434cb7fe4 deleted and replaced by tablet d5c82820278b4a0c8f7d586f87104bf0
I20240502 08:16:47.255071 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 202fa612a085450ca9c1225434cb7fe4 with delete_type TABLET_DATA_DELETED (replaced by tablet d5c82820278b4a0c8f7d586f87104bf0) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.255573 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet 202fa612a085450ca9c1225434cb7fe4 with delete_type TABLET_DATA_DELETED (replaced by tablet d5c82820278b4a0c8f7d586f87104bf0) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:47.266448 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0ba7fb682ff94893b3ef11abc45a7f6d" } modify_external_catalogs: true
I20240502 08:16:47.266755 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0ba7fb682ff94893b3ef11abc45a7f6d" } modify_external_catalogs: true
I20240502 08:16:47.269554 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_cce64bbdca4946bfb1cdbd52d1d2d59c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.270167 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cce64bbdca4946bfb1cdbd52d1d2d59c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.272482 24174 tablet_bootstrap.cc:492] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:47.272641 24174 ts_tablet_manager.cc:1367] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.292s user 0.056s sys 0.096s
I20240502 08:16:47.273404 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319fabf91458438ba5e40ab68b323055" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:47.273455 24174 raft_consensus.cc:348] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.274323 24174 raft_consensus.cc:374] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.274399 24174 raft_consensus.cc:729] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.274600 24174 consensus_queue.cc:260] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.275110 24174 ts_tablet_manager.cc:1398] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:47.275383 24174 tablet_bootstrap.cc:492] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:47.275959 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 4baa2bc2752c450fa212471c150cf098
I20240502 08:16:47.277597 24174 tablet_bootstrap.cc:654] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.279453 24175 tablet_bootstrap.cc:492] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:47.279592 24175 ts_tablet_manager.cc:1367] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.202s user 0.026s sys 0.027s
I20240502 08:16:47.280354 24175 raft_consensus.cc:348] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.280550 24175 raft_consensus.cc:374] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.280612 24175 raft_consensus.cc:729] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.280823 24175 consensus_queue.cc:260] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.284330 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 4baa2bc2752c450fa212471c150cf098 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.286084 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet bb4b4a4d6a364de8af3b54d99a0ae138 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:47.287746 23993 leader_election.cc:341] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: b8b1fa17dad143f0a7f1751e651eb81a
I20240502 08:16:47.287935 23993 leader_election.cc:302] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:47.287963 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet 4baa2bc2752c450fa212471c150cf098 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:47.288504 23993 leader_election.cc:341] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:47.288619 23607 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet b811b81ddf2b4dc9b5e5f80331bff294 (table default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418]). Expected version 2 got 0
I20240502 08:16:47.288662 23993 leader_election.cc:302] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:47.289003 24958 raft_consensus.cc:482] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.289120 24958 raft_consensus.cc:504] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.289335 24944 raft_consensus.cc:2726] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:47.289491 24944 raft_consensus.cc:2726] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:47.289700 24958 leader_election.cc:288] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:47.290045 24175 ts_tablet_manager.cc:1398] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240502 08:16:47.290293 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5de793e74247dd9e73b7b1d58027d9" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:47.290768 23996 leader_election.cc:341] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:47.292093 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0692aa66a8ad444eab3af22ad78c527e" } new_table_name: "default.Table_a5705fc20e5449ffb41ac300e8b3b959" modify_external_catalogs: true
I20240502 08:16:47.292546 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 48d96e42813f4169969dd5931e169923 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.292354 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ddf5aa41088d4de4bcf2768d82c7d1fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.293061 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddf5aa41088d4de4bcf2768d82c7d1fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.293033 23603 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet b811b81ddf2b4dc9b5e5f80331bff294 (table default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418]). Expected version 2 got 0
I20240502 08:16:47.295791 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1a8e51c22c1a4da9928bb3cf87102b6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.296448 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a8e51c22c1a4da9928bb3cf87102b6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.297847 24175 tablet_bootstrap.cc:492] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:47.298969 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ee8e3c4f16e74576b385099a280d5312" } modify_external_catalogs: true
I20240502 08:16:47.299170 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ee8e3c4f16e74576b385099a280d5312" } modify_external_catalogs: true
I20240502 08:16:47.299335 24175 tablet_bootstrap.cc:654] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.310165 23609 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet f4b291bdb81a4efbac7f6691e5577bbe (table default.Table_2d6b281d29d14441ac6abe9705633102 [id=73559755c4594bcfa591d054964154ae]). Expected version 2 got 0
I20240502 08:16:47.310460 24175 tablet_bootstrap.cc:492] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:47.310621 24175 ts_tablet_manager.cc:1367] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240502 08:16:47.311342 24175 raft_consensus.cc:348] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.311544 24175 raft_consensus.cc:374] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.311604 24175 raft_consensus.cc:729] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.311789 24175 consensus_queue.cc:260] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:47.312122 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c234ea188e104d87a951bc195e968c0a consensus state has no leader
I20240502 08:16:47.314507 24180 ts_tablet_manager.cc:1888] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:47.318295 24181 ts_tablet_manager.cc:1888] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:47.319181 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet d36e7cd33f9a4edabe690e75602590b9
I20240502 08:16:47.322521 23603 catalog_manager.cc:5484] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
W20240502 08:16:47.323086 23434 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 90045.4us
I20240502 08:16:47.323714 24175 ts_tablet_manager.cc:1398] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240502 08:16:47.324972 24180 log.cc:1196] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/27346a7eae6a488fa3415646262a4b88
I20240502 08:16:47.325742 24181 log.cc:1196] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/d9fcc7accb1f45ad9cf55b8f64d21b91
I20240502 08:16:47.326149 24181 ts_tablet_manager.cc:1909] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:47.320463 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet d36e7cd33f9a4edabe690e75602590b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:47.327463 23993 leader_election.cc:341] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 1e5de793e74247dd9e73b7b1d58027d9
I20240502 08:16:47.327677 23993 leader_election.cc:302] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:47.328390 24944 raft_consensus.cc:2726] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:47.325517 24180 ts_tablet_manager.cc:1909] T 27346a7eae6a488fa3415646262a4b88 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:47.330631 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet d9fcc7accb1f45ad9cf55b8f64d21b91 (table default.Table_24d1bd1b4dcc4882884e13873cf896ff [id=9d91876e10fd42f883117eb1434d3a8f]) successfully deleted
I20240502 08:16:47.331028 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 27346a7eae6a488fa3415646262a4b88 (table default.Table_5d5022ea23c34e4f960c366d5ac96c39 [id=999c7344afec4d3c858dc29a986a8426]) successfully deleted
I20240502 08:16:47.331238 24175 tablet_bootstrap.cc:492] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:47.331511 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 202fa612a085450ca9c1225434cb7fe4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 202fa612a085450ca9c1225434cb7fe4 already in progress: creating tablet
W20240502 08:16:47.331784 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 4baa2bc2752c450fa212471c150cf098 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4baa2bc2752c450fa212471c150cf098 already in progress: creating tablet
I20240502 08:16:47.319943 24774 raft_consensus.cc:482] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.332737 24790 raft_consensus.cc:482] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.323165 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet d36e7cd33f9a4edabe690e75602590b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:47.332855 24790 raft_consensus.cc:504] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.333065 24774 raft_consensus.cc:504] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.333470 24790 leader_election.cc:288] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:47.336125 24774 leader_election.cc:288] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:47.336628 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7e64e6023d104b96b6f3ae5c7963e32e" } new_table_name: "default.Table_4c4a02d4a6eb4c87971a03d670f5fdaf" modify_external_catalogs: true
I20240502 08:16:47.336854 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b291bdb81a4efbac7f6691e5577bbe" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:47.336669 23723 leader_election.cc:341] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 4a57ed188315493198e95cefb6907ab6
I20240502 08:16:47.341702 24181 tablet_replica.cc:330] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:47.341872 24969 tablet.cc:1722] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:47.341943 24181 raft_consensus.cc:2229] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:47.342259 24181 pending_rounds.cc:62] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5: Trying to abort 1 pending ops.
W20240502 08:16:47.344771 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07344b78894c42809391d0fe2a9cd8f7 consensus state has no leader
I20240502 08:16:47.344964 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_52cae77d343b4c43bb8cf4fa6f4727af" } new_table_name: "default.Table_b2bb1aeaff464b1cb086d31a723ab834" modify_external_catalogs: true
I20240502 08:16:47.351395 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fc2c7b3f23234f139178c7719ab5f044"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.351564 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet d36e7cd33f9a4edabe690e75602590b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d36e7cd33f9a4edabe690e75602590b9 already in progress: creating tablet
I20240502 08:16:47.352114 23607 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet f4b291bdb81a4efbac7f6691e5577bbe (table default.Table_2d6b281d29d14441ac6abe9705633102 [id=73559755c4594bcfa591d054964154ae]). Expected version 2 got 0
W20240502 08:16:47.352427 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc2c7b3f23234f139178c7719ab5f044 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.353015 24790 raft_consensus.cc:482] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.353144 24790 raft_consensus.cc:504] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.353751 24790 leader_election.cc:288] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:47.354408 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c456616beb454b6ea33638f85512f97e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.354688 23861 leader_election.cc:341] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 643ce929e74c499da835fbe9b63b8880
W20240502 08:16:47.354920 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c456616beb454b6ea33638f85512f97e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.355098 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8ac26bacf7bc4a05bacf7f35c2755a74" } modify_external_catalogs: true
I20240502 08:16:47.355298 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8ac26bacf7bc4a05bacf7f35c2755a74" } modify_external_catalogs: true
I20240502 08:16:47.356671 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a57ed188315493198e95cefb6907ab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:47.357160 24175 tablet_bootstrap.cc:654] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.357740 24180 tablet_replica.cc:330] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:47.358017 24180 raft_consensus.cc:2229] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:47.359747 24715 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:47.360371 24180 raft_consensus.cc:2258] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:47.363937 24180 ts_tablet_manager.cc:1875] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:47.363994 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3cc189507d0b49728fb22ca590126312 consensus state has no leader
I20240502 08:16:47.365332 24972 tablet.cc:1722] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:47.369112 24790 raft_consensus.cc:482] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.369253 24790 raft_consensus.cc:504] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.369884 24790 leader_election.cc:288] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:47.344079 24970 tablet.cc:1722] T f4b291bdb81a4efbac7f6691e5577bbe P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:47.370069 24181 pending_rounds.cc:68] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156457185300480 op_type: NO_OP noop_request { }
I20240502 08:16:47.370167 24181 raft_consensus.cc:2866] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:47.370234 24181 raft_consensus.cc:2258] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:47.370376 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b811b81ddf2b4dc9b5e5f80331bff294" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:47.370792 24057 raft_consensus.cc:2379] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:16:47.369933 24056 raft_consensus.cc:1261] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:47.374617 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9a9569245562493dafbeb639c6492826" } new_table_name: "default.Table_791a010a6e484e10a6749eefcfdc293d" modify_external_catalogs: true
I20240502 08:16:47.378038 24715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:47.378947 24181 ts_tablet_manager.cc:1875] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:47.380003 24715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:47.382318 24976 tablet.cc:1722] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:47.382488 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ab1b6d7ec1d44a4f8559d75bb38fb8a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.383136 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab1b6d7ec1d44a4f8559d75bb38fb8a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.383175 24891 consensus_queue.cc:1035] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:47.385102 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_91a1219d5c2542d9a45c7b8499786337" } modify_external_catalogs: true
I20240502 08:16:47.385368 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_91a1219d5c2542d9a45c7b8499786337" } modify_external_catalogs: true
I20240502 08:16:47.385814 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7b367902a5d84aa5ac4aa4a0b97b48f5" } modify_external_catalogs: true
I20240502 08:16:47.385977 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7b367902a5d84aa5ac4aa4a0b97b48f5" } modify_external_catalogs: true
I20240502 08:16:47.386509 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_59fc9761e46d4ff0bad5029d2b0b25fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.386940 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59fc9761e46d4ff0bad5029d2b0b25fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.389053 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_18f465f6e9d048ac8bb97321715d2a6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.389595 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18f465f6e9d048ac8bb97321715d2a6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.398085 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_4b0a59aad44148b6b3656ff66dfca548"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.398609 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b0a59aad44148b6b3656ff66dfca548 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.409739 24976 tablet.cc:1722] T 27093862fef54c5c85d810ebeff9819c P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:47.410485 24972 tablet.cc:1722] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:47.419435 24173 tablet_bootstrap.cc:492] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:47.419644 24173 ts_tablet_manager.cc:1367] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.380s user 0.050s sys 0.070s
I20240502 08:16:47.420367 24173 raft_consensus.cc:348] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.420621 24173 raft_consensus.cc:374] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.420714 24173 raft_consensus.cc:729] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.421006 24173 consensus_queue.cc:260] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.421872 24173 ts_tablet_manager.cc:1398] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:47.422142 23608 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_2d6b281d29d14441ac6abe9705633102 [id=73559755c4594bcfa591d054964154ae] alter complete (version 2)
I20240502 08:16:47.422202 24173 tablet_bootstrap.cc:492] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:47.423023 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0501575a84ec454780f48574bf3155fe consensus state has no leader
I20240502 08:16:47.423645 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8dbc74ff11934c76b732768528c3ed39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:47.425088 24173 tablet_bootstrap.cc:654] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:47.425652 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8dbc74ff11934c76b732768528c3ed39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.418949 24180 ts_tablet_manager.cc:1888] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:47.426172 24180 log.cc:1196] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/4a57ed188315493198e95cefb6907ab6
I20240502 08:16:47.426208 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f15d0ef92f374df0b6e9efb21261adfb" } new_table_name: "default.Table_8bed20f633e348679b0087687298cc42" modify_external_catalogs: true
I20240502 08:16:47.426604 24180 ts_tablet_manager.cc:1909] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:47.427469 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 4a57ed188315493198e95cefb6907ab6 (table default.Table_8c4dfc5c68e84b6ab815e9f333143678 [id=58ae4f5729f44a9c8be960be8a1a13a1]) successfully deleted
I20240502 08:16:47.436060 24173 tablet_bootstrap.cc:492] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:47.436290 24173 ts_tablet_manager.cc:1367] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20240502 08:16:47.437021 24173 raft_consensus.cc:348] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.437892 24173 raft_consensus.cc:374] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.438123 24173 raft_consensus.cc:729] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.438709 24173 consensus_queue.cc:260] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.440694 24173 ts_tablet_manager.cc:1398] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 08:16:47.441357 24173 tablet_bootstrap.cc:492] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:47.445703 24944 raft_consensus.cc:482] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.445814 24944 raft_consensus.cc:504] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.446440 24944 leader_election.cc:288] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:47.449201 24173 tablet_bootstrap.cc:654] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.455719 24790 raft_consensus.cc:482] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.455897 24790 raft_consensus.cc:504] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.457163 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c072eb45e1b41eeb5aafa6d5a786944" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:47.458458 23996 leader_election.cc:341] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:47.459028 24790 leader_election.cc:288] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:47.459169 24790 raft_consensus.cc:482] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.459302 24790 raft_consensus.cc:504] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.459983 24790 leader_election.cc:288] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:47.461009 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27346a7eae6a488fa3415646262a4b88" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:47.462109 23861 leader_election.cc:341] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 2bfb532c9d4c41afb7acaa4831841306
I20240502 08:16:47.463049 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c072eb45e1b41eeb5aafa6d5a786944" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:47.463793 23993 leader_election.cc:341] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:47.463965 23993 leader_election.cc:302] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:47.464218 24175 tablet_bootstrap.cc:492] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:47.464282 24944 raft_consensus.cc:2726] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:47.464362 24175 ts_tablet_manager.cc:1367] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.133s user 0.014s sys 0.027s
I20240502 08:16:47.470409 24175 raft_consensus.cc:348] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.470635 24175 raft_consensus.cc:374] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.470698 24175 raft_consensus.cc:729] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.470886 24175 consensus_queue.cc:260] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.472883 24174 tablet_bootstrap.cc:492] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:47.473049 24174 ts_tablet_manager.cc:1367] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.198s user 0.025s sys 0.032s
I20240502 08:16:47.473863 24174 raft_consensus.cc:348] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.474547 24174 raft_consensus.cc:374] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.474642 24174 raft_consensus.cc:729] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.474858 24174 consensus_queue.cc:260] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.482364 24175 ts_tablet_manager.cc:1398] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240502 08:16:47.486919 24175 tablet_bootstrap.cc:492] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:47.488445 24175 tablet_bootstrap.cc:654] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.490236 24173 tablet_bootstrap.cc:492] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:47.490397 24173 ts_tablet_manager.cc:1367] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.004s
I20240502 08:16:47.491240 24173 raft_consensus.cc:348] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.491453 24173 raft_consensus.cc:374] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.491529 24173 raft_consensus.cc:729] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.491446 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fb39fa7949f3412c83974132184b5d9f" } modify_external_catalogs: true
I20240502 08:16:47.491644 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fb39fa7949f3412c83974132184b5d9f" } modify_external_catalogs: true
I20240502 08:16:47.492565 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7227bf354dec4869bd921dd397b17de3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.493134 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7227bf354dec4869bd921dd397b17de3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.493745 24173 consensus_queue.cc:260] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.494181 24944 raft_consensus.cc:482] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.494282 24944 raft_consensus.cc:504] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.494834 24944 leader_election.cc:288] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:47.497591 24174 ts_tablet_manager.cc:1398] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240502 08:16:47.497813 23608 master_service.cc:946] ReplaceTablet: received request to replace tablet 40964817cf2a4712be42f40e37639bc9 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:47.497943 24174 tablet_bootstrap.cc:492] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:47.499384 24944 raft_consensus.cc:482] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.499519 24944 raft_consensus.cc:504] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.500102 24173 ts_tablet_manager.cc:1398] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240502 08:16:47.500128 24944 leader_election.cc:288] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:47.500401 24173 tablet_bootstrap.cc:492] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:47.503599 24174 tablet_bootstrap.cc:654] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.503677 24173 tablet_bootstrap.cc:654] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.505641 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4a31e5aa834adeb9f59267247c6980" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:47.506270 24175 tablet_bootstrap.cc:492] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:47.506400 24175 ts_tablet_manager.cc:1367] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.001s
I20240502 08:16:47.506328 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_97534a2075bc4b7297b36916b7a5f77c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.506935 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97534a2075bc4b7297b36916b7a5f77c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.507068 24175 raft_consensus.cc:348] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.507299 24175 raft_consensus.cc:374] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.507349 24175 raft_consensus.cc:729] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.507552 24175 consensus_queue.cc:260] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.509763 24175 ts_tablet_manager.cc:1398] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:16:47.509905 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9402cf8abc4b4d893d42f9a3a99016" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:47.510895 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4a31e5aa834adeb9f59267247c6980" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:47.511364 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9402cf8abc4b4d893d42f9a3a99016" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:47.511910 23996 leader_election.cc:341] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:47.512148 23996 leader_election.cc:341] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:47.513984 24175 tablet_bootstrap.cc:492] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:47.515398 23993 leader_election.cc:341] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:47.515661 24175 tablet_bootstrap.cc:654] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.515630 23993 leader_election.cc:302] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:47.515988 24944 raft_consensus.cc:2726] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:47.516547 23993 leader_election.cc:341] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:47.516731 23993 leader_election.cc:302] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:47.517016 24944 raft_consensus.cc:2726] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:47.521420 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_bbc5d6ae8e7246129746580755d1a812"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.522105 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbc5d6ae8e7246129746580755d1a812 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.524355 24174 tablet_bootstrap.cc:492] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:47.524554 24174 ts_tablet_manager.cc:1367] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.002s
I20240502 08:16:47.526108 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_785d5ddac96746379d1117bc8379566b" } new_table_name: "default.Table_81ee0423dbbe4ff899c42249227cea34" modify_external_catalogs: true
I20240502 08:16:47.525877 24174 raft_consensus.cc:348] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.527827 24174 raft_consensus.cc:374] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.527962 24174 raft_consensus.cc:729] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.528307 24174 consensus_queue.cc:260] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.531538 24174 ts_tablet_manager.cc:1398] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:16:47.535130 24174 tablet_bootstrap.cc:492] T 319fabf91458438ba5e40ab68b323055 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:47.537055 24174 tablet_bootstrap.cc:654] T 319fabf91458438ba5e40ab68b323055 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.539295 24175 tablet_bootstrap.cc:492] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:47.539516 24175 ts_tablet_manager.cc:1367] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240502 08:16:47.540315 24175 raft_consensus.cc:348] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.540585 24175 raft_consensus.cc:374] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.540604 24173 tablet_bootstrap.cc:492] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:47.540704 24175 raft_consensus.cc:729] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.540760 24173 ts_tablet_manager.cc:1367] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.012s
I20240502 08:16:47.541013 24175 consensus_queue.cc:260] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.541563 24173 raft_consensus.cc:348] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.541826 24173 raft_consensus.cc:374] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.541898 24173 raft_consensus.cc:729] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.542121 24173 consensus_queue.cc:260] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.546605 24175 ts_tablet_manager.cc:1398] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 08:16:47.547811 24173 ts_tablet_manager.cc:1398] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 08:16:47.548170 24173 tablet_bootstrap.cc:492] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:47.551192 24982 raft_consensus.cc:482] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.551319 24982 raft_consensus.cc:504] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.551692 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b5aef32596c94f51bb917536c2bc7341" } modify_external_catalogs: true
I20240502 08:16:47.551903 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b5aef32596c94f51bb917536c2bc7341" } modify_external_catalogs: true
I20240502 08:16:47.551973 24982 leader_election.cc:288] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:47.552863 23861 leader_election.cc:341] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: bcfacc302ff04178bfffc48111cd7a3f
I20240502 08:16:47.553408 24175 tablet_bootstrap.cc:492] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:47.557914 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_02cd49f90ab14e919f913d9eb37ebdc1" } new_table_name: "default.Table_68edbb83e66948898f02d4e24a28a9fe" modify_external_catalogs: true
I20240502 08:16:47.565973 24173 tablet_bootstrap.cc:654] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.566488 24175 tablet_bootstrap.cc:654] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.573836 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b291bdb81a4efbac7f6691e5577bbe" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:47.576221 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b811b81ddf2b4dc9b5e5f80331bff294" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:47.576330 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 725982e4712e442d8ff1fd3f796e18f5 consensus state has no leader
I20240502 08:16:47.576833 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7937f18be63b41d7945c72ed597adfcb" } new_table_name: "default.Table_2e5b16fa5b3b4be3af4d6843606d91f6" modify_external_catalogs: true
I20240502 08:16:47.578886 24982 raft_consensus.cc:482] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.579049 24982 raft_consensus.cc:504] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.579824 24982 leader_election.cc:288] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:47.581233 24175 tablet_bootstrap.cc:492] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:47.581372 24175 ts_tablet_manager.cc:1367] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.001s
I20240502 08:16:47.581961 24175 raft_consensus.cc:348] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.582129 24175 raft_consensus.cc:374] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.582170 24175 raft_consensus.cc:729] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.582345 24175 consensus_queue.cc:260] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.582782 24175 ts_tablet_manager.cc:1398] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:16:47.586016 24175 tablet_bootstrap.cc:492] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:47.587651 24175 tablet_bootstrap.cc:654] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:47.589253 23861 leader_election.cc:341] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 71aa916247c449fea718b35869a6df20
I20240502 08:16:47.596009 24181 ts_tablet_manager.cc:1888] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:47.596256 24181 log.cc:1196] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/98a96292b28a4b0aad88fe0a96730a68
I20240502 08:16:47.597416 24181 ts_tablet_manager.cc:1909] T 98a96292b28a4b0aad88fe0a96730a68 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
W20240502 08:16:47.597697 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 40964817cf2a4712be42f40e37639bc9 consensus state has no leader
I20240502 08:16:47.599735 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5768cb928e244d2914f10f4eda3c1cd" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:47.600571 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5768cb928e244d2914f10f4eda3c1cd" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:47.603179 23924 raft_consensus.cc:3037] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:47.604040 24175 tablet_bootstrap.cc:492] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:47.604208 24175 ts_tablet_manager.cc:1367] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s
I20240502 08:16:47.605216 24175 raft_consensus.cc:348] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.605518 24175 raft_consensus.cc:374] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.605585 24175 raft_consensus.cc:729] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.605818 24175 consensus_queue.cc:260] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.606771 24175 ts_tablet_manager.cc:1398] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:47.607131 24175 tablet_bootstrap.cc:492] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:47.616086 24972 tablet.cc:1722] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240502 08:16:47.617172 24970 tablet.cc:1722] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240502 08:16:47.609304 24175 tablet_bootstrap.cc:654] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.618175 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a57ed188315493198e95cefb6907ab6" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:47.620142 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 069ee3844b514038a6cdf9313766b702 (DEFAULT_TABLE table=default.Table_5ffe8637fd9b4cd3b5b5aa0eb542b267 [id=4a4481b21b9d42929ba223dc5f44fd15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.620611 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 069ee3844b514038a6cdf9313766b702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.622327 24175 tablet_bootstrap.cc:492] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:47.622424 24175 ts_tablet_manager.cc:1367] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.001s
I20240502 08:16:47.623072 24175 raft_consensus.cc:348] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.623256 24175 raft_consensus.cc:374] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.623311 24175 raft_consensus.cc:729] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.623476 24175 consensus_queue.cc:260] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.623987 24175 ts_tablet_manager.cc:1398] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 08:16:47.624217 24175 tablet_bootstrap.cc:492] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:47.625818 24175 tablet_bootstrap.cc:654] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.630825 24175 tablet_bootstrap.cc:492] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:47.630949 24175 ts_tablet_manager.cc:1367] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240502 08:16:47.631822 24175 raft_consensus.cc:348] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.632004 24175 raft_consensus.cc:374] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.632051 24175 raft_consensus.cc:729] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.632264 24175 consensus_queue.cc:260] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.632779 24175 ts_tablet_manager.cc:1398] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:47.633018 24175 tablet_bootstrap.cc:492] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:47.634367 24174 tablet_bootstrap.cc:492] T 319fabf91458438ba5e40ab68b323055 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:47.634503 24174 ts_tablet_manager.cc:1367] T 319fabf91458438ba5e40ab68b323055 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.100s user 0.028s sys 0.011s
I20240502 08:16:47.635111 24174 raft_consensus.cc:348] T 319fabf91458438ba5e40ab68b323055 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.635282 24174 raft_consensus.cc:374] T 319fabf91458438ba5e40ab68b323055 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.635329 24174 raft_consensus.cc:729] T 319fabf91458438ba5e40ab68b323055 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.635587 24174 consensus_queue.cc:260] T 319fabf91458438ba5e40ab68b323055 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.636044 24174 ts_tablet_manager.cc:1398] T 319fabf91458438ba5e40ab68b323055 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 08:16:47.636317 24174 tablet_bootstrap.cc:492] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:47.637673 24174 tablet_bootstrap.cc:654] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.639636 24175 tablet_bootstrap.cc:654] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.649857 24175 tablet_bootstrap.cc:492] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:47.650009 24175 ts_tablet_manager.cc:1367] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20240502 08:16:47.650707 24175 raft_consensus.cc:348] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.650911 24175 raft_consensus.cc:374] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.650955 24175 raft_consensus.cc:729] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.651124 24175 consensus_queue.cc:260] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.651641 24175 ts_tablet_manager.cc:1398] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:47.651898 24175 tablet_bootstrap.cc:492] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:47.654603 24174 tablet_bootstrap.cc:492] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:47.654742 24174 ts_tablet_manager.cc:1367] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.008s
I20240502 08:16:47.655010 24982 raft_consensus.cc:482] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.655174 24982 raft_consensus.cc:504] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.655398 24174 raft_consensus.cc:348] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.655598 24174 raft_consensus.cc:374] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.655645 24174 raft_consensus.cc:729] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.655831 24174 consensus_queue.cc:260] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.655951 24982 leader_election.cc:288] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:47.656287 24174 ts_tablet_manager.cc:1398] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:16:47.656553 24174 tablet_bootstrap.cc:492] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:47.657984 24174 tablet_bootstrap.cc:654] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.659555 24175 tablet_bootstrap.cc:654] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:47.664757 23861 leader_election.cc:341] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: d9fcc7accb1f45ad9cf55b8f64d21b91
I20240502 08:16:47.666267 24175 tablet_bootstrap.cc:492] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:47.666432 24175 ts_tablet_manager.cc:1367] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.003s
I20240502 08:16:47.667289 24175 raft_consensus.cc:348] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.667528 24175 raft_consensus.cc:374] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.670202 24174 tablet_bootstrap.cc:492] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:47.670359 24174 ts_tablet_manager.cc:1367] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20240502 08:16:47.671061 24174 raft_consensus.cc:348] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.671272 24174 raft_consensus.cc:374] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.671329 24174 raft_consensus.cc:729] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.671551 24174 consensus_queue.cc:260] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.672102 24174 ts_tablet_manager.cc:1398] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:47.672428 24174 tablet_bootstrap.cc:492] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:47.667598 24175 raft_consensus.cc:729] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.673921 24175 consensus_queue.cc:260] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.674542 24175 ts_tablet_manager.cc:1398] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 08:16:47.674850 24175 tablet_bootstrap.cc:492] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:47.676412 24174 tablet_bootstrap.cc:654] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.678115 24175 tablet_bootstrap.cc:654] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.680456 24175 tablet_bootstrap.cc:492] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:47.680599 24175 ts_tablet_manager.cc:1367] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:16:47.681401 24175 raft_consensus.cc:348] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.681667 24175 raft_consensus.cc:374] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.681736 24175 raft_consensus.cc:729] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.682423 24175 consensus_queue.cc:260] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.682898 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 98a96292b28a4b0aad88fe0a96730a68 (table default.Table_2849e243f1a94b139d3234f0c8e3c755 [id=120f59a131ed47bb8f56dbc3a4ebec3b]) successfully deleted
I20240502 08:16:47.682997 24175 ts_tablet_manager.cc:1398] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:16:47.683286 24175 tablet_bootstrap.cc:492] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:47.683640 24181 tablet_replica.cc:330] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:47.684108 24181 raft_consensus.cc:2229] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:47.684502 24181 raft_consensus.cc:2258] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:47.685045 24175 tablet_bootstrap.cc:654] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.685989 24181 ts_tablet_manager.cc:1875] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:47.689164 24174 tablet_bootstrap.cc:492] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:47.689349 24174 ts_tablet_manager.cc:1367] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.001s
I20240502 08:16:47.690182 23924 raft_consensus.cc:1261] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:47.690157 24174 raft_consensus.cc:348] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.690409 24174 raft_consensus.cc:374] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.690474 24174 raft_consensus.cc:729] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.690718 24174 consensus_queue.cc:260] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.691313 24174 ts_tablet_manager.cc:1398] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:47.691610 24174 tablet_bootstrap.cc:492] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:47.693331 24174 tablet_bootstrap.cc:654] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:47.701197 23597 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 40964817cf2a4712be42f40e37639bc9 (table default.Table_2e5b16fa5b3b4be3af4d6843606d91f6 [id=362af6ee46a740b28b0fa3e9ae3391fd], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 40964817cf2a4712be42f40e37639bc9 consensus state has no leader
I20240502 08:16:47.704308 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0882b7509a74475780eb700818aab398 with delete_type TABLET_DATA_DELETED (replaced by tablet eb96ad2289754e428a1c109b2d59c53b) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:47.705027 25012 tablet_replica.cc:330] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:47.705113 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 71aa916247c449fea718b35869a6df20 with delete_type TABLET_DATA_DELETED (replaced by tablet 0501575a84ec454780f48574bf3155fe) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:47.705320 25012 raft_consensus.cc:2229] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:47.705621 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 2411259b872e4be4b8eb8832b36094c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:46 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:47.705919 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet bb4b4a4d6a364de8af3b54d99a0ae138 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:47.706221 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet 27346a7eae6a488fa3415646262a4b88 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:47.706444 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bfb532c9d4c41afb7acaa4831841306 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:36 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:47.706655 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet a3043c5ed98c401b82c6e967ca3f3613 with delete_type TABLET_DATA_DELETED (replaced by tablet 0281c8123eab4c48bddc377f5695c6e0) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:47.706809 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet bcfacc302ff04178bfffc48111cd7a3f with delete_type TABLET_DATA_DELETED (replaced by tablet b0cf1a6de3d649b984c50f45393ec0f1) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:47.706861 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 643ce929e74c499da835fbe9b63b8880 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:47.707214 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 48d96e42813f4169969dd5931e169923 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:47.707304 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1736b496354e2dadd844242d3867f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 6746c25111d04f20a6f30ce0838d71f5) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:47.706221 25012 raft_consensus.cc:2258] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:47.707664 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet ba3aa2621fdb43fd8299f81af486ef1e with delete_type TABLET_DATA_DELETED (replaced by tablet e20e126b1c9546e4a194bf9842f444a5) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:47.708289 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 202fa612a085450ca9c1225434cb7fe4 with delete_type TABLET_DATA_DELETED (replaced by tablet d5c82820278b4a0c8f7d586f87104bf0) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:47.708683 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 4baa2bc2752c450fa212471c150cf098 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:47.709170 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet d36e7cd33f9a4edabe690e75602590b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:47.709561 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 25c8a6aaabe2413d8ce88f54ce45a4c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 1f42627a877c4f83a75320f1b5ab539f) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:47.709751 25012 ts_tablet_manager.cc:1875] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:47.732618 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet e686db13bb7541c69ae88f0fb7e27061 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:47.733417 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c072eb45e1b41eeb5aafa6d5a786944 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:47.733896 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e5de793e74247dd9e73b7b1d58027d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:47.734346 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet d9fcc7accb1f45ad9cf55b8f64d21b91 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:47.736733 23723 leader_election.cc:341] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:47.737421 23723 leader_election.cc:341] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:47.738837 23723 leader_election.cc:341] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:47.738976 23723 leader_election.cc:302] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
W20240502 08:16:47.739167 23723 raft_consensus.cc:2670] T 4a57ed188315493198e95cefb6907ab6 P a297a6c7c2504ca68e87b24f6691ffb2: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:16:47.740350 23858 leader_election.cc:302] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:47.740835 24982 raft_consensus.cc:2726] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:47.741348 23858 leader_election.cc:341] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 643ce929e74c499da835fbe9b63b8880
I20240502 08:16:47.741534 23858 leader_election.cc:302] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:47.741904 24982 raft_consensus.cc:2726] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:47.742338 23858 leader_election.cc:302] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:47.743209 24982 raft_consensus.cc:3037] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:47.743696 24173 tablet_bootstrap.cc:492] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:47.743836 24173 ts_tablet_manager.cc:1367] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.196s user 0.040s sys 0.044s
I20240502 08:16:47.744568 24173 raft_consensus.cc:348] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.744791 24173 raft_consensus.cc:374] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.744839 24173 raft_consensus.cc:729] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.745121 24173 consensus_queue.cc:260] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:47.746374 23858 leader_election.cc:341] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 27346a7eae6a488fa3415646262a4b88
I20240502 08:16:47.746542 23858 leader_election.cc:302] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:47.746793 24790 raft_consensus.cc:2726] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:47.747227 23858 leader_election.cc:341] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 2bfb532c9d4c41afb7acaa4831841306
I20240502 08:16:47.747768 24173 ts_tablet_manager.cc:1398] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:16:47.748049 24173 tablet_bootstrap.cc:492] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:47.750032 24173 tablet_bootstrap.cc:654] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.763513 24173 tablet_bootstrap.cc:492] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:47.763691 24173 ts_tablet_manager.cc:1367] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20240502 08:16:47.765081 24173 raft_consensus.cc:348] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.765300 24173 raft_consensus.cc:374] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.765357 24173 raft_consensus.cc:729] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.765586 24173 consensus_queue.cc:260] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.767504 24173 ts_tablet_manager.cc:1398] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:47.767774 24173 tablet_bootstrap.cc:492] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:47.769152 24173 tablet_bootstrap.cc:654] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.774874 24173 tablet_bootstrap.cc:492] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:47.775030 24173 ts_tablet_manager.cc:1367] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 08:16:47.775744 24173 raft_consensus.cc:348] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.775949 24173 raft_consensus.cc:374] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.776002 24173 raft_consensus.cc:729] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.777851 24173 consensus_queue.cc:260] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.779996 24173 ts_tablet_manager.cc:1398] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:16:47.780288 24173 tablet_bootstrap.cc:492] T 96fcb895c8f449deb0190de75af18c3c P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:47.782829 24173 tablet_bootstrap.cc:654] T 96fcb895c8f449deb0190de75af18c3c P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.818591 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 13cbab4352bb44f389d70d4379e92457
I20240502 08:16:47.820483 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 21d2db2158d4447f92eedc5780b46056
I20240502 08:16:47.821297 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a2055a6e13d8495b83033c1f6596b54f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.822001 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2055a6e13d8495b83033c1f6596b54f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.823365 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 13cbab4352bb44f389d70d4379e92457 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.824525 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d2db2158d4447f92eedc5780b46056 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.819681 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9522d368c4c84f25bce3eb822a9bcad8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.826910 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9522d368c4c84f25bce3eb822a9bcad8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.836072 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 0ae92276246c4d39b194d7e9b868fc02
I20240502 08:16:47.837059 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a1f81503886a490bad183964519aee17" } new_table_name: "default.Table_3c059ec418cd4c6dbf7bf93c0856437f" modify_external_catalogs: true
I20240502 08:16:47.839625 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ae92276246c4d39b194d7e9b868fc02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.839617 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet 13cbab4352bb44f389d70d4379e92457 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:47.839877 24973 raft_consensus.cc:482] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:16:47.842445 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 103610us
I20240502 08:16:47.842847 24774 raft_consensus.cc:482] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.842957 24774 raft_consensus.cc:504] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.840020 24973 raft_consensus.cc:504] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:47.843400 23723 consensus_peers.cc:485] T 27093862fef54c5c85d810ebeff9819c P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:47.844286 24891 consensus_queue.cc:1035] T f4b291bdb81a4efbac7f6691e5577bbe P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:47.846575 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ae92276246c4d39b194d7e9b868fc02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:47.846884 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d2db2158d4447f92eedc5780b46056 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:47.853480 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b4905971ad3f4250a6b2e8f12c6ecd96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:47.853255 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_346dec12764b4564b56f28b373f24b0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.855878 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_346dec12764b4564b56f28b373f24b0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:47.855769 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4905971ad3f4250a6b2e8f12c6ecd96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.858969 24973 leader_election.cc:288] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:47.859987 24774 leader_election.cc:288] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:47.860991 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 13cbab4352bb44f389d70d4379e92457 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13cbab4352bb44f389d70d4379e92457 already in progress: creating tablet
W20240502 08:16:47.861330 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 0ae92276246c4d39b194d7e9b868fc02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ae92276246c4d39b194d7e9b868fc02 already in progress: creating tablet
W20240502 08:16:47.861734 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 21d2db2158d4447f92eedc5780b46056 because the tablet was not found. No further retry: Not found: Tablet not found: 21d2db2158d4447f92eedc5780b46056
W20240502 08:16:47.868566 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 598cba0f8b9f4c8dae9e34872c4212fa has no consensus state
I20240502 08:16:47.868992 24174 tablet_bootstrap.cc:492] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:47.869140 24174 ts_tablet_manager.cc:1367] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.178s user 0.023s sys 0.060s
I20240502 08:16:47.869071 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_f17a7a3f2d4547799bdbea376e3771b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.869707 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f17a7a3f2d4547799bdbea376e3771b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.872610 24774 raft_consensus.cc:482] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.872404 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_31208fdb293844fc838187320c87cdf9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:47.872792 24774 raft_consensus.cc:504] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:47.873142 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31208fdb293844fc838187320c87cdf9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.873600 24774 leader_election.cc:288] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:47.875454 24174 raft_consensus.cc:348] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.875631 24174 raft_consensus.cc:374] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.875686 24174 raft_consensus.cc:729] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.875878 24174 consensus_queue.cc:260] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.876092 23605 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_f15d0ef92f374df0b6e9efb21261adfb [id=18ab2c0716464d82a0e33bd99bd0735d] alter complete (version 2)
I20240502 08:16:47.876336 24174 ts_tablet_manager.cc:1398] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:47.876596 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_cc6ea8499b734e0790c5964d4b8bd631"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.877213 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc6ea8499b734e0790c5964d4b8bd631 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.879647 24174 tablet_bootstrap.cc:492] T aacdf506221044ffada0fec0c7d42d0b P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:47.881898 24174 tablet_bootstrap.cc:654] T aacdf506221044ffada0fec0c7d42d0b P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.897547 24181 ts_tablet_manager.cc:1888] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:47.897819 24181 log.cc:1196] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/9c072eb45e1b41eeb5aafa6d5a786944
I20240502 08:16:47.898218 24181 ts_tablet_manager.cc:1909] T 9c072eb45e1b41eeb5aafa6d5a786944 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:47.883627 23663 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 21d2db2158d4447f92eedc5780b46056
I20240502 08:16:47.899709 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d2db2158d4447f92eedc5780b46056 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:47.900487 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 21d2db2158d4447f92eedc5780b46056 because the tablet was not found. No further retry: Not found: Tablet not found: 21d2db2158d4447f92eedc5780b46056
I20240502 08:16:47.902261 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d2db2158d4447f92eedc5780b46056 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.903712 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 812c1fbec6504df19cc7e51be4d0a406 (DEFAULT_TABLE table=default.Table_774c13246f3442b28645d6c755c67800 [id=474c484491824b0194ccd5892d06c2f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.904230 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812c1fbec6504df19cc7e51be4d0a406. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.913630 24944 raft_consensus.cc:482] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.913815 24944 raft_consensus.cc:504] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.914544 24944 leader_election.cc:288] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:47.915438 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1e208733844829b13222e00a432439" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:47.916091 23993 leader_election.cc:341] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:47.916813 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 812c1fbec6504df19cc7e51be4d0a406 (DEFAULT_TABLE table=default.Table_774c13246f3442b28645d6c755c67800 [id=474c484491824b0194ccd5892d06c2f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.917325 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812c1fbec6504df19cc7e51be4d0a406. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.919205 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 40964817cf2a4712be42f40e37639bc9
I20240502 08:16:47.919731 23608 catalog_manager.cc:6554] ReplaceTablet: tablet 40964817cf2a4712be42f40e37639bc9 deleted and replaced by tablet 506644a1ead54a1faeaa8583ef96b449
I20240502 08:16:47.919986 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d03c41e82d684920af0c093caa7cd1e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:47.920171 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9e3aabcb11df4be58f87a1f242912dec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.920568 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d03c41e82d684920af0c093caa7cd1e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:47.920779 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e3aabcb11df4be58f87a1f242912dec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:47.921607 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 4a0684db8dd947d79343afa5a2090614
I20240502 08:16:47.922163 23610 master_service.cc:946] ReplaceTablet: received request to replace tablet 28a947b7e7484120b3f4069f6eef5520 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:47.922698 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet a4d151770ee44130adfd822518b9b017
I20240502 08:16:47.923503 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_acb191a2bab44ddd84855f88d5b7e073" } new_table_name: "default.Table_c5355707de4c493ca4a53084390dda58" modify_external_catalogs: true
I20240502 08:16:47.925069 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 40964817cf2a4712be42f40e37639bc9 with delete_type TABLET_DATA_DELETED (replaced by tablet 506644a1ead54a1faeaa8583ef96b449) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.925263 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a0684db8dd947d79343afa5a2090614 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.925420 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet a4d151770ee44130adfd822518b9b017 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:47.925956 24774 raft_consensus.cc:482] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.926088 24774 raft_consensus.cc:504] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.926746 24774 leader_election.cc:288] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:47.931640 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 40964817cf2a4712be42f40e37639bc9 with delete_type TABLET_DATA_DELETED (replaced by tablet 506644a1ead54a1faeaa8583ef96b449) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:47.932698 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a0684db8dd947d79343afa5a2090614 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:47.933959 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet a4d151770ee44130adfd822518b9b017 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:47.934757 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 40964817cf2a4712be42f40e37639bc9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40964817cf2a4712be42f40e37639bc9 already in progress: creating tablet
W20240502 08:16:47.935245 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 4a0684db8dd947d79343afa5a2090614 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a0684db8dd947d79343afa5a2090614 already in progress: creating tablet
I20240502 08:16:47.935282 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a92dfc1659cf4b24a49358808fd9663b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:47.935999 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a92dfc1659cf4b24a49358808fd9663b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:47.941212 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet a4d151770ee44130adfd822518b9b017 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4d151770ee44130adfd822518b9b017 already in progress: creating tablet
I20240502 08:16:47.942531 23601 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_aded763332144f979b418931264d1eb5 [id=0035c98028aa49f7a7dcea576bb23351] alter complete (version 1)
I20240502 08:16:47.943065 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9771b72641674c9497872784242b435d" } new_table_name: "default.Table_7c74c34063104d54b3ab77c878892069" modify_external_catalogs: true
I20240502 08:16:47.946180 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_44bdfbf3075343e0a2ea5b79c0ca93e5" } new_table_name: "default.Table_e5e87b9f9dc64e9e8324057fc970d2fa" modify_external_catalogs: true
I20240502 08:16:47.947571 24774 raft_consensus.cc:482] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.947750 24774 raft_consensus.cc:504] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.948576 24774 leader_election.cc:288] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:47.949476 23753 tablet_service.cc:1467] Processing CreateTablet for tablet c84b27ed8cf7429ca720e4cb8d396342 (DEFAULT_TABLE table=default.Table_52e9e35e99234042ae5e823f448c5cb7 [id=02c97cde17304fe9becb5d5c6f7d09f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.950026 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c84b27ed8cf7429ca720e4cb8d396342. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.797245 24790 raft_consensus.cc:482] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.954048 24790 raft_consensus.cc:504] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.954818 24790 leader_election.cc:288] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:47.798112 24982 raft_consensus.cc:2726] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:47.843180 24175 tablet_bootstrap.cc:492] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:47.955209 24175 ts_tablet_manager.cc:1367] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.272s user 0.021s sys 0.047s
I20240502 08:16:47.956207 24175 raft_consensus.cc:348] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.956408 24175 raft_consensus.cc:374] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.956459 24175 raft_consensus.cc:729] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.956648 24175 consensus_queue.cc:260] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.957273 24175 ts_tablet_manager.cc:1398] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:47.889753 25012 ts_tablet_manager.cc:1888] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:47.958083 25012 log.cc:1196] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/0882b7509a74475780eb700818aab398
I20240502 08:16:47.958423 25012 ts_tablet_manager.cc:1909] T 0882b7509a74475780eb700818aab398 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:47.958912 25012 tablet_replica.cc:330] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:47.959121 25012 raft_consensus.cc:2229] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:47.959133 24056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b1fa17dad143f0a7f1751e651eb81a" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:47.959261 25012 raft_consensus.cc:2258] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:47.959388 24056 raft_consensus.cc:2454] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:47.959839 25012 ts_tablet_manager.cc:1875] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:47.960139 23861 leader_election.cc:302] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:47.960393 24982 raft_consensus.cc:2781] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:47.960469 24982 raft_consensus.cc:482] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:47.960532 24982 raft_consensus.cc:3037] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:47.961823 24175 tablet_bootstrap.cc:492] T aacdf506221044ffada0fec0c7d42d0b P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:47.966624 24175 tablet_bootstrap.cc:654] T aacdf506221044ffada0fec0c7d42d0b P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.972033 23858 leader_election.cc:302] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:47.972311 24790 raft_consensus.cc:2726] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:47.972610 23858 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 225378us
W20240502 08:16:47.972841 23858 leader_election.cc:341] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 71aa916247c449fea718b35869a6df20
I20240502 08:16:47.972957 23858 leader_election.cc:302] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:47.973117 23858 raft_consensus.cc:2670] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240502 08:16:47.973374 23858 leader_election.cc:341] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: bcfacc302ff04178bfffc48111cd7a3f
I20240502 08:16:47.973479 23858 leader_election.cc:302] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:47.973707 24790 raft_consensus.cc:2726] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:47.974880 24790 raft_consensus.cc:482] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.974993 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 0882b7509a74475780eb700818aab398 (table default.Table_bc1e9d346aa846f8abca7504185f7cdb [id=8d27878c6c8f4234908a2c78cb8bbd29]) successfully deleted
I20240502 08:16:47.975055 24790 raft_consensus.cc:504] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.975795 24790 leader_election.cc:288] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:47.901247 25018 raft_consensus.cc:482] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.902920 25024 raft_consensus.cc:482] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.915218 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1e208733844829b13222e00a432439" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:47.979387 25024 raft_consensus.cc:504] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:47.979837 23996 leader_election.cc:341] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:47.980003 23996 leader_election.cc:302] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:47.980059 25024 leader_election.cc:288] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:47.980396 24944 raft_consensus.cc:2726] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:47.980334 23767 tablet_service.cc:1467] Processing CreateTablet for tablet c2144cbe36904d0a9ed9ada369186d02 (DEFAULT_TABLE table=default.Table_22be413d6296475eb6266315d70808d9 [id=8d9a421debf74d89beacd1b6b147d735]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.979363 25018 raft_consensus.cc:504] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.980773 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2144cbe36904d0a9ed9ada369186d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.980921 25018 leader_election.cc:288] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:47.982003 23861 leader_election.cc:341] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: a3043c5ed98c401b82c6e967ca3f3613
I20240502 08:16:47.982162 24175 tablet_bootstrap.cc:492] T aacdf506221044ffada0fec0c7d42d0b P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
W20240502 08:16:47.982254 23861 leader_election.cc:341] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 4f1736b496354e2dadd844242d3867f8
I20240502 08:16:47.982287 24175 ts_tablet_manager.cc:1367] T aacdf506221044ffada0fec0c7d42d0b P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240502 08:16:47.982949 24175 raft_consensus.cc:348] T aacdf506221044ffada0fec0c7d42d0b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.983131 24175 raft_consensus.cc:374] T aacdf506221044ffada0fec0c7d42d0b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.983182 24175 raft_consensus.cc:729] T aacdf506221044ffada0fec0c7d42d0b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.983325 24175 consensus_queue.cc:260] T aacdf506221044ffada0fec0c7d42d0b P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:47.983769 24175 ts_tablet_manager.cc:1398] T aacdf506221044ffada0fec0c7d42d0b P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:16:47.984040 24175 tablet_bootstrap.cc:492] T 27093862fef54c5c85d810ebeff9819c P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:47.985613 24175 tablet_bootstrap.cc:654] T 27093862fef54c5c85d810ebeff9819c P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.995468 24181 tablet_replica.cc:330] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:47.995748 24181 raft_consensus.cc:2229] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:47.995946 24181 raft_consensus.cc:2258] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:47.996634 24181 ts_tablet_manager.cc:1875] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:47.997946 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 9c072eb45e1b41eeb5aafa6d5a786944 (table default.Table_20c5ace604d64b6cabead6037cdc04e1 [id=2e5831a5849049c8bd445667b01b1bd8]) successfully deleted
I20240502 08:16:47.998684 24056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c072eb45e1b41eeb5aafa6d5a786944" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:48.000116 23861 leader_election.cc:341] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: Tablet not RUNNING: SHUTDOWN
I20240502 08:16:48.003494 24982 raft_consensus.cc:504] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.004806 24982 leader_election.cc:288] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.005281 24056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b1fa17dad143f0a7f1751e651eb81a" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:48.005442 24056 raft_consensus.cc:3037] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.006597 23604 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 642744194e22461f8316a78689fd99ff (table default.Table_8bed20f633e348679b0087687298cc42 [id=18ab2c0716464d82a0e33bd99bd0735d]). Expected version 3 got 2
I20240502 08:16:48.006763 24174 tablet_bootstrap.cc:492] T aacdf506221044ffada0fec0c7d42d0b P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:48.006850 23604 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 77c49da1e0f0435ba241a6a66526e790 (table default.Table_8a9cdf1f53b0494694035cd549988bf0 [id=0de18f9c7e194e698b4f37603bfd4780]). Expected version 2 got 0
I20240502 08:16:48.006889 24174 ts_tablet_manager.cc:1367] T aacdf506221044ffada0fec0c7d42d0b P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.127s user 0.018s sys 0.026s
I20240502 08:16:48.007547 23604 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet b811b81ddf2b4dc9b5e5f80331bff294 (table default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418]). Expected version 2 got 0
I20240502 08:16:48.007627 24174 raft_consensus.cc:348] T aacdf506221044ffada0fec0c7d42d0b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.007800 24174 raft_consensus.cc:374] T aacdf506221044ffada0fec0c7d42d0b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.007844 24174 raft_consensus.cc:729] T aacdf506221044ffada0fec0c7d42d0b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.008121 24174 consensus_queue.cc:260] T aacdf506221044ffada0fec0c7d42d0b P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.008671 24174 ts_tablet_manager.cc:1398] T aacdf506221044ffada0fec0c7d42d0b P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:48.010023 23601 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 77c49da1e0f0435ba241a6a66526e790 (table default.Table_8a9cdf1f53b0494694035cd549988bf0 [id=0de18f9c7e194e698b4f37603bfd4780]). Expected version 2 got 0
I20240502 08:16:48.010965 24174 tablet_bootstrap.cc:492] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:48.011477 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 28a947b7e7484120b3f4069f6eef5520
I20240502 08:16:48.011921 23610 catalog_manager.cc:6554] ReplaceTablet: tablet 28a947b7e7484120b3f4069f6eef5520 deleted and replaced by tablet 336f0eb5085546dba763e0d5c5dd5324
I20240502 08:16:48.012631 24174 tablet_bootstrap.cc:654] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.012995 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 28a947b7e7484120b3f4069f6eef5520 with delete_type TABLET_DATA_DELETED (replaced by tablet 336f0eb5085546dba763e0d5c5dd5324) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.022123 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 28a947b7e7484120b3f4069f6eef5520 with delete_type TABLET_DATA_DELETED (replaced by tablet 336f0eb5085546dba763e0d5c5dd5324) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:48.023638 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 28a947b7e7484120b3f4069f6eef5520 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28a947b7e7484120b3f4069f6eef5520 already in progress: creating tablet
I20240502 08:16:48.026392 24944 raft_consensus.cc:482] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.026536 24944 raft_consensus.cc:504] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.027281 24944 leader_election.cc:288] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:48.028213 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e746013fde4de4862f213792e235ed" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:48.028685 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e746013fde4de4862f213792e235ed" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:48.029278 23996 leader_election.cc:341] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:48.029678 23993 leader_election.cc:341] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:48.029778 23993 leader_election.cc:302] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:48.030155 24944 raft_consensus.cc:2726] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:48.031102 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_890e6db207d54accbbd91ba88c78095e" } modify_external_catalogs: true
I20240502 08:16:48.031343 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_890e6db207d54accbbd91ba88c78095e" } modify_external_catalogs: true
I20240502 08:16:48.032542 24039 tablet_service.cc:1467] Processing CreateTablet for tablet c84b27ed8cf7429ca720e4cb8d396342 (DEFAULT_TABLE table=default.Table_52e9e35e99234042ae5e823f448c5cb7 [id=02c97cde17304fe9becb5d5c6f7d09f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.032944 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f8c00933a68d47e7abd51a6d9b1b27af" } modify_external_catalogs: true
I20240502 08:16:48.033298 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f8c00933a68d47e7abd51a6d9b1b27af" } modify_external_catalogs: true
I20240502 08:16:48.035678 24982 raft_consensus.cc:482] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.035926 24982 raft_consensus.cc:504] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.036350 24944 raft_consensus.cc:482] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.036468 24944 raft_consensus.cc:504] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.037050 24944 leader_election.cc:288] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:48.037122 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c17e1be6ef294769a01981f4d361be21" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:48.037273 24059 raft_consensus.cc:2454] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:48.037592 24982 leader_election.cc:288] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.037719 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c17e1be6ef294769a01981f4d361be21" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:48.037926 23786 raft_consensus.cc:2454] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:48.038201 23861 leader_election.cc:302] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
W20240502 08:16:48.038462 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 657bcfa3d93e4809a78c7ba5692d9b90 consensus state has no leader
I20240502 08:16:48.038697 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c17e1be6ef294769a01981f4d361be21" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:48.038906 23924 raft_consensus.cc:2454] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:48.039374 23993 leader_election.cc:302] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:48.040149 24944 raft_consensus.cc:2781] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.040243 24944 raft_consensus.cc:482] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.040305 24944 raft_consensus.cc:3037] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.041222 24982 raft_consensus.cc:2781] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.041397 24982 raft_consensus.cc:482] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.041519 24982 raft_consensus.cc:3037] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:48.041965 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 198272us
I20240502 08:16:48.045758 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 46e577132e5249b28426bfe38a28635a (DEFAULT_TABLE table=default.Table_7b3b1098fbed4d2792dadff76cbc948b [id=ce3d0ff6ef76431b85d0a98efd113b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.046284 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e577132e5249b28426bfe38a28635a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.048110 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e17483d2fa3440d8acafbd48cb0f8c51" } modify_external_catalogs: true
I20240502 08:16:48.048380 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e17483d2fa3440d8acafbd48cb0f8c51" } modify_external_catalogs: true
I20240502 08:16:48.051249 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5520c58206f4499bac98e73fb4122056"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:48.051788 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_282e8b7fa027438c9e38f997527d7406"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.052112 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5520c58206f4499bac98e73fb4122056 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:48.052276 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_282e8b7fa027438c9e38f997527d7406 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.053982 25018 raft_consensus.cc:482] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.054226 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 13cbab4352bb44f389d70d4379e92457 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.054208 25018 raft_consensus.cc:504] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.054746 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d2db2158d4447f92eedc5780b46056 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.055172 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ae92276246c4d39b194d7e9b868fc02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.055558 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d2db2158d4447f92eedc5780b46056 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:48.055896 23861 leader_election.cc:341] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: e686db13bb7541c69ae88f0fb7e27061
I20240502 08:16:48.056150 25018 leader_election.cc:288] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.056205 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 812c1fbec6504df19cc7e51be4d0a406 (DEFAULT_TABLE table=default.Table_774c13246f3442b28645d6c755c67800 [id=474c484491824b0194ccd5892d06c2f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.056771 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 40964817cf2a4712be42f40e37639bc9 with delete_type TABLET_DATA_DELETED (replaced by tablet 506644a1ead54a1faeaa8583ef96b449) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.057148 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a0684db8dd947d79343afa5a2090614 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.059522 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 788256e092b7475d8fca0904bf615cef (DEFAULT_TABLE table=default.Table_91d4de88c52a461d8d4006baab592636 [id=bc67017e4b6046d5b03a2d644db4dc72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.060218 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 788256e092b7475d8fca0904bf615cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.067864 24032 tablet_service.cc:1467] Processing CreateTablet for tablet c2144cbe36904d0a9ed9ada369186d02 (DEFAULT_TABLE table=default.Table_22be413d6296475eb6266315d70808d9 [id=8d9a421debf74d89beacd1b6b147d735]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.068238 24175 tablet_bootstrap.cc:492] T 27093862fef54c5c85d810ebeff9819c P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:48.068248 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 46e577132e5249b28426bfe38a28635a (DEFAULT_TABLE table=default.Table_7b3b1098fbed4d2792dadff76cbc948b [id=ce3d0ff6ef76431b85d0a98efd113b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.068409 24175 ts_tablet_manager.cc:1367] T 27093862fef54c5c85d810ebeff9819c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.028s
I20240502 08:16:48.068811 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c49da1e0f0435ba241a6a66526e790" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:48.068864 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2126bf2d4ab44628b52481dc2ec3a99" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:48.069010 24059 raft_consensus.cc:2454] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:48.069034 24057 raft_consensus.cc:2454] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:48.069087 24175 raft_consensus.cc:348] T 27093862fef54c5c85d810ebeff9819c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.069257 24175 raft_consensus.cc:374] T 27093862fef54c5c85d810ebeff9819c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.069284 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcc7169796e4ee0910f5e7084849012" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:48.069474 24058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9402cf8abc4b4d893d42f9a3a99016" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:48.069483 23723 leader_election.cc:302] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:48.069636 24058 raft_consensus.cc:2454] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:48.069813 23723 leader_election.cc:302] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:48.069453 24055 raft_consensus.cc:2454] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
W20240502 08:16:48.070031 23723 leader_election.cc:341] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 9c072eb45e1b41eeb5aafa6d5a786944
I20240502 08:16:48.070202 24891 raft_consensus.cc:2781] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.070261 24891 raft_consensus.cc:482] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.070324 24891 raft_consensus.cc:3037] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.071511 25021 raft_consensus.cc:2781] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.071702 25021 raft_consensus.cc:482] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.071853 25021 raft_consensus.cc:3037] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.072965 23723 leader_election.cc:302] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:48.073386 23723 leader_election.cc:302] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:48.073661 24774 raft_consensus.cc:2781] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.073740 24774 raft_consensus.cc:482] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.073807 24774 raft_consensus.cc:3037] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.074065 24973 consensus_queue.cc:1035] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:48.074525 24973 raft_consensus.cc:2781] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.074590 24973 raft_consensus.cc:482] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.074651 24973 raft_consensus.cc:3037] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.075261 24173 tablet_bootstrap.cc:492] T 96fcb895c8f449deb0190de75af18c3c P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:48.075408 24173 ts_tablet_manager.cc:1367] T 96fcb895c8f449deb0190de75af18c3c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.295s user 0.028s sys 0.065s
I20240502 08:16:48.069305 24175 raft_consensus.cc:729] T 27093862fef54c5c85d810ebeff9819c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.076136 24175 consensus_queue.cc:260] T 27093862fef54c5c85d810ebeff9819c P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.076247 24173 raft_consensus.cc:348] T 96fcb895c8f449deb0190de75af18c3c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.076514 24173 raft_consensus.cc:374] T 96fcb895c8f449deb0190de75af18c3c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.076581 24173 raft_consensus.cc:729] T 96fcb895c8f449deb0190de75af18c3c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.076963 24175 ts_tablet_manager.cc:1398] T 27093862fef54c5c85d810ebeff9819c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 08:16:48.077271 24175 tablet_bootstrap.cc:492] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:48.077296 24173 consensus_queue.cc:260] T 96fcb895c8f449deb0190de75af18c3c P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.078833 24175 tablet_bootstrap.cc:654] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.078532 24173 ts_tablet_manager.cc:1398] T 96fcb895c8f449deb0190de75af18c3c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:16:48.079243 24173 tablet_bootstrap.cc:492] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:48.081449 24173 tablet_bootstrap.cc:654] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.083315 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cc6ea8499b734e0790c5964d4b8bd631" } new_table_name: "default.Table_722b7dc3bd6f4bdcb2ffc44d2b77e0a1" modify_external_catalogs: true
I20240502 08:16:48.087822 24175 tablet_bootstrap.cc:492] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:48.087963 24175 ts_tablet_manager.cc:1367] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 08:16:48.088671 24175 raft_consensus.cc:348] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.088871 24175 raft_consensus.cc:374] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.088925 24175 raft_consensus.cc:729] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
W20240502 08:16:48.089231 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 55f3f09fc1f94c00a47b9087b98e221d consensus state has no leader
I20240502 08:16:48.089161 24175 consensus_queue.cc:260] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.090298 24173 tablet_bootstrap.cc:492] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:48.090421 24173 ts_tablet_manager.cc:1367] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240502 08:16:48.090735 24982 raft_consensus.cc:504] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.091480 24982 leader_election.cc:288] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.092217 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c17e1be6ef294769a01981f4d361be21" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:48.092391 24175 ts_tablet_manager.cc:1398] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:48.092792 24175 tablet_bootstrap.cc:492] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:48.095921 24056 raft_consensus.cc:2454] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:48.096318 24891 raft_consensus.cc:504] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.097134 24891 leader_election.cc:288] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.097477 24175 tablet_bootstrap.cc:654] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.097927 24173 raft_consensus.cc:348] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.098119 24173 raft_consensus.cc:374] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.098162 24173 raft_consensus.cc:729] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.099414 24173 consensus_queue.cc:260] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.100363 24173 ts_tablet_manager.cc:1398] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 08:16:48.100620 24173 tablet_bootstrap.cc:492] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:48.102782 23602 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 27093862fef54c5c85d810ebeff9819c (table default.Table_aded763332144f979b418931264d1eb5 [id=0035c98028aa49f7a7dcea576bb23351]). Expected version 1 got 0
I20240502 08:16:48.103304 23861 leader_election.cc:302] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:48.103415 24056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c49da1e0f0435ba241a6a66526e790" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:48.103597 24056 raft_consensus.cc:3037] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.103626 25018 raft_consensus.cc:2781] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:48.104220 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2144cbe36904d0a9ed9ada369186d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.104491 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e577132e5249b28426bfe38a28635a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.105186 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812c1fbec6504df19cc7e51be4d0a406. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.103931 25018 raft_consensus.cc:686] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:16:48.106293 24982 raft_consensus.cc:482] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.106401 24982 raft_consensus.cc:504] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.107008 24982 leader_election.cc:288] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.107610 24058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5768cb928e244d2914f10f4eda3c1cd" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:48.107883 24058 raft_consensus.cc:2379] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:16:48.107847 25018 consensus_queue.cc:237] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.110018 25012 ts_tablet_manager.cc:1888] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:48.110260 25012 log.cc:1196] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/71aa916247c449fea718b35869a6df20
I20240502 08:16:48.110689 24174 tablet_bootstrap.cc:492] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:48.110823 24174 ts_tablet_manager.cc:1367] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.100s user 0.004s sys 0.035s
I20240502 08:16:48.111300 25012 ts_tablet_manager.cc:1909] T 71aa916247c449fea718b35869a6df20 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:48.111498 24174 raft_consensus.cc:348] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.111688 24174 raft_consensus.cc:374] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.111732 24174 raft_consensus.cc:729] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.111925 24174 consensus_queue.cc:260] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.112355 24174 ts_tablet_manager.cc:1398] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 08:16:48.112588 24174 tablet_bootstrap.cc:492] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:48.114512 24174 tablet_bootstrap.cc:654] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.115811 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet a4d151770ee44130adfd822518b9b017 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.118228 23899 tablet_service.cc:1467] Processing CreateTablet for tablet c84b27ed8cf7429ca720e4cb8d396342 (DEFAULT_TABLE table=default.Table_52e9e35e99234042ae5e823f448c5cb7 [id=02c97cde17304fe9becb5d5c6f7d09f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.118844 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c84b27ed8cf7429ca720e4cb8d396342. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.122994 24174 tablet_bootstrap.cc:492] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:48.123124 24174 ts_tablet_manager.cc:1367] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20240502 08:16:48.123801 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 28a947b7e7484120b3f4069f6eef5520 with delete_type TABLET_DATA_DELETED (replaced by tablet 336f0eb5085546dba763e0d5c5dd5324) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.124516 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_59270ebb70304dec9598e44f5d3bf506" } modify_external_catalogs: true
I20240502 08:16:48.124708 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_59270ebb70304dec9598e44f5d3bf506" } modify_external_catalogs: true
I20240502 08:16:48.123769 24174 raft_consensus.cc:348] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.127745 24174 raft_consensus.cc:374] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.127815 24174 raft_consensus.cc:729] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.128405 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a57ed188315493198e95cefb6907ab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.128711 24173 tablet_bootstrap.cc:654] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.130160 23607 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_81ee0423dbbe4ff899c42249227cea34 [id=8521bfc13ef040a78cfb2cc4501b7f04] alter complete (version 5)
I20240502 08:16:48.128888 24174 consensus_queue.cc:260] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.132213 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c9c50670737d40c58a305ff256334951"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:48.132679 24174 ts_tablet_manager.cc:1398] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
W20240502 08:16:48.132694 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9c50670737d40c58a305ff256334951 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.132963 24174 tablet_bootstrap.cc:492] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:48.133849 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 788256e092b7475d8fca0904bf615cef (DEFAULT_TABLE table=default.Table_91d4de88c52a461d8d4006baab592636 [id=bc67017e4b6046d5b03a2d644db4dc72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.136421 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a1e208733844829b13222e00a432439 with delete_type TABLET_DATA_DELETED (replaced by tablet 8910bc67c6234a538c8160e7c020653c) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.136637 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet c1e746013fde4de4862f213792e235ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.136896 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bcc7169796e4ee0910f5e7084849012 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.137107 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet aacdf506221044ffada0fec0c7d42d0b with delete_type TABLET_DATA_DELETED (replaced by tablet 90d4e2f23c9649a4b1dfb6a179ca7be5) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.137285 24026 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a97427e819b4826b1ed89e6d021c8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.137490 24021 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4a31e5aa834adeb9f59267247c6980 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.138202 24173 tablet_bootstrap.cc:492] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:48.138324 24173 ts_tablet_manager.cc:1367] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20240502 08:16:48.138367 24972 tablet.cc:1722] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:48.138624 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 8e2c611d7af6428db289a7e2fbd586d4
W20240502 08:16:48.139087 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e9133e69e1724a26bc3650ea658ca349 consensus state has no leader
I20240502 08:16:48.139294 24174 tablet_bootstrap.cc:654] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.141175 24173 raft_consensus.cc:348] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.141350 24173 raft_consensus.cc:374] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.142774 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 788256e092b7475d8fca0904bf615cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.141391 24173 raft_consensus.cc:729] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.144847 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e2c611d7af6428db289a7e2fbd586d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:48.145124 24173 consensus_queue.cc:260] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:48.145771 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 8e2c611d7af6428db289a7e2fbd586d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e2c611d7af6428db289a7e2fbd586d4 already in progress: creating tablet
I20240502 08:16:48.146610 24021 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e2c611d7af6428db289a7e2fbd586d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.146787 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fc2d835b22c1417bad53bb6f66bde481"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.147382 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc2d835b22c1417bad53bb6f66bde481 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.147914 24173 ts_tablet_manager.cc:1398] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 08:16:48.148245 24173 tablet_bootstrap.cc:492] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:48.149051 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6e14e5ad32c946d286dc2416e1a16f61" } modify_external_catalogs: true
I20240502 08:16:48.149230 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6e14e5ad32c946d286dc2416e1a16f61" } modify_external_catalogs: true
I20240502 08:16:48.151888 24173 tablet_bootstrap.cc:654] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.152076 24970 tablet.cc:1722] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:48.158313 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 319fabf91458438ba5e40ab68b323055 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:48.158598 24180 tablet_replica.cc:330] T 319fabf91458438ba5e40ab68b323055 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:48.158797 24180 raft_consensus.cc:2229] T 319fabf91458438ba5e40ab68b323055 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:48.158975 24180 raft_consensus.cc:2258] T 319fabf91458438ba5e40ab68b323055 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:48.159579 24180 ts_tablet_manager.cc:1875] T 319fabf91458438ba5e40ab68b323055 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:48.160120 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c84b27ed8cf7429ca720e4cb8d396342. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.161916 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bcc7169796e4ee0910f5e7084849012 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:48.162356 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c072eb45e1b41eeb5aafa6d5a786944 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:48.163038 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet c17e1be6ef294769a01981f4d361be21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:48.164752 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet df9402cf8abc4b4d893d42f9a3a99016 with delete_type TABLET_DATA_DELETED (replaced by tablet fddac683eda14ec1a0140926380f2c07) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:48.168195 25029 raft_consensus.cc:482] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.168326 25029 raft_consensus.cc:504] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.168601 23610 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet dc6d60e7bfba4b3a9d73e5941355b165
I20240502 08:16:48.168797 24175 tablet_bootstrap.cc:492] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:48.168922 24175 ts_tablet_manager.cc:1367] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.017s
I20240502 08:16:48.168974 25029 leader_election.cc:288] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:48.169549 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacdf506221044ffada0fec0c7d42d0b" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:48.169754 23786 raft_consensus.cc:2454] T aacdf506221044ffada0fec0c7d42d0b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:48.169646 24175 raft_consensus.cc:348] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.169853 24175 raft_consensus.cc:374] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.169907 24175 raft_consensus.cc:729] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.169947 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 4c71f7398eab4c099cb45a55e50eee51 (DEFAULT_TABLE table=default.Table_f7042786757e4da189a33bb4c3190322 [id=aa63e2a9ef2b4c128f028a1d6a553ca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.170121 24175 consensus_queue.cc:260] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.170375 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacdf506221044ffada0fec0c7d42d0b" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:48.170568 23924 raft_consensus.cc:2454] T aacdf506221044ffada0fec0c7d42d0b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:48.170708 24175 ts_tablet_manager.cc:1398] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:48.171202 23993 leader_election.cc:302] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:48.171245 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a662f92a7fe845a990eb07796fc9d8a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.171744 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a662f92a7fe845a990eb07796fc9d8a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.172289 25029 raft_consensus.cc:2781] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.172371 25029 raft_consensus.cc:482] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.172428 25029 raft_consensus.cc:3037] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.172753 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e608dd4f99004bb795de5e05ea700e08
I20240502 08:16:48.173345 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_47fe484d653742f582516ba50ff30131"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.173828 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47fe484d653742f582516ba50ff30131 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.173801 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet c17e1be6ef294769a01981f4d361be21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.173818 24175 tablet_bootstrap.cc:492] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:48.177639 24175 tablet_bootstrap.cc:654] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.181391 24021 tablet_service.cc:1514] Processing DeleteTablet for tablet e608dd4f99004bb795de5e05ea700e08 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.181649 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 4eed19c1aa7c455eb23bad90c3df0b46 (DEFAULT_TABLE table=default.Table_2d1d85e9244d4f45b19e7961c5aba776 [id=ad9784dc678b41559322d8e6ab2c23b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.182368 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 89a17261fbe0490fbe73ffe199ebb7be (DEFAULT_TABLE table=default.Table_03bc1e423ebc4377b041699c570137b1 [id=4095aa23df1141d688b0677ec51186ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.181541 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2cbe751dc4450eaa3eb60268ad510d (DEFAULT_TABLE table=default.Table_bbd16cb34af94cf4a869e14b72fc9f81 [id=7cfa85c8299145a3b07858fe7fb23cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.181386 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 4c71f7398eab4c099cb45a55e50eee51 (DEFAULT_TABLE table=default.Table_f7042786757e4da189a33bb4c3190322 [id=aa63e2a9ef2b4c128f028a1d6a553ca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.183480 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_872fcb31253f47e1976fbed00a641287"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:48.183903 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4a31e5aa834adeb9f59267247c6980 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:48.184290 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet aacdf506221044ffada0fec0c7d42d0b with delete_type TABLET_DATA_DELETED (replaced by tablet 90d4e2f23c9649a4b1dfb6a179ca7be5) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:48.184474 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet e608dd4f99004bb795de5e05ea700e08 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:48.184448 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2cbe751dc4450eaa3eb60268ad510d (DEFAULT_TABLE table=default.Table_bbd16cb34af94cf4a869e14b72fc9f81 [id=7cfa85c8299145a3b07858fe7fb23cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.184752 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 4eed19c1aa7c455eb23bad90c3df0b46 (DEFAULT_TABLE table=default.Table_2d1d85e9244d4f45b19e7961c5aba776 [id=ad9784dc678b41559322d8e6ab2c23b5]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:48.184960 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_872fcb31253f47e1976fbed00a641287 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.185276 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 89a17261fbe0490fbe73ffe199ebb7be (DEFAULT_TABLE table=default.Table_03bc1e423ebc4377b041699c570137b1 [id=4095aa23df1141d688b0677ec51186ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.186614 24175 tablet_bootstrap.cc:492] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:48.186733 24175 ts_tablet_manager.cc:1367] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.003s
I20240502 08:16:48.187482 24175 raft_consensus.cc:348] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.187660 24175 raft_consensus.cc:374] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.187706 24175 raft_consensus.cc:729] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.187899 24175 consensus_queue.cc:260] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.194486 24175 ts_tablet_manager.cc:1398] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 08:16:48.194790 24175 tablet_bootstrap.cc:492] T 7a97427e819b4826b1ed89e6d021c8d4 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:48.195062 24774 raft_consensus.cc:504] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.195946 24774 leader_election.cc:288] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.196291 24175 tablet_bootstrap.cc:654] T 7a97427e819b4826b1ed89e6d021c8d4 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.198086 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9402cf8abc4b4d893d42f9a3a99016" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:48.198153 24944 raft_consensus.cc:504] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.198263 24054 raft_consensus.cc:3037] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.198872 24057 raft_consensus.cc:2379] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in this term.
I20240502 08:16:48.199775 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c17e1be6ef294769a01981f4d361be21" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:48.199961 23786 raft_consensus.cc:3037] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.204511 24175 tablet_bootstrap.cc:492] T 7a97427e819b4826b1ed89e6d021c8d4 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:48.201516 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_53c08a7447a44323883744c033efcbfb" } new_table_name: "default.Table_c99c7506b13949cf9c5534c4c5b9d3b4" modify_external_catalogs: true
I20240502 08:16:48.207217 24944 leader_election.cc:288] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:48.208634 24175 ts_tablet_manager.cc:1367] T 7a97427e819b4826b1ed89e6d021c8d4 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.002s
I20240502 08:16:48.209697 24175 raft_consensus.cc:348] T 7a97427e819b4826b1ed89e6d021c8d4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.209931 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2cbe751dc4450eaa3eb60268ad510d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.210666 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eed19c1aa7c455eb23bad90c3df0b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.211185 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89a17261fbe0490fbe73ffe199ebb7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.211314 24175 raft_consensus.cc:374] T 7a97427e819b4826b1ed89e6d021c8d4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.211382 24175 raft_consensus.cc:729] T 7a97427e819b4826b1ed89e6d021c8d4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.206996 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c17e1be6ef294769a01981f4d361be21" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:48.211571 24175 consensus_queue.cc:260] T 7a97427e819b4826b1ed89e6d021c8d4 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.211794 23924 raft_consensus.cc:2379] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 149ce744feab49b5aa5bcaa14bd6aba5 in current term 1: Already voted for candidate 5b89a1f0485a44b7ac1710e2369a809b in this term.
I20240502 08:16:48.212080 24175 ts_tablet_manager.cc:1398] T 7a97427e819b4826b1ed89e6d021c8d4 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:16:48.212335 24175 tablet_bootstrap.cc:492] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:48.213716 24175 tablet_bootstrap.cc:654] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.220445 24175 tablet_bootstrap.cc:492] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:48.220564 24175 ts_tablet_manager.cc:1367] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240502 08:16:48.221197 24175 raft_consensus.cc:348] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.221372 24175 raft_consensus.cc:374] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.221447 24175 raft_consensus.cc:729] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.226480 25012 tablet_replica.cc:330] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:48.230406 25012 raft_consensus.cc:2229] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:48.230773 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c71f7398eab4c099cb45a55e50eee51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.231864 25012 raft_consensus.cc:2258] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:48.232582 24944 raft_consensus.cc:482] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.232795 24944 raft_consensus.cc:504] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.235018 24944 leader_election.cc:288] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:48.235116 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a97427e819b4826b1ed89e6d021c8d4" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:48.235684 23993 leader_election.cc:341] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:48.234176 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a97427e819b4826b1ed89e6d021c8d4" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:48.242107 23924 raft_consensus.cc:2454] T 7a97427e819b4826b1ed89e6d021c8d4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:48.242659 23996 leader_election.cc:302] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:48.249693 24944 raft_consensus.cc:2781] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.249814 24944 raft_consensus.cc:482] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.249882 24944 raft_consensus.cc:3037] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.253986 25012 ts_tablet_manager.cc:1875] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:48.263595 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0902a73da6e34e8e9cd59bb53a36ec0e has no consensus state
I20240502 08:16:48.221643 24175 consensus_queue.cc:260] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.266266 24175 ts_tablet_manager.cc:1398] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20240502 08:16:48.266655 24175 tablet_bootstrap.cc:492] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:48.266646 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58
I20240502 08:16:48.267371 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_33297782b5e64503b4186fe4a347f8e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.267976 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33297782b5e64503b4186fe4a347f8e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.268751 24175 tablet_bootstrap.cc:654] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.269460 24021 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.270359 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ba6d528fd464438eb4c79eeade694e24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.270813 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba6d528fd464438eb4c79eeade694e24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.274469 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:48.277848 24774 raft_consensus.cc:482] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.277992 24774 raft_consensus.cc:504] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.278563 24774 leader_election.cc:288] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.279212 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fcb895c8f449deb0190de75af18c3c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:48.279415 24057 raft_consensus.cc:2454] T 96fcb895c8f449deb0190de75af18c3c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:48.279939 23723 leader_election.cc:302] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:48.280275 24774 raft_consensus.cc:2781] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.280354 24774 raft_consensus.cc:482] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.280418 24774 raft_consensus.cc:3037] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.282059 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 319fabf91458438ba5e40ab68b323055 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.282620 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b1fa17dad143f0a7f1751e651eb81a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.282945 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 98a96292b28a4b0aad88fe0a96730a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.283289 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet aacdf506221044ffada0fec0c7d42d0b with delete_type TABLET_DATA_DELETED (replaced by tablet 90d4e2f23c9649a4b1dfb6a179ca7be5) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.283986 23900 tablet_service.cc:1467] Processing CreateTablet for tablet c2144cbe36904d0a9ed9ada369186d02 (DEFAULT_TABLE table=default.Table_22be413d6296475eb6266315d70808d9 [id=8d9a421debf74d89beacd1b6b147d735]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.284485 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2144cbe36904d0a9ed9ada369186d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.285755 24054 raft_consensus.cc:2454] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:48.288852 23723 leader_election.cc:302] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:48.289579 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fa986dbf19ba412dbe12164c322d7558" } new_table_name: "default.Table_0f452d0e88414d98a241e34d44354db3" modify_external_catalogs: true
I20240502 08:16:48.290105 24891 raft_consensus.cc:2781] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:48.290217 24891 raft_consensus.cc:686] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:48.291368 25031 raft_consensus.cc:482] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.291486 25031 raft_consensus.cc:504] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.290436 24891 consensus_queue.cc:237] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.292096 25031 leader_election.cc:288] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:48.293072 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e549cad1b3be4b28abb0bf5af720eb52" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:48.293810 23996 leader_election.cc:341] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:48.294453 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e549cad1b3be4b28abb0bf5af720eb52" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:48.297878 23993 leader_election.cc:341] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:48.298008 23993 leader_election.cc:302] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:48.298301 25031 raft_consensus.cc:2726] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:48.298535 24174 tablet_bootstrap.cc:492] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:48.298694 24174 ts_tablet_manager.cc:1367] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.166s user 0.022s sys 0.022s
I20240502 08:16:48.299466 24174 raft_consensus.cc:348] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.299649 24174 raft_consensus.cc:374] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.299813 24174 raft_consensus.cc:729] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.300069 24174 consensus_queue.cc:260] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.300745 24174 ts_tablet_manager.cc:1398] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:48.301023 24174 tablet_bootstrap.cc:492] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:48.302991 24174 tablet_bootstrap.cc:654] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.303258 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_adc39e4ac3924f079897f27ed4f1981f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.304076 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adc39e4ac3924f079897f27ed4f1981f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.307881 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 7d07efb303b64ce19d993c4c76a47dfc
I20240502 08:16:48.309147 24021 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d07efb303b64ce19d993c4c76a47dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.310077 24174 tablet_bootstrap.cc:492] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:48.310217 24174 ts_tablet_manager.cc:1367] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 08:16:48.310281 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d07efb303b64ce19d993c4c76a47dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:48.311017 24174 raft_consensus.cc:348] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.311208 24174 raft_consensus.cc:374] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.311276 24174 raft_consensus.cc:729] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.311517 24174 consensus_queue.cc:260] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.312280 24174 ts_tablet_manager.cc:1398] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:48.313028 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d8a5bcad2db34ac7bc3a55ce30fee5da" } new_table_name: "default.Table_5af2a55b1ec14ac0ab16ca949eb002b8" modify_external_catalogs: true
I20240502 08:16:48.313143 24174 tablet_bootstrap.cc:492] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:48.314991 24174 tablet_bootstrap.cc:654] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.315680 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eed19c1aa7c455eb23bad90c3df0b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.316325 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2cbe751dc4450eaa3eb60268ad510d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.317109 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c71f7398eab4c099cb45a55e50eee51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.317801 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89a17261fbe0490fbe73ffe199ebb7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.322566 24181 ts_tablet_manager.cc:1888] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:48.324466 24181 log.cc:1196] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/1e5de793e74247dd9e73b7b1d58027d9
I20240502 08:16:48.331360 24181 ts_tablet_manager.cc:1909] T 1e5de793e74247dd9e73b7b1d58027d9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:48.331388 24174 tablet_bootstrap.cc:492] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:48.331843 24174 ts_tablet_manager.cc:1367] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240502 08:16:48.332537 24174 raft_consensus.cc:348] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.332743 24174 raft_consensus.cc:374] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.332798 24174 raft_consensus.cc:729] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.332973 24174 consensus_queue.cc:260] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.333473 24174 ts_tablet_manager.cc:1398] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:48.333750 24174 tablet_bootstrap.cc:492] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:48.337045 24174 tablet_bootstrap.cc:654] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.338562 24778 raft_consensus.cc:482] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.338701 24778 raft_consensus.cc:504] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.339390 24778 leader_election.cc:288] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.339871 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e746013fde4de4862f213792e235ed" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:48.339927 25021 raft_consensus.cc:504] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.340087 24054 raft_consensus.cc:2454] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:48.340478 25021 leader_election.cc:288] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.340585 23723 leader_election.cc:302] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:48.341127 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_af07f004e79840bebc030f84c14bb080" } new_table_name: "default.Table_27db1c680fe04c8fb2adcfe67650f0a5" modify_external_catalogs: true
I20240502 08:16:48.341250 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2126bf2d4ab44628b52481dc2ec3a99" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:48.341396 24054 raft_consensus.cc:3037] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.341722 25021 raft_consensus.cc:2781] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.341809 25021 raft_consensus.cc:482] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.341879 25021 raft_consensus.cc:3037] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.342304 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fc2c7b3f23234f139178c7719ab5f044" } modify_external_catalogs: true
I20240502 08:16:48.342468 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fc2c7b3f23234f139178c7719ab5f044" } modify_external_catalogs: true
W20240502 08:16:48.344929 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3777d2e9d4464552ad1c4c8c0d3e9888 consensus state has no leader
I20240502 08:16:48.351032 24174 tablet_bootstrap.cc:492] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:48.351179 24174 ts_tablet_manager.cc:1367] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20240502 08:16:48.351891 24174 raft_consensus.cc:348] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.352083 24174 raft_consensus.cc:374] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.352136 24174 raft_consensus.cc:729] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.352327 24174 consensus_queue.cc:260] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.352824 24174 ts_tablet_manager.cc:1398] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:48.353085 24174 tablet_bootstrap.cc:492] T 7a97427e819b4826b1ed89e6d021c8d4 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:48.359092 23601 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_8bed20f633e348679b0087687298cc42 [id=18ab2c0716464d82a0e33bd99bd0735d] alter complete (version 3)
I20240502 08:16:48.361419 24174 tablet_bootstrap.cc:654] T 7a97427e819b4826b1ed89e6d021c8d4 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.363337 25029 raft_consensus.cc:504] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.364100 25029 leader_election.cc:288] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:48.367280 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacdf506221044ffada0fec0c7d42d0b" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:48.367532 23924 raft_consensus.cc:3037] T aacdf506221044ffada0fec0c7d42d0b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.371837 24174 tablet_bootstrap.cc:492] T 7a97427e819b4826b1ed89e6d021c8d4 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:48.371965 24174 ts_tablet_manager.cc:1367] T 7a97427e819b4826b1ed89e6d021c8d4 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.001s
W20240502 08:16:48.372386 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d2f7ccb4385d4c3cad54758d765e8452 consensus state has no leader
I20240502 08:16:48.372689 24174 raft_consensus.cc:348] T 7a97427e819b4826b1ed89e6d021c8d4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.372880 24174 raft_consensus.cc:374] T 7a97427e819b4826b1ed89e6d021c8d4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.372932 24174 raft_consensus.cc:729] T 7a97427e819b4826b1ed89e6d021c8d4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.373147 24174 consensus_queue.cc:260] T 7a97427e819b4826b1ed89e6d021c8d4 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.373723 24174 ts_tablet_manager.cc:1398] T 7a97427e819b4826b1ed89e6d021c8d4 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:48.373997 24174 tablet_bootstrap.cc:492] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:48.379619 25018 raft_consensus.cc:482] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.379801 25018 raft_consensus.cc:504] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.380441 25018 leader_election.cc:288] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.381199 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2126bf2d4ab44628b52481dc2ec3a99" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:48.382105 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e02044df8b8c4ca9a9bdccaaf14a2eb4" } modify_external_catalogs: true
I20240502 08:16:48.382290 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e02044df8b8c4ca9a9bdccaaf14a2eb4" } modify_external_catalogs: true
I20240502 08:16:48.383544 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_af464bf1ebe54d7281005831aa79b8bf" } modify_external_catalogs: true
I20240502 08:16:48.383718 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_af464bf1ebe54d7281005831aa79b8bf" } modify_external_catalogs: true
I20240502 08:16:48.385170 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacdf506221044ffada0fec0c7d42d0b" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:48.385393 23788 raft_consensus.cc:3037] T aacdf506221044ffada0fec0c7d42d0b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.387720 24180 ts_tablet_manager.cc:1888] T 319fabf91458438ba5e40ab68b323055 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:48.387949 24180 log.cc:1196] T 319fabf91458438ba5e40ab68b323055 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/319fabf91458438ba5e40ab68b323055
I20240502 08:16:48.388546 24180 ts_tablet_manager.cc:1909] T 319fabf91458438ba5e40ab68b323055 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:48.391041 24174 tablet_bootstrap.cc:654] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:48.393237 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82c49bba3df548558123344a13c946ee consensus state has no leader
I20240502 08:16:48.397883 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5fb8fa42fdb945729f5026cab15051b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.398458 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fb8fa42fdb945729f5026cab15051b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.400226 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 46e577132e5249b28426bfe38a28635a (DEFAULT_TABLE table=default.Table_7b3b1098fbed4d2792dadff76cbc948b [id=ce3d0ff6ef76431b85d0a98efd113b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.401114 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e577132e5249b28426bfe38a28635a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.403555 23607 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet b811b81ddf2b4dc9b5e5f80331bff294 (table default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418]). Expected version 2 got 0
I20240502 08:16:48.405524 24021 tablet_service.cc:1514] Processing DeleteTablet for tablet e549cad1b3be4b28abb0bf5af720eb52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:48.407655 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ae2b20ae3a014c85af33d5ee51b704e1 consensus state has no leader
I20240502 08:16:48.418337 24056 raft_consensus.cc:2454] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:48.422940 23723 leader_election.cc:302] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:48.423468 24975 raft_consensus.cc:2781] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:48.423596 24975 raft_consensus.cc:686] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:48.423890 24975 consensus_queue.cc:237] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.426506 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet f797d3c6bdc4493398954fbbf9943971 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.427608 23604 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 37c63bf3741f402fbf0b9d96458f90c3
I20240502 08:16:48.428172 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 18625a1d971c43549b9f80e6046efc9c
I20240502 08:16:48.429394 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 18625a1d971c43549b9f80e6046efc9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:48.430441 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 18625a1d971c43549b9f80e6046efc9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.430689 24778 consensus_queue.cc:1035] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:48.430840 25029 raft_consensus.cc:482] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.430975 25029 raft_consensus.cc:504] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.431603 25029 leader_election.cc:288] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:48.434031 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e00f744a28c6496e98f875d47bdccf95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.434620 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e00f744a28c6496e98f875d47bdccf95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.432339 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd8338500f54604abcd6d42702c2eca" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:48.436817 23996 leader_election.cc:341] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:48.445552 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 4bea89eccaec4d049c0ec3900ed600c2 (DEFAULT_TABLE table=default.Table_01f87df24c614973b564898fe3bfe511 [id=0746ccdaa1f44f00a4f3a4e4d8b2946c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.446060 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bea89eccaec4d049c0ec3900ed600c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.449882 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet 571886fdbc2c43e19c790b2b7830ec27 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:48.449883 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 69246bf9f6be42b9a0671ae68a633255 (DEFAULT_TABLE table=default.Table_2c882aad1a40484297ed2b6af34de3c6 [id=2458f4f98e1f450eb77541227e68e6d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.450775 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69246bf9f6be42b9a0671ae68a633255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.452975 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e134e90efb4a48bb82b53a80bda708b3
I20240502 08:16:48.449836 23764 tablet_service.cc:1467] Processing CreateTablet for tablet c44ce35c463f416794edccc7149aa0cf (DEFAULT_TABLE table=default.Table_06da23829ca541b7ae4be7b15db26c34 [id=11350612be614bbb9436d12d15cfb895]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.453920 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c44ce35c463f416794edccc7149aa0cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.457796 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet e134e90efb4a48bb82b53a80bda708b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:48.458521 24174 tablet_bootstrap.cc:492] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:48.458664 24174 ts_tablet_manager.cc:1367] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.085s user 0.013s sys 0.002s
I20240502 08:16:48.459468 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet e134e90efb4a48bb82b53a80bda708b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.459425 24174 raft_consensus.cc:348] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.459610 24174 raft_consensus.cc:374] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.459672 24174 raft_consensus.cc:729] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.459777 24025 tablet_service.cc:1467] Processing CreateTablet for tablet c44ce35c463f416794edccc7149aa0cf (DEFAULT_TABLE table=default.Table_06da23829ca541b7ae4be7b15db26c34 [id=11350612be614bbb9436d12d15cfb895]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.459909 24174 consensus_queue.cc:260] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.460192 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c44ce35c463f416794edccc7149aa0cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.460438 24174 ts_tablet_manager.cc:1398] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:48.460711 24174 tablet_bootstrap.cc:492] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:48.463860 24174 tablet_bootstrap.cc:654] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.463863 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 4bea89eccaec4d049c0ec3900ed600c2 (DEFAULT_TABLE table=default.Table_01f87df24c614973b564898fe3bfe511 [id=0746ccdaa1f44f00a4f3a4e4d8b2946c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.464592 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bea89eccaec4d049c0ec3900ed600c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.470582 23924 raft_consensus.cc:2454] T aacdf506221044ffada0fec0c7d42d0b P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:48.470613 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 69246bf9f6be42b9a0671ae68a633255 (DEFAULT_TABLE table=default.Table_2c882aad1a40484297ed2b6af34de3c6 [id=2458f4f98e1f450eb77541227e68e6d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.470988 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69246bf9f6be42b9a0671ae68a633255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.471315 23996 leader_election.cc:302] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:48.471616 25029 raft_consensus.cc:2781] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:48.471746 25029 raft_consensus.cc:686] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:48.471982 25029 consensus_queue.cc:237] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.474872 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4ee153a309b04ab68b03ab96694a1952" } modify_external_catalogs: true
I20240502 08:16:48.475080 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4ee153a309b04ab68b03ab96694a1952" } modify_external_catalogs: true
I20240502 08:16:48.476501 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5b5eb6d7519d43cbb31e928e05c9569f" } new_table_name: "default.Table_1159b00d82f4483c8209c879e2d3c06a" modify_external_catalogs: true
I20240502 08:16:48.482653 24173 tablet_bootstrap.cc:492] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:48.482867 24173 ts_tablet_manager.cc:1367] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.335s user 0.026s sys 0.084s
I20240502 08:16:48.488396 24174 tablet_bootstrap.cc:492] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:48.488618 24174 ts_tablet_manager.cc:1367] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.004s
I20240502 08:16:48.489329 24174 raft_consensus.cc:348] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.489609 24174 raft_consensus.cc:374] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.489701 24174 raft_consensus.cc:729] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.489926 24174 consensus_queue.cc:260] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.490396 24174 ts_tablet_manager.cc:1398] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:48.490670 24174 tablet_bootstrap.cc:492] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:48.491079 24778 raft_consensus.cc:482] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.491286 24778 raft_consensus.cc:504] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.492435 24174 tablet_bootstrap.cc:654] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.492652 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e549cad1b3be4b28abb0bf5af720eb52" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:48.492878 24057 raft_consensus.cc:2454] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:48.493108 24778 leader_election.cc:288] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.493484 23723 leader_election.cc:302] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:48.493619 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 2da25b412cec4b0a90bce47f977cafd1 (DEFAULT_TABLE table=default.Table_989d6e2cbf2547d3859e400799f166c2 [id=2ae1dc10ab5443e9b91d5e3e690cdf15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.493803 24778 raft_consensus.cc:2781] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.493916 24778 raft_consensus.cc:482] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.493993 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2da25b412cec4b0a90bce47f977cafd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.494071 24778 raft_consensus.cc:3037] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.497515 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd8338500f54604abcd6d42702c2eca" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:48.497712 23787 raft_consensus.cc:2454] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:48.499403 23759 tablet_service.cc:1467] Processing CreateTablet for tablet ee3d91d55ffd49ceb20d2c33408f5a5a (DEFAULT_TABLE table=default.Table_c002eb9168244875bfe05cafb44aae75 [id=0435c821335b45d0bbd90ae6747690ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.499743 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee3d91d55ffd49ceb20d2c33408f5a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.499878 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 571886fdbc2c43e19c790b2b7830ec27
I20240502 08:16:48.500370 23604 catalog_manager.cc:6554] ReplaceTablet: tablet 571886fdbc2c43e19c790b2b7830ec27 deleted and replaced by tablet 4e32319fa9244ed2a323b2b3f5940748
I20240502 08:16:48.500741 24021 tablet_service.cc:1514] Processing DeleteTablet for tablet 571886fdbc2c43e19c790b2b7830ec27 with delete_type TABLET_DATA_DELETED (replaced by tablet 4e32319fa9244ed2a323b2b3f5940748) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.500772 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e37d58fbcf6b476fb8c4558b3763c94f
I20240502 08:16:48.501403 24021 tablet_service.cc:1514] Processing DeleteTablet for tablet e37d58fbcf6b476fb8c4558b3763c94f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.501461 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d3cd93eb7315400aa078bbb480182814"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.502010 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3cd93eb7315400aa078bbb480182814 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.502432 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 571886fdbc2c43e19c790b2b7830ec27 with delete_type TABLET_DATA_DELETED (replaced by tablet 4e32319fa9244ed2a323b2b3f5940748) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:48.502621 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet e37d58fbcf6b476fb8c4558b3763c94f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:48.502811 23993 leader_election.cc:302] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:48.503365 25029 raft_consensus.cc:2781] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.503492 25029 raft_consensus.cc:482] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.503597 25029 raft_consensus.cc:3037] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.504312 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 2da25b412cec4b0a90bce47f977cafd1 (DEFAULT_TABLE table=default.Table_989d6e2cbf2547d3859e400799f166c2 [id=2ae1dc10ab5443e9b91d5e3e690cdf15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.504706 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2da25b412cec4b0a90bce47f977cafd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.505887 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a33cd662ce1043e3853792e92624c9be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.506426 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a33cd662ce1043e3853792e92624c9be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.510571 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9c8c85b0e8084a3698ebc3aae915c41c" } modify_external_catalogs: true
I20240502 08:16:48.510766 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9c8c85b0e8084a3698ebc3aae915c41c" } modify_external_catalogs: true
I20240502 08:16:48.511209 24021 tablet_service.cc:1467] Processing CreateTablet for tablet ee3d91d55ffd49ceb20d2c33408f5a5a (DEFAULT_TABLE table=default.Table_c002eb9168244875bfe05cafb44aae75 [id=0435c821335b45d0bbd90ae6747690ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.511415 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 1638e78b6f544209bd8d6da6556fbce4 (DEFAULT_TABLE table=default.Table_a86b6b9ac2d8498c97341dc60ca5ee34 [id=fd93259a378b4ea582a6661a59104abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.511551 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee3d91d55ffd49ceb20d2c33408f5a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.511760 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1638e78b6f544209bd8d6da6556fbce4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:48.517779 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c395a29c7a1485eab39ca107dd91fca consensus state has no leader
I20240502 08:16:48.519908 23749 tablet_service.cc:1467] Processing CreateTablet for tablet c6f15b4b87a04707a3b4f37538e3b64a (DEFAULT_TABLE table=default.Table_ee25478ce0e04c4091efb71e8277beff [id=ebd2506244af43c688bba2cf3a84fe10]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.520294 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6f15b4b87a04707a3b4f37538e3b64a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.520251 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 1638e78b6f544209bd8d6da6556fbce4 (DEFAULT_TABLE table=default.Table_a86b6b9ac2d8498c97341dc60ca5ee34 [id=fd93259a378b4ea582a6661a59104abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.520591 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1638e78b6f544209bd8d6da6556fbce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.523939 23786 raft_consensus.cc:2454] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:48.524852 24036 tablet_service.cc:1467] Processing CreateTablet for tablet c6f15b4b87a04707a3b4f37538e3b64a (DEFAULT_TABLE table=default.Table_ee25478ce0e04c4091efb71e8277beff [id=ebd2506244af43c688bba2cf3a84fe10]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.525233 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6f15b4b87a04707a3b4f37538e3b64a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.525429 24174 tablet_bootstrap.cc:492] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:48.525552 24174 ts_tablet_manager.cc:1367] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.002s
I20240502 08:16:48.525523 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 462f77a57ebf45948cbd3dc20c271650 (DEFAULT_TABLE table=default.Table_5a6711c7a8e74b05af6d0c8fb3cfa33f [id=ffe60ec845b84b53bc4a63d604362018]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.525871 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 462f77a57ebf45948cbd3dc20c271650. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.526036 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 82c49bba3df548558123344a13c946ee (DEFAULT_TABLE table=default.Table_5af2a55b1ec14ac0ab16ca949eb002b8 [id=f316b32b17bd44049fbdaffe040eb0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.526379 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c49bba3df548558123344a13c946ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.526434 24174 raft_consensus.cc:348] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.526633 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 21d2db2158d4447f92eedc5780b46056 (DEFAULT_TABLE table=default.Table_7b367902a5d84aa5ac4aa4a0b97b48f5 [id=afca80b25b6547baabe49be50efadad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.526940 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d2db2158d4447f92eedc5780b46056. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.528280 23758 tablet_service.cc:1467] Processing CreateTablet for tablet e5de31a3db014615989d9e018521bfc9 (DEFAULT_TABLE table=default.Table_5a39648854e8433eac74de3c8f04b5c5 [id=0cbbb3e6ffed4f31b5b43042cad3cffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.528586 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5de31a3db014615989d9e018521bfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.526625 24174 raft_consensus.cc:374] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.529954 24174 raft_consensus.cc:729] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.530177 24174 consensus_queue.cc:260] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.530673 24174 ts_tablet_manager.cc:1398] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 08:16:48.530839 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 0e559ce43d2c440fa50715f25cc2c7e9
I20240502 08:16:48.531651 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e559ce43d2c440fa50715f25cc2c7e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:48.534194 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 82c49bba3df548558123344a13c946ee (DEFAULT_TABLE table=default.Table_5af2a55b1ec14ac0ab16ca949eb002b8 [id=f316b32b17bd44049fbdaffe040eb0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.534498 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c49bba3df548558123344a13c946ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.535368 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 462f77a57ebf45948cbd3dc20c271650 (DEFAULT_TABLE table=default.Table_5a6711c7a8e74b05af6d0c8fb3cfa33f [id=ffe60ec845b84b53bc4a63d604362018]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.535800 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 462f77a57ebf45948cbd3dc20c271650. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.537811 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 21d2db2158d4447f92eedc5780b46056 (DEFAULT_TABLE table=default.Table_7b367902a5d84aa5ac4aa4a0b97b48f5 [id=afca80b25b6547baabe49be50efadad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.538113 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d2db2158d4447f92eedc5780b46056. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.541854 25018 raft_consensus.cc:482] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.542063 25018 raft_consensus.cc:504] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.542732 24022 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e559ce43d2c440fa50715f25cc2c7e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.542896 25018 leader_election.cc:288] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.543313 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_f30d039709f9465e96e1280298ae2aaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.543907 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f30d039709f9465e96e1280298ae2aaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:48.544034 23861 leader_election.cc:341] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 98a96292b28a4b0aad88fe0a96730a68
I20240502 08:16:48.544884 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_3ee62693a3d14323a553988181673d32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.545378 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ee62693a3d14323a553988181673d32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.546249 24174 tablet_bootstrap.cc:492] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:48.547642 23993 leader_election.cc:302] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:48.547686 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5a39648854e8433eac74de3c8f04b5c5" } new_table_name: "default.Table_5889de778d2842aba3b3a02fb663425a" modify_external_catalogs: true
I20240502 08:16:48.548056 24174 tablet_bootstrap.cc:654] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.548066 25031 raft_consensus.cc:2781] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:48.549130 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2c0098a6f7e3460e8a2bed24e0ecfb77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.549671 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c0098a6f7e3460e8a2bed24e0ecfb77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.553948 25029 raft_consensus.cc:504] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.560047 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0e7c8c39d3c4443d8b770a580c137b2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:48.560387 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd8338500f54604abcd6d42702c2eca" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:48.560580 23786 raft_consensus.cc:3037] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:48.560582 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e7c8c39d3c4443d8b770a580c137b2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.561821 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd8338500f54604abcd6d42702c2eca" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:48.562356 23996 leader_election.cc:341] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:48.563017 25029 leader_election.cc:288] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:48.563354 25031 raft_consensus.cc:686] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:48.563891 25031 consensus_queue.cc:237] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.569219 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b1d809964c99404891db2d83cb16887f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.570185 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1d809964c99404891db2d83cb16887f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.570601 24174 tablet_bootstrap.cc:492] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:48.570747 24174 ts_tablet_manager.cc:1367] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.006s
W20240502 08:16:48.575400 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5de31a3db014615989d9e018521bfc9 consensus state has no leader
I20240502 08:16:48.576648 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bcc7169796e4ee0910f5e7084849012 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.576732 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 788256e092b7475d8fca0904bf615cef (DEFAULT_TABLE table=default.Table_91d4de88c52a461d8d4006baab592636 [id=bc67017e4b6046d5b03a2d644db4dc72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.577066 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e2c611d7af6428db289a7e2fbd586d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.577456 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet e608dd4f99004bb795de5e05ea700e08 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.578069 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 4c71f7398eab4c099cb45a55e50eee51 (DEFAULT_TABLE table=default.Table_f7042786757e4da189a33bb4c3190322 [id=aa63e2a9ef2b4c128f028a1d6a553ca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.578796 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 4eed19c1aa7c455eb23bad90c3df0b46 (DEFAULT_TABLE table=default.Table_2d1d85e9244d4f45b19e7961c5aba776 [id=ad9784dc678b41559322d8e6ab2c23b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.579612 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 89a17261fbe0490fbe73ffe199ebb7be (DEFAULT_TABLE table=default.Table_03bc1e423ebc4377b041699c570137b1 [id=4095aa23df1141d688b0677ec51186ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.580569 24174 raft_consensus.cc:348] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.580646 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2cbe751dc4450eaa3eb60268ad510d (DEFAULT_TABLE table=default.Table_bbd16cb34af94cf4a869e14b72fc9f81 [id=7cfa85c8299145a3b07858fe7fb23cad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.580749 24174 raft_consensus.cc:374] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.580803 24174 raft_consensus.cc:729] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.580963 24174 consensus_queue.cc:260] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.581429 24174 ts_tablet_manager.cc:1398] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240502 08:16:48.582247 24174 tablet_bootstrap.cc:492] T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:48.584131 24174 tablet_bootstrap.cc:654] T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.590019 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f7042786757e4da189a33bb4c3190322" } new_table_name: "default.Table_340594e848c94913801c031a70a94575" modify_external_catalogs: true
I20240502 08:16:48.591701 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_69664f97f411484e8277f8a0a02b6f90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:48.591885 24175 tablet_bootstrap.cc:492] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:48.592041 24175 ts_tablet_manager.cc:1367] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.325s user 0.005s sys 0.004s
W20240502 08:16:48.592263 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69664f97f411484e8277f8a0a02b6f90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.592830 24175 raft_consensus.cc:348] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.593055 24175 raft_consensus.cc:374] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.593122 24175 raft_consensus.cc:729] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.593405 24175 consensus_queue.cc:260] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.595743 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1a8e51c22c1a4da9928bb3cf87102b6e" } new_table_name: "default.Table_a10aba559706443aae94d509cbb5eefa" modify_external_catalogs: true
I20240502 08:16:48.596971 25070 raft_consensus.cc:482] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.597148 25070 raft_consensus.cc:504] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.598415 25070 leader_election.cc:288] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.598469 24056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1e208733844829b13222e00a432439" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:48.598708 24056 raft_consensus.cc:2454] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:48.599267 23723 leader_election.cc:302] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:48.599673 25070 raft_consensus.cc:2781] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.599890 25070 raft_consensus.cc:482] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.600081 25070 raft_consensus.cc:3037] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:48.600934 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c71f7398eab4c099cb45a55e50eee51 consensus state has no leader
I20240502 08:16:48.602147 24173 raft_consensus.cc:348] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.602326 24173 raft_consensus.cc:374] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.602370 24173 raft_consensus.cc:729] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.602540 24173 consensus_queue.cc:260] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.603021 24173 ts_tablet_manager.cc:1398] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.120s user 0.001s sys 0.001s
I20240502 08:16:48.603296 24173 tablet_bootstrap.cc:492] T f5ccd43e78574f3288b199fa14bcc0e8 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:48.603404 24174 tablet_bootstrap.cc:492] T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:48.603555 24174 ts_tablet_manager.cc:1367] T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.004s
I20240502 08:16:48.604189 24174 raft_consensus.cc:348] T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.604395 24174 raft_consensus.cc:374] T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.604465 24174 raft_consensus.cc:729] T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.604732 24173 tablet_bootstrap.cc:654] T f5ccd43e78574f3288b199fa14bcc0e8 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.604714 24174 consensus_queue.cc:260] T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.605203 24174 ts_tablet_manager.cc:1398] T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:48.607198 24174 tablet_bootstrap.cc:492] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:48.608189 23603 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet b811b81ddf2b4dc9b5e5f80331bff294 (table default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418]). Expected version 2 got 0
I20240502 08:16:48.608734 24174 tablet_bootstrap.cc:654] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.612231 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_27db1c680fe04c8fb2adcfe67650f0a5" } new_table_name: "default.Table_01d1b1f9058e48178e6bfab9c6981d97" modify_external_catalogs: true
W20240502 08:16:48.615593 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88bf97d0a0a54e828c3b0902976e0500 has no consensus state
W20240502 08:16:48.616674 24973 consensus_meta.cc:287] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent flushing consensus metadata: real 0.542s user 0.000s sys 0.003s
I20240502 08:16:48.617390 24973 raft_consensus.cc:504] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.618042 24973 leader_election.cc:288] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:48.621153 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ae2b20ae3a014c85af33d5ee51b704e1 consensus state has no leader
I20240502 08:16:48.625417 24022 tablet_service.cc:1514] Processing DeleteTablet for tablet c5c377e037894728ac084b6500ad20de with delete_type TABLET_DATA_DELETED (replaced by tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.626569 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 719) took 1564 ms. Trace:
I20240502 08:16:48.626665 24181 rpcz_store.cc:276] 0502 08:16:47.062244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.062772 (+ 528us) service_pool.cc:225] Handling call
0502 08:16:47.995395 (+932623us) ts_tablet_manager.cc:1120] Deleting tablet 1e5de793e74247dd9e73b7b1d58027d9
0502 08:16:47.995405 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:48.104685 (+109280us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:48.322484 (+217799us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:48.626547 (+304063us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":68237,"tablet-delete.queue_time_us":932488}
I20240502 08:16:48.630722 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e5e87b9f9dc64e9e8324057fc970d2fa" } modify_external_catalogs: true
I20240502 08:16:48.630926 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e5e87b9f9dc64e9e8324057fc970d2fa" } modify_external_catalogs: true
I20240502 08:16:48.632089 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f194510125f340b58d41408e911d2721" } modify_external_catalogs: true
I20240502 08:16:48.632279 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f194510125f340b58d41408e911d2721" } modify_external_catalogs: true
I20240502 08:16:48.634546 24972 tablet.cc:1722] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:48.636055 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 56d6373938824840934211f259e14d4f (DEFAULT_TABLE table=default.Table_d3d08ffbf0a743a1ac0642fa87cf6f7e [id=4865a49282a348f1ae2ccf8f292c0672]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.636549 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56d6373938824840934211f259e14d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.637933 24054 raft_consensus.cc:2454] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:48.638612 23723 leader_election.cc:302] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:48.646256 24173 tablet_bootstrap.cc:492] T f5ccd43e78574f3288b199fa14bcc0e8 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:48.646392 24173 ts_tablet_manager.cc:1367] T f5ccd43e78574f3288b199fa14bcc0e8 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.005s
I20240502 08:16:48.647091 24173 raft_consensus.cc:348] T f5ccd43e78574f3288b199fa14bcc0e8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.647333 24173 raft_consensus.cc:374] T f5ccd43e78574f3288b199fa14bcc0e8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.647388 24173 raft_consensus.cc:729] T f5ccd43e78574f3288b199fa14bcc0e8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.647588 24173 consensus_queue.cc:260] T f5ccd43e78574f3288b199fa14bcc0e8 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.648077 24173 ts_tablet_manager.cc:1398] T f5ccd43e78574f3288b199fa14bcc0e8 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:48.653105 24973 raft_consensus.cc:2781] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:48.653273 24973 raft_consensus.cc:686] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:48.653584 24973 consensus_queue.cc:237] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.655223 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_68edbb83e66948898f02d4e24a28a9fe" } modify_external_catalogs: true
I20240502 08:16:48.655380 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 1e5de793e74247dd9e73b7b1d58027d9 (table default.Table_a286e67e441e43a1b4738cd219dbea98 [id=b56e3c0686244ad6b49afdd589c5ed34]) successfully deleted
I20240502 08:16:48.656365 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 598cba0f8b9f4c8dae9e34872c4212fa (DEFAULT_TABLE table=default.Table_3c059ec418cd4c6dbf7bf93c0856437f [id=f29b115e70604df1bc9c0efbe6172ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.656826 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 56d6373938824840934211f259e14d4f (DEFAULT_TABLE table=default.Table_d3d08ffbf0a743a1ac0642fa87cf6f7e [id=4865a49282a348f1ae2ccf8f292c0672]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.657196 24022 tablet_service.cc:1467] Processing CreateTablet for tablet e5de31a3db014615989d9e018521bfc9 (DEFAULT_TABLE table=default.Table_5889de778d2842aba3b3a02fb663425a [id=0cbbb3e6ffed4f31b5b43042cad3cffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.660209 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e9133e69e1724a26bc3650ea658ca349
I20240502 08:16:48.660975 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 598cba0f8b9f4c8dae9e34872c4212fa (DEFAULT_TABLE table=default.Table_3c059ec418cd4c6dbf7bf93c0856437f [id=f29b115e70604df1bc9c0efbe6172ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.661191 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_68edbb83e66948898f02d4e24a28a9fe" } modify_external_catalogs: true
I20240502 08:16:48.661442 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 598cba0f8b9f4c8dae9e34872c4212fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.663220 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 982333cbce18439da3b35693b3f6ca0a
I20240502 08:16:48.664392 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_39434b47dd9b421796ea5924f95292b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.664964 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39434b47dd9b421796ea5924f95292b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.666743 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet 982333cbce18439da3b35693b3f6ca0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:48.667078 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet e9133e69e1724a26bc3650ea658ca349 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.666997 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet e9133e69e1724a26bc3650ea658ca349 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:48.667343 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet 982333cbce18439da3b35693b3f6ca0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.672580 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 725982e4712e442d8ff1fd3f796e18f5
I20240502 08:16:48.673404 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6a6b2f29d9aa4a679e4c41863e3096bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:48.673599 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 725982e4712e442d8ff1fd3f796e18f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:48.674010 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a6b2f29d9aa4a679e4c41863e3096bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.676143 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet 725982e4712e442d8ff1fd3f796e18f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.686498 24891 raft_consensus.cc:482] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.686782 24891 raft_consensus.cc:504] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.688167 24891 leader_election.cc:288] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.691406 24174 tablet_bootstrap.cc:492] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:48.691615 24174 ts_tablet_manager.cc:1367] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.085s user 0.028s sys 0.032s
I20240502 08:16:48.692483 24174 raft_consensus.cc:348] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.692705 24174 raft_consensus.cc:374] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.692804 24174 raft_consensus.cc:729] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.693048 24174 consensus_queue.cc:260] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.693590 24174 ts_tablet_manager.cc:1398] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:48.693917 24174 tablet_bootstrap.cc:492] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:48.695684 24174 tablet_bootstrap.cc:654] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.596810 25018 raft_consensus.cc:482] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.699342 25018 raft_consensus.cc:504] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.700007 25018 leader_election.cc:288] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.611145 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet df9402cf8abc4b4d893d42f9a3a99016 with delete_type TABLET_DATA_DELETED (replaced by tablet fddac683eda14ec1a0140926380f2c07) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.611487 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.684428 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 788256e092b7475d8fca0904bf615cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.705277 25018 raft_consensus.cc:482] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.705441 25018 raft_consensus.cc:504] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.706046 25018 leader_election.cc:288] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.706224 24175 ts_tablet_manager.cc:1398] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.114s user 0.000s sys 0.002s
I20240502 08:16:48.706516 24175 tablet_bootstrap.cc:492] T e549cad1b3be4b28abb0bf5af720eb52 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:48.707373 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2cbe751dc4450eaa3eb60268ad510d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:48.708657 23597 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet d2f7ccb4385d4c3cad54758d765e8452 (table default.Table_0f452d0e88414d98a241e34d44354db3 [id=790ab064f86b4f65a609846ba2192ef8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d2f7ccb4385d4c3cad54758d765e8452 consensus state has no leader
I20240502 08:16:48.709082 24175 tablet_bootstrap.cc:654] T e549cad1b3be4b28abb0bf5af720eb52 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.709883 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eed19c1aa7c455eb23bad90c3df0b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.709782 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bbd16cb34af94cf4a869e14b72fc9f81" } modify_external_catalogs: true
I20240502 08:16:48.709973 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bbd16cb34af94cf4a869e14b72fc9f81" } modify_external_catalogs: true
I20240502 08:16:48.716387 25021 raft_consensus.cc:504] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.717231 25021 leader_election.cc:288] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.719372 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c71f7398eab4c099cb45a55e50eee51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.721830 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89a17261fbe0490fbe73ffe199ebb7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.722054 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 9a2cbe751dc4450eaa3eb60268ad510d
I20240502 08:16:48.725109 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a2cbe751dc4450eaa3eb60268ad510d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.730458 25018 raft_consensus.cc:482] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.730618 25018 raft_consensus.cc:504] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.731112 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_07e89c5c963e42b19f037caf9bbfa7af" } new_table_name: "default.Table_4c6989f7e6bb4e99b447a3c1188f5b92" modify_external_catalogs: true
I20240502 08:16:48.731232 25018 leader_election.cc:288] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.733963 25021 raft_consensus.cc:482] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.734120 25021 raft_consensus.cc:504] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.734944 25021 leader_election.cc:288] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.735205 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_45e91d35e2c245d59419f7105a5b9c8b" } modify_external_catalogs: true
I20240502 08:16:48.735409 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_45e91d35e2c245d59419f7105a5b9c8b" } modify_external_catalogs: true
I20240502 08:16:48.736725 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_41d744e17dee458382533bb204fd1197"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.737246 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41d744e17dee458382533bb204fd1197 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.739282 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet 2b98cc8728c042f4b7f6496fcc5ef913 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:48.746148 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet b123bb81d0b44d4ea6443536168d085c
I20240502 08:16:48.747234 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet b123bb81d0b44d4ea6443536168d085c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:48.747538 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet b123bb81d0b44d4ea6443536168d085c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:48.748610 24032 consensus_meta.cc:287] T c2144cbe36904d0a9ed9ada369186d02 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent flushing consensus metadata: real 0.595s user 0.002s sys 0.000s
I20240502 08:16:48.749866 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a2cbe751dc4450eaa3eb60268ad510d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:48.750053 25031 raft_consensus.cc:482] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.750160 25031 raft_consensus.cc:504] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.751480 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797d3c6bdc4493398954fbbf9943971" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:48.751902 23996 leader_election.cc:341] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:48.752084 25031 leader_election.cc:288] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:48.752882 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet edd7244b9f5741a8aa1c6835fe544676 consensus state has no leader
I20240502 08:16:48.753438 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797d3c6bdc4493398954fbbf9943971" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:48.753686 23787 raft_consensus.cc:2454] T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:48.754099 23993 leader_election.cc:302] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:48.754463 25031 raft_consensus.cc:2781] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.754529 25031 raft_consensus.cc:482] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.754577 25031 raft_consensus.cc:3037] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.756953 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7952191ae36044e9aef62593bbe0bb27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:48.748625 24944 raft_consensus.cc:504] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:48.757486 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7952191ae36044e9aef62593bbe0bb27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.758992 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a97427e819b4826b1ed89e6d021c8d4" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:48.759229 23787 raft_consensus.cc:3037] T 7a97427e819b4826b1ed89e6d021c8d4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.759707 24944 leader_election.cc:288] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:48.760063 23924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a97427e819b4826b1ed89e6d021c8d4" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:48.760288 23924 raft_consensus.cc:3037] T 7a97427e819b4826b1ed89e6d021c8d4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.760798 25012 ts_tablet_manager.cc:1888] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:48.762390 25012 log.cc:1196] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/27346a7eae6a488fa3415646262a4b88
I20240502 08:16:48.763973 24060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4a31e5aa834adeb9f59267247c6980" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:48.764133 24060 raft_consensus.cc:2454] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:48.764346 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9402cf8abc4b4d893d42f9a3a99016" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:48.764530 24053 raft_consensus.cc:2379] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:16:48.763948 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ccd43e78574f3288b199fa14bcc0e8" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:48.765149 24057 raft_consensus.cc:2454] T f5ccd43e78574f3288b199fa14bcc0e8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:48.765373 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e746013fde4de4862f213792e235ed" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:48.765515 24059 raft_consensus.cc:3037] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.765655 23723 leader_election.cc:302] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:48.765779 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4a31e5aa834adeb9f59267247c6980" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:48.765964 24052 raft_consensus.cc:2454] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:48.763938 24058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fcb895c8f449deb0190de75af18c3c" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:48.766112 25021 raft_consensus.cc:2781] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.766172 24058 raft_consensus.cc:2454] T 96fcb895c8f449deb0190de75af18c3c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:48.766240 25021 raft_consensus.cc:482] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.766340 25021 raft_consensus.cc:3037] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.766590 23861 leader_election.cc:302] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:48.766777 23723 leader_election.cc:302] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:48.766961 23861 leader_election.cc:302] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:48.767347 25018 raft_consensus.cc:2781] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.767427 25018 raft_consensus.cc:482] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.767503 25018 raft_consensus.cc:3037] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.768425 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 598cba0f8b9f4c8dae9e34872c4212fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.769451 24173 tablet_bootstrap.cc:492] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:48.772737 24891 raft_consensus.cc:2781] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.772848 24891 raft_consensus.cc:482] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.772965 24891 raft_consensus.cc:3037] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.774045 25076 raft_consensus.cc:2781] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.774122 25076 raft_consensus.cc:482] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.774183 25076 raft_consensus.cc:3037] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.781302 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6bab5c3f6452414eba871948cffb5714"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:48.781546 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_57eccbf54577471a93890cd6fc133102" } new_table_name: "default.Table_853fc43b2ae744be88b3f417db4ae6db" modify_external_catalogs: true
W20240502 08:16:48.781981 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bab5c3f6452414eba871948cffb5714 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.783093 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_973ce52632594e5f9c980d6d7ae7371b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.783519 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_973ce52632594e5f9c980d6d7ae7371b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.787111 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_effefe18d3cd42f7855fed709d87de8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.787602 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_effefe18d3cd42f7855fed709d87de8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.796818 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e97c69e9998149d98b2954d150b8d220"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.797358 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e97c69e9998149d98b2954d150b8d220 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.797932 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 718) took 1741 ms. Trace:
I20240502 08:16:48.798043 24181 rpcz_store.cc:276] 0502 08:16:47.056709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.081714 (+ 25005us) service_pool.cc:225] Handling call
0502 08:16:48.626977 (+1545263us) ts_tablet_manager.cc:1120] Deleting tablet 25c8a6aaabe2413d8ce88f54ce45a4c4
0502 08:16:48.763285 (+136308us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:48.797889 (+ 34604us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1545011}
W20240502 08:16:48.798616 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 25c8a6aaabe2413d8ce88f54ce45a4c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25c8a6aaabe2413d8ce88f54ce45a4c4 already in progress: creating tablet
I20240502 08:16:48.798585 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7a0267df98fe45a583a32684e35f9416"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:48.799010 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a0267df98fe45a583a32684e35f9416 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:48.800670 24829 kernel_stack_watchdog.cc:198] Thread 24715 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 332ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f9e61db4980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f9e5ea22d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f9e65709107 kudu::rpc::ServicePool::Init() at ??:0
@ 0x7f9e679de5f1 kudu::RpcServer::RegisterService() at ??:0
@ 0x7f9e679ff308 kudu::server::ServerBase::RegisterService() at ??:0
@ 0x7f9e71f6c772 kudu::master::Master::StartAsync() at ??:0
@ 0x7f9e71f6bde7 kudu::master::Master::Start() at ??:0
@ 0x7f9e71fcc8ab kudu::master::RunMasterServer() at ??:0
@ 0xa7a0b7 kudu::tools::(anonymous namespace)::MasterRun() at ??:?
@ 0x98d981 std::_Function_handler<>::_M_invoke() at ??:?
@ 0x7f9e728f3b79 std::function<>::operator()() at ??:0
@ 0x7f9e728ef655 kudu::tools::Action::Run() at ??:0
@ 0xb970f4 kudu::tools::DispatchCommand() at ??:?
I20240502 08:16:48.800907 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:48.802404 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8c1e23fe00844168a9f1755bedd3625a" } new_table_name: "default.Table_50a5854b27c94d7a9a5f626cc6c8327e" modify_external_catalogs: true
I20240502 08:16:48.804232 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcc7169796e4ee0910f5e7084849012" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:48.804417 24055 raft_consensus.cc:3037] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.805080 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 723) took 1713 ms. Trace:
W20240502 08:16:48.805657 24715 thread.cc:636] rpc worker (service pool kudu.master.MasterService) Time spent creating pthread: real 0.732s user 0.039s sys 0.093s
W20240502 08:16:48.805764 24715 thread.cc:602] rpc worker (service pool kudu.master.MasterService) Time spent starting thread: real 0.732s user 0.039s sys 0.093s
I20240502 08:16:48.809125 23886 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a97427e819b4826b1ed89e6d021c8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.809381 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4a31e5aa834adeb9f59267247c6980 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.805168 24181 rpcz_store.cc:276] 0502 08:16:47.091839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.092513 (+ 674us) service_pool.cc:225] Handling call
0502 08:16:48.804893 (+1712380us) ts_tablet_manager.cc:1120] Deleting tablet 2411259b872e4be4b8eb8832b36094c4
0502 08:16:48.804914 (+ 21us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:48.805056 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1712138}
W20240502 08:16:48.810089 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 2411259b872e4be4b8eb8832b36094c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2411259b872e4be4b8eb8832b36094c4 already in progress: creating tablet
I20240502 08:16:48.810443 24181 tablet_replica.cc:330] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:48.810672 24181 raft_consensus.cc:2229] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:48.810850 24181 raft_consensus.cc:2258] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:48.811689 24141 log.cc:924] Time spent T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20240502 08:16:48.811690 24565 log.cc:924] Time spent T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240502 08:16:48.814800 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5de31a3db014615989d9e018521bfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.815521 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56d6373938824840934211f259e14d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.819480 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d07efb303b64ce19d993c4c76a47dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.819588 23886 tablet_service.cc:1514] Processing DeleteTablet for tablet 18625a1d971c43549b9f80e6046efc9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.820195 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 4bea89eccaec4d049c0ec3900ed600c2 (DEFAULT_TABLE table=default.Table_01f87df24c614973b564898fe3bfe511 [id=0746ccdaa1f44f00a4f3a4e4d8b2946c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.820844 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet e134e90efb4a48bb82b53a80bda708b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.820909 23893 tablet_service.cc:1467] Processing CreateTablet for tablet c44ce35c463f416794edccc7149aa0cf (DEFAULT_TABLE table=default.Table_06da23829ca541b7ae4be7b15db26c34 [id=11350612be614bbb9436d12d15cfb895]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.821713 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_375988d4437044bc890604e6d5ab8af2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:48.822458 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5509aa65d43f4547853d26666042d04d" } new_table_name: "default.Table_23a2c8534b8b4671a4158faff46116a4" modify_external_catalogs: true
I20240502 08:16:48.825731 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 69246bf9f6be42b9a0671ae68a633255 (DEFAULT_TABLE table=default.Table_2c882aad1a40484297ed2b6af34de3c6 [id=2458f4f98e1f450eb77541227e68e6d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.826406 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 2da25b412cec4b0a90bce47f977cafd1 (DEFAULT_TABLE table=default.Table_989d6e2cbf2547d3859e400799f166c2 [id=2ae1dc10ab5443e9b91d5e3e690cdf15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.826887 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet 571886fdbc2c43e19c790b2b7830ec27 with delete_type TABLET_DATA_DELETED (replaced by tablet 4e32319fa9244ed2a323b2b3f5940748) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.827064 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet e37d58fbcf6b476fb8c4558b3763c94f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.827561 23892 tablet_service.cc:1467] Processing CreateTablet for tablet ee3d91d55ffd49ceb20d2c33408f5a5a (DEFAULT_TABLE table=default.Table_c002eb9168244875bfe05cafb44aae75 [id=0435c821335b45d0bbd90ae6747690ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.828231 23894 tablet_service.cc:1467] Processing CreateTablet for tablet c6f15b4b87a04707a3b4f37538e3b64a (DEFAULT_TABLE table=default.Table_ee25478ce0e04c4091efb71e8277beff [id=ebd2506244af43c688bba2cf3a84fe10]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.828231 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 1638e78b6f544209bd8d6da6556fbce4 (DEFAULT_TABLE table=default.Table_a86b6b9ac2d8498c97341dc60ca5ee34 [id=fd93259a378b4ea582a6661a59104abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.831823 24173 tablet_bootstrap.cc:654] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.832592 24175 tablet_bootstrap.cc:492] T e549cad1b3be4b28abb0bf5af720eb52 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:48.832722 24175 ts_tablet_manager.cc:1367] T e549cad1b3be4b28abb0bf5af720eb52 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.126s user 0.001s sys 0.006s
I20240502 08:16:48.833412 24175 raft_consensus.cc:348] T e549cad1b3be4b28abb0bf5af720eb52 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.833624 24175 raft_consensus.cc:374] T e549cad1b3be4b28abb0bf5af720eb52 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.833678 24174 tablet_bootstrap.cc:492] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:48.833684 24175 raft_consensus.cc:729] T e549cad1b3be4b28abb0bf5af720eb52 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.833824 24174 ts_tablet_manager.cc:1367] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.140s user 0.001s sys 0.003s
I20240502 08:16:48.833859 24175 consensus_queue.cc:260] T e549cad1b3be4b28abb0bf5af720eb52 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.834318 24175 ts_tablet_manager.cc:1398] T e549cad1b3be4b28abb0bf5af720eb52 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:48.834592 24175 tablet_bootstrap.cc:492] T 5a1e208733844829b13222e00a432439 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:48.834563 24174 raft_consensus.cc:348] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.834734 24174 raft_consensus.cc:374] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.834784 24174 raft_consensus.cc:729] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.834985 24174 consensus_queue.cc:260] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.835495 24174 ts_tablet_manager.cc:1398] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:48.835750 24174 tablet_bootstrap.cc:492] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:48.836012 23788 raft_consensus.cc:2454] T aacdf506221044ffada0fec0c7d42d0b P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:48.836112 24175 tablet_bootstrap.cc:654] T 5a1e208733844829b13222e00a432439 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.837190 24174 tablet_bootstrap.cc:654] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:48.839862 25055 log.cc:924] Time spent T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20240502 08:16:48.841595 24180 tablet_replica.cc:330] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:48.841768 24180 raft_consensus.cc:2229] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:48.841960 24180 raft_consensus.cc:2258] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:48.842509 24180 ts_tablet_manager.cc:1875] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:48.844410 23761 consensus_meta.cc:287] T 4c71f7398eab4c099cb45a55e50eee51 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent flushing consensus metadata: real 0.605s user 0.003s sys 0.001s
I20240502 08:16:48.846807 24181 ts_tablet_manager.cc:1875] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:48.847698 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 319fabf91458438ba5e40ab68b323055 (table default.Table_becb097785284ffba69cdbe40e6480ab [id=4454421c70094eed8a8f3e42dcc8a524]) successfully deleted
I20240502 08:16:48.848582 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bea89eccaec4d049c0ec3900ed600c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.849092 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c44ce35c463f416794edccc7149aa0cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.849473 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2da25b412cec4b0a90bce47f977cafd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.849846 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee3d91d55ffd49ceb20d2c33408f5a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.850180 24175 tablet_bootstrap.cc:492] T 5a1e208733844829b13222e00a432439 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:48.850196 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1638e78b6f544209bd8d6da6556fbce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.850314 24175 ts_tablet_manager.cc:1367] T 5a1e208733844829b13222e00a432439 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.007s
I20240502 08:16:48.850611 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6f15b4b87a04707a3b4f37538e3b64a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.851258 24175 raft_consensus.cc:348] T 5a1e208733844829b13222e00a432439 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.851440 24175 raft_consensus.cc:374] T 5a1e208733844829b13222e00a432439 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.851507 24175 raft_consensus.cc:729] T 5a1e208733844829b13222e00a432439 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.851697 24175 consensus_queue.cc:260] T 5a1e208733844829b13222e00a432439 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.852250 25081 tablet.cc:1722] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:48.856078 24175 ts_tablet_manager.cc:1398] T 5a1e208733844829b13222e00a432439 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20240502 08:16:48.856426 24175 tablet_bootstrap.cc:492] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:48.858116 24175 tablet_bootstrap.cc:654] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:48.863150 23861 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 96194.4us
I20240502 08:16:48.864097 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69246bf9f6be42b9a0671ae68a633255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.865897 24174 tablet_bootstrap.cc:492] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:48.866021 24175 tablet_bootstrap.cc:492] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:48.866124 24175 ts_tablet_manager.cc:1367] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20240502 08:16:48.866149 24174 ts_tablet_manager.cc:1367] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.003s
I20240502 08:16:48.866838 24175 raft_consensus.cc:348] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.867004 24175 raft_consensus.cc:374] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.867053 24175 raft_consensus.cc:729] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.867087 24174 raft_consensus.cc:348] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.867272 24175 consensus_queue.cc:260] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.867409 24174 raft_consensus.cc:374] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.867496 24174 raft_consensus.cc:729] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.867800 24175 ts_tablet_manager.cc:1398] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:16:48.868072 24175 tablet_bootstrap.cc:492] T f5ccd43e78574f3288b199fa14bcc0e8 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:48.867780 24174 consensus_queue.cc:260] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.868980 24174 ts_tablet_manager.cc:1398] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:48.869333 24175 tablet_bootstrap.cc:654] T f5ccd43e78574f3288b199fa14bcc0e8 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.869470 24174 tablet_bootstrap.cc:492] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
W20240502 08:16:48.870270 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_375988d4437044bc890604e6d5ab8af2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.848280 25012 ts_tablet_manager.cc:1909] T 27346a7eae6a488fa3415646262a4b88 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:48.873162 24174 tablet_bootstrap.cc:654] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.873703 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 462f77a57ebf45948cbd3dc20c271650 (DEFAULT_TABLE table=default.Table_5a6711c7a8e74b05af6d0c8fb3cfa33f [id=ffe60ec845b84b53bc4a63d604362018]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.874076 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 462f77a57ebf45948cbd3dc20c271650. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.877662 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 82c49bba3df548558123344a13c946ee (DEFAULT_TABLE table=default.Table_5af2a55b1ec14ac0ab16ca949eb002b8 [id=f316b32b17bd44049fbdaffe040eb0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.877995 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c49bba3df548558123344a13c946ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.880877 24174 tablet_bootstrap.cc:492] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:48.881009 24174 ts_tablet_manager.cc:1367] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240502 08:16:48.881990 24173 tablet_bootstrap.cc:492] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:48.882156 24173 ts_tablet_manager.cc:1367] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.113s user 0.005s sys 0.000s
I20240502 08:16:48.882875 24173 raft_consensus.cc:348] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.883056 24173 raft_consensus.cc:374] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.883102 24173 raft_consensus.cc:729] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.883332 24173 consensus_queue.cc:260] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.883924 24173 ts_tablet_manager.cc:1398] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:48.885824 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4114c8f606d84b01963b9e68bb1571f0" } new_table_name: "default.Table_5fd33bc029004a7da5c214b5ca057f93" modify_external_catalogs: true
I20240502 08:16:48.884231 24173 tablet_bootstrap.cc:492] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:48.892908 24173 tablet_bootstrap.cc:654] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.892966 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8bed20f633e348679b0087687298cc42" } modify_external_catalogs: true
I20240502 08:16:48.893158 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8bed20f633e348679b0087687298cc42" } modify_external_catalogs: true
I20240502 08:16:48.894167 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5782cc56973c43cfac93b2560c62c688" } modify_external_catalogs: true
I20240502 08:16:48.894353 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5782cc56973c43cfac93b2560c62c688" } modify_external_catalogs: true
I20240502 08:16:48.894402 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 21d2db2158d4447f92eedc5780b46056 (DEFAULT_TABLE table=default.Table_7b367902a5d84aa5ac4aa4a0b97b48f5 [id=afca80b25b6547baabe49be50efadad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.894788 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d2db2158d4447f92eedc5780b46056. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.895104 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c002eb9168244875bfe05cafb44aae75" } modify_external_catalogs: true
I20240502 08:16:48.895279 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c002eb9168244875bfe05cafb44aae75" } modify_external_catalogs: true
W20240502 08:16:48.898886 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5704dff6114142d3b27c365ab99e2277 consensus state has no leader
I20240502 08:16:48.898954 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e559ce43d2c440fa50715f25cc2c7e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.899195 23903 tablet_service.cc:1467] Processing CreateTablet for tablet e5de31a3db014615989d9e018521bfc9 (DEFAULT_TABLE table=default.Table_5889de778d2842aba3b3a02fb663425a [id=0cbbb3e6ffed4f31b5b43042cad3cffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.899277 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 56d6373938824840934211f259e14d4f (DEFAULT_TABLE table=default.Table_d3d08ffbf0a743a1ac0642fa87cf6f7e [id=4865a49282a348f1ae2ccf8f292c0672]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.899315 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fc86e7c02217467c9a9e596e55aeee08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:48.899627 24175 tablet_bootstrap.cc:492] T f5ccd43e78574f3288b199fa14bcc0e8 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:48.899742 24175 ts_tablet_manager.cc:1367] T f5ccd43e78574f3288b199fa14bcc0e8 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.003s
I20240502 08:16:48.899864 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 598cba0f8b9f4c8dae9e34872c4212fa (DEFAULT_TABLE table=default.Table_3c059ec418cd4c6dbf7bf93c0856437f [id=f29b115e70604df1bc9c0efbe6172ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.899969 23897 tablet_service.cc:1514] Processing DeleteTablet for tablet 982333cbce18439da3b35693b3f6ca0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.900107 23897 tablet_service.cc:1514] Processing DeleteTablet for tablet e549cad1b3be4b28abb0bf5af720eb52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.900218 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet b123bb81d0b44d4ea6443536168d085c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.900256 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 725982e4712e442d8ff1fd3f796e18f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.900440 23895 tablet_service.cc:1514] Processing DeleteTablet for tablet c1e746013fde4de4862f213792e235ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.900519 24175 raft_consensus.cc:348] T f5ccd43e78574f3288b199fa14bcc0e8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.900627 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a2cbe751dc4450eaa3eb60268ad510d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.900743 24175 raft_consensus.cc:374] T f5ccd43e78574f3288b199fa14bcc0e8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.900792 24175 raft_consensus.cc:729] T f5ccd43e78574f3288b199fa14bcc0e8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.900931 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c49da1e0f0435ba241a6a66526e790" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:48.900965 24175 consensus_queue.cc:260] T f5ccd43e78574f3288b199fa14bcc0e8 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.901096 23920 raft_consensus.cc:2454] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:48.901428 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2126bf2d4ab44628b52481dc2ec3a99" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:48.901584 24175 ts_tablet_manager.cc:1398] T f5ccd43e78574f3288b199fa14bcc0e8 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:48.901592 23920 raft_consensus.cc:2454] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:48.902254 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcc7169796e4ee0910f5e7084849012" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:48.902382 23920 raft_consensus.cc:2454] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:48.902690 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c072eb45e1b41eeb5aafa6d5a786944" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:48.902832 23920 raft_consensus.cc:2454] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:48.903204 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9402cf8abc4b4d893d42f9a3a99016" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:48.903357 23920 raft_consensus.cc:2454] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:48.899884 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet e9133e69e1724a26bc3650ea658ca349 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:48.903862 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5de31a3db014615989d9e018521bfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.903966 24175 tablet_bootstrap.cc:492] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:48.905603 24175 tablet_bootstrap.cc:654] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.906677 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56d6373938824840934211f259e14d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.908950 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 598cba0f8b9f4c8dae9e34872c4212fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.910688 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c49da1e0f0435ba241a6a66526e790" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:48.910823 23923 raft_consensus.cc:3037] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.911381 23919 raft_consensus.cc:3037] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.911617 23918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fcb895c8f449deb0190de75af18c3c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:48.912050 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e746013fde4de4862f213792e235ed" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:48.912200 23917 raft_consensus.cc:2454] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:48.912215 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9402cf8abc4b4d893d42f9a3a99016" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:48.912381 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2126bf2d4ab44628b52481dc2ec3a99" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:48.912515 23922 raft_consensus.cc:3037] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.912544 23916 raft_consensus.cc:3037] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.913158 23917 raft_consensus.cc:3037] T 27093862fef54c5c85d810ebeff9819c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.913506 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e549cad1b3be4b28abb0bf5af720eb52" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:48.913717 23915 raft_consensus.cc:2454] T e549cad1b3be4b28abb0bf5af720eb52 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:48.913977 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1e208733844829b13222e00a432439" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:48.914113 23915 raft_consensus.cc:2454] T 5a1e208733844829b13222e00a432439 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:48.914323 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcc7169796e4ee0910f5e7084849012" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:48.914484 23914 raft_consensus.cc:3037] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.914939 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ccd43e78574f3288b199fa14bcc0e8" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:48.915098 23915 raft_consensus.cc:2454] T f5ccd43e78574f3288b199fa14bcc0e8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:48.915361 23913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e746013fde4de4862f213792e235ed" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:48.915541 23913 raft_consensus.cc:3037] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.915500 23912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4a31e5aa834adeb9f59267247c6980" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:48.916481 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c072eb45e1b41eeb5aafa6d5a786944" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:48.916684 23788 raft_consensus.cc:2454] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:48.918243 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c17e1be6ef294769a01981f4d361be21" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:48.918486 23788 raft_consensus.cc:2379] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in this term.
I20240502 08:16:48.919414 23785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c17e1be6ef294769a01981f4d361be21" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:48.919636 23785 raft_consensus.cc:2379] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in this term.
I20240502 08:16:48.920421 23785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5768cb928e244d2914f10f4eda3c1cd" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:48.921290 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 71aa916247c449fea718b35869a6df20 (table default.Table_791a010a6e484e10a6749eefcfdc293d [id=5cee878b3a31426c879a1bf95045bb09]) successfully deleted
W20240502 08:16:48.921494 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 2411259b872e4be4b8eb8832b36094c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2411259b872e4be4b8eb8832b36094c4 already in progress: creating tablet
I20240502 08:16:48.921602 23785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2126bf2d4ab44628b52481dc2ec3a99" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:48.921764 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet bb4b4a4d6a364de8af3b54d99a0ae138 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb4b4a4d6a364de8af3b54d99a0ae138 already in progress: creating tablet
I20240502 08:16:48.923681 23785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fcb895c8f449deb0190de75af18c3c" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:48.924615 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4a31e5aa834adeb9f59267247c6980" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:48.924974 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9402cf8abc4b4d893d42f9a3a99016" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:48.926049 25066 consensus_queue.cc:1035] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:48.926393 24175 tablet_bootstrap.cc:492] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:48.926525 24175 ts_tablet_manager.cc:1367] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.000s
I20240502 08:16:48.927289 24175 raft_consensus.cc:348] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.927482 24175 raft_consensus.cc:374] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.927546 24175 raft_consensus.cc:729] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.927764 24175 consensus_queue.cc:260] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.927994 24173 tablet_bootstrap.cc:492] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:48.928155 24173 ts_tablet_manager.cc:1367] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.003s
I20240502 08:16:48.929005 24173 raft_consensus.cc:348] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.929219 24173 raft_consensus.cc:374] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.929286 24173 raft_consensus.cc:729] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.929528 24173 consensus_queue.cc:260] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.932925 25018 raft_consensus.cc:504] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.933554 25018 leader_election.cc:288] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:48.933869 24774 consensus_meta.cc:287] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent flushing consensus metadata: real 0.653s user 0.003s sys 0.000s
I20240502 08:16:48.933971 24774 raft_consensus.cc:504] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.934695 24774 leader_election.cc:288] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.935102 23924 raft_consensus.cc:2454] T 7a97427e819b4826b1ed89e6d021c8d4 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:48.938047 24175 ts_tablet_manager.cc:1398] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240502 08:16:48.938342 24175 tablet_bootstrap.cc:492] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:48.939165 25097 tablet.cc:1722] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:48.939821 24175 tablet_bootstrap.cc:654] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.940269 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_64b18a844e8d44ba8bca05606e5b34e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:48.940449 23912 raft_consensus.cc:2379] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a297a6c7c2504ca68e87b24f6691ffb2 in current term 1: Already voted for candidate 5b89a1f0485a44b7ac1710e2369a809b in this term.
I20240502 08:16:48.941104 23785 raft_consensus.cc:2379] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:16:48.945427 24173 ts_tablet_manager.cc:1398] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240502 08:16:48.945739 24173 tablet_bootstrap.cc:492] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:48.947259 24173 tablet_bootstrap.cc:654] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.953752 25018 raft_consensus.cc:482] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.951323 23996 leader_election.cc:302] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
W20240502 08:16:49.011595 23434 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 47944.6us
I20240502 08:16:48.955268 24056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4a31e5aa834adeb9f59267247c6980" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:16:48.961534 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc86e7c02217467c9a9e596e55aeee08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:49.291915 24056 raft_consensus.cc:3037] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:49.176478 24715 rpc_server.cc:307] RPC server started. Bound to: 127.22.225.125:36189
W20240502 08:16:48.961726 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64b18a844e8d44ba8bca05606e5b34e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:48.951627 23723 leader_election.cc:302] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
W20240502 08:16:49.293932 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 342320us
I20240502 08:16:48.956058 24058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fcb895c8f449deb0190de75af18c3c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:48.953941 25018 raft_consensus.cc:504] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.294621 24058 raft_consensus.cc:3037] T 96fcb895c8f449deb0190de75af18c3c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:49.295279 25018 leader_election.cc:288] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:49.176551 25142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.225.125:36189 every 8 connection(s)
I20240502 08:16:49.295631 24973 raft_consensus.cc:2781] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:49.295727 24973 raft_consensus.cc:482] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:49.295799 24973 raft_consensus.cc:3037] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.937577 25031 raft_consensus.cc:504] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.296835 25031 leader_election.cc:288] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:49.297012 25031 raft_consensus.cc:2781] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:49.297108 25031 raft_consensus.cc:686] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:49.297379 25031 consensus_queue.cc:237] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:48.982290 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31775513f3a446b08c0aac25ae0cab20 consensus state has no leader
I20240502 08:16:48.954303 25078 raft_consensus.cc:482] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.954574 25091 raft_consensus.cc:482] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:49.298528 25078 raft_consensus.cc:504] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.298552 25091 raft_consensus.cc:504] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.299263 25078 leader_election.cc:288] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:48.961905 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 710) took 1255 ms. Trace:
I20240502 08:16:49.299378 25012 rpcz_store.cc:276] 0502 08:16:47.706144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.706183 (+ 39us) service_pool.cc:225] Handling call
0502 08:16:48.226422 (+520239us) ts_tablet_manager.cc:1120] Deleting tablet 27346a7eae6a488fa3415646262a4b88
0502 08:16:48.226431 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:48.543297 (+316866us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:48.686119 (+142822us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:48.961867 (+275748us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":520088}
I20240502 08:16:48.972463 24173 tablet_bootstrap.cc:492] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:49.299793 25012 tablet_replica.cc:330] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:49.299877 24173 ts_tablet_manager.cc:1367] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.354s user 0.000s sys 0.007s
I20240502 08:16:49.300000 25012 raft_consensus.cc:2229] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:49.300081 25091 leader_election.cc:288] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:49.300120 23602 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet f4b291bdb81a4efbac7f6691e5577bbe (table default.Table_2d6b281d29d14441ac6abe9705633102 [id=73559755c4594bcfa591d054964154ae]). Expected version 2 got 0
I20240502 08:16:49.300168 25012 raft_consensus.cc:2258] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:49.300679 25012 ts_tablet_manager.cc:1875] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:48.999298 25110 tablet.cc:1722] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:49.300683 24173 raft_consensus.cc:348] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.300870 24173 raft_consensus.cc:374] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:49.300943 24173 raft_consensus.cc:729] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:49.301112 24173 consensus_queue.cc:260] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:48.881686 24174 raft_consensus.cc:348] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.301350 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319fabf91458438ba5e40ab68b323055" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:49.301460 24174 raft_consensus.cc:374] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:49.301532 24174 raft_consensus.cc:729] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:49.301590 24057 raft_consensus.cc:2454] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:49.301735 24174 consensus_queue.cc:260] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:48.993068 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 665bbee2d15b4112b651844431535d0e consensus state has no leader
I20240502 08:16:49.000326 25111 tablet.cc:1722] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
W20240502 08:16:49.302711 23861 leader_election.cc:341] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 4a57ed188315493198e95cefb6907ab6
I20240502 08:16:49.025760 24175 tablet_bootstrap.cc:492] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:49.001864 24774 raft_consensus.cc:482] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:49.302897 24175 ts_tablet_manager.cc:1367] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.365s user 0.010s sys 0.019s
I20240502 08:16:49.303011 24774 raft_consensus.cc:504] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.303599 24774 leader_election.cc:288] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:49.303544 24175 raft_consensus.cc:348] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.303709 24175 raft_consensus.cc:374] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:49.303754 24175 raft_consensus.cc:729] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:49.303972 24175 consensus_queue.cc:260] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.010499 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5b4e617d1c8949daa34aa358c1528f88" } modify_external_catalogs: true
W20240502 08:16:49.021121 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d5cf3e4cb0a9411da55bd24d9a6b60ea consensus state has no leader
I20240502 08:16:49.026758 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 642744194e22461f8316a78689fd99ff
I20240502 08:16:49.304419 24175 ts_tablet_manager.cc:1398] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:16:49.304672 24175 tablet_bootstrap.cc:492] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:48.954651 23858 leader_election.cc:341] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: d9fcc7accb1f45ad9cf55b8f64d21b91
I20240502 08:16:49.304783 23858 leader_election.cc:302] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:49.305078 23858 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 350411us
I20240502 08:16:49.305123 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_63545ead8a7a4aae8b2daa7af0fe5113" } new_table_name: "default.Table_c7de500cdfa341798dbbd1cc9ab5e4fb" modify_external_catalogs: true
I20240502 08:16:49.061913 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_850349fb9fa64ccb884825b10bbec754" } modify_external_catalogs: true
I20240502 08:16:49.068104 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet a0224317e65747e1add533d33472f7f3
W20240502 08:16:49.305280 23858 leader_election.cc:341] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: b8b1fa17dad143f0a7f1751e651eb81a
I20240502 08:16:49.070927 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet ee3d91d55ffd49ceb20d2c33408f5a5a
I20240502 08:16:49.305550 23858 leader_election.cc:302] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:49.032537 24059 raft_consensus.cc:2454] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:49.033143 23923 raft_consensus.cc:2454] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
W20240502 08:16:49.305908 23858 leader_election.cc:341] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: a3043c5ed98c401b82c6e967ca3f3613
W20240502 08:16:49.032294 24778 consensus_meta.cc:287] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent flushing consensus metadata: real 0.538s user 0.002s sys 0.000s
I20240502 08:16:49.306001 23858 leader_election.cc:302] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:49.306066 24778 raft_consensus.cc:504] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.306386 25091 raft_consensus.cc:2726] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:49.306308 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e29dfece3b88466ba6e6f6db0c1a31fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:49.306522 25091 raft_consensus.cc:2781] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:49.306581 25091 raft_consensus.cc:482] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:49.306632 25091 raft_consensus.cc:3037] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:49.306876 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e29dfece3b88466ba6e6f6db0c1a31fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:49.302908 23861 leader_election.cc:302] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:49.307176 24778 leader_election.cc:288] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:49.307197 23861 leader_election.cc:341] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 1e5de793e74247dd9e73b7b1d58027d9
I20240502 08:16:49.132339 23916 raft_consensus.cc:2454] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:49.038726 24181 ts_tablet_manager.cc:1888] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:49.308203 24181 log.cc:1196] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/b8b1fa17dad143f0a7f1751e651eb81a
I20240502 08:16:49.308611 24181 ts_tablet_manager.cc:1909] T b8b1fa17dad143f0a7f1751e651eb81a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
W20240502 08:16:49.148084 23847 kernel_stack_watchdog.cc:198] Thread 23858 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 126ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fb1f89c6980 (unknown) at ??:0
@ 0x7fb1f89c09c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7fb1f2946904 google::LogMessage::Flush() at ??:0
@ 0x7fb1f2946a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fb200ded5d3 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7fb200df012a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7fb200defeee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fb1fc2868cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7fb1fc2864ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fb1fc2724e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fb1fc272a2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fb1fc20f51a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fb1fc20df45 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fb1fc2202c5 ev::base<>::method_thunk<>() at ??:0
I20240502 08:16:49.032912 24180 ts_tablet_manager.cc:1888] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:49.309013 24180 log.cc:1196] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/5bcc7169796e4ee0910f5e7084849012
I20240502 08:16:49.309381 24180 ts_tablet_manager.cc:1909] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:49.180759 25108 raft_consensus.cc:482] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:49.309609 25108 raft_consensus.cc:504] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.310103 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet c48cea94856f43b6861c3ba360414888 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:49.310441 25108 leader_election.cc:288] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:49.310628 25108 raft_consensus.cc:2726] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:49.310770 25108 raft_consensus.cc:2781] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:49.310832 25108 raft_consensus.cc:482] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:49.310901 25108 raft_consensus.cc:3037] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:49.065114 24025 tablet_service.cc:1467] Processing CreateTablet for tablet c61243a38bfa4964b225eeae0e8cc28e (DEFAULT_TABLE table=default.Table_0e7c8c39d3c4443d8b770a580c137b2a [id=80c2e205a47743e8a24ac52b79aa246d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.074370 24026 tablet_service.cc:1467] Processing CreateTablet for tablet e4ac2b5073e34cd98e0246620fc1515d (DEFAULT_TABLE table=default.Table_f30d039709f9465e96e1280298ae2aaf [id=23ef44470f504e7d8db27dcc8b5b75b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.074553 24022 tablet_service.cc:1467] Processing CreateTablet for tablet 5101590c3db24c9ebbb45ee23c974524 (DEFAULT_TABLE table=default.Table_3ee62693a3d14323a553988181673d32 [id=05bf70b65ed94236aa5582f5ceda90c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.074661 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 11ebacd4c41941cd8a6ec634e3aeeaf5 (DEFAULT_TABLE table=default.Table_a33cd662ce1043e3853792e92624c9be [id=64c9d98633614fe78bb291ff36dc5742]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.311529 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c61243a38bfa4964b225eeae0e8cc28e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.311765 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ac2b5073e34cd98e0246620fc1515d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.218268 23914 raft_consensus.cc:2454] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:49.063237 23766 tablet_service.cc:1467] Processing CreateTablet for tablet c61243a38bfa4964b225eeae0e8cc28e (DEFAULT_TABLE table=default.Table_0e7c8c39d3c4443d8b770a580c137b2a [id=80c2e205a47743e8a24ac52b79aa246d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.314078 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c61243a38bfa4964b225eeae0e8cc28e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.315076 24175 tablet_bootstrap.cc:654] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:49.315089 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5101590c3db24c9ebbb45ee23c974524. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.223805 25076 raft_consensus.cc:504] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.316239 25076 leader_election.cc:288] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:49.220844 23913 raft_consensus.cc:2454] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:49.074849 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 8a06fc827d204d02ab81d5eeb44f7421 (DEFAULT_TABLE table=default.Table_adc39e4ac3924f079897f27ed4f1981f [id=02010c592d4a48c2a862b5c294d90d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.074903 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 12bd0a3bf7d24d939288c3eca83e1540 (DEFAULT_TABLE table=default.Table_d3cd93eb7315400aa078bbb480182814 [id=6af7860d646f4f5c9afaf744baf025c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.317739 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a06fc827d204d02ab81d5eeb44f7421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.318231 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12bd0a3bf7d24d939288c3eca83e1540. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.321269 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11ebacd4c41941cd8a6ec634e3aeeaf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.064823 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 5101590c3db24c9ebbb45ee23c974524 (DEFAULT_TABLE table=default.Table_3ee62693a3d14323a553988181673d32 [id=05bf70b65ed94236aa5582f5ceda90c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.324056 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5101590c3db24c9ebbb45ee23c974524. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.324441 24056 raft_consensus.cc:2454] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:49.075177 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 70b8cd6a22804ad79681a2921f5c12c9 (DEFAULT_TABLE table=default.Table_a662f92a7fe845a990eb07796fc9d8a6 [id=0ec249e5ea9842d5ba593c3585897b66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.327754 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b8cd6a22804ad79681a2921f5c12c9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:49.331604 25070 consensus_meta.cc:287] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent flushing consensus metadata: real 0.731s user 0.002s sys 0.000s
I20240502 08:16:49.331728 25070 raft_consensus.cc:504] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.332463 25070 leader_election.cc:288] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:49.066761 23757 tablet_service.cc:1467] Processing CreateTablet for tablet e4ac2b5073e34cd98e0246620fc1515d (DEFAULT_TABLE table=default.Table_f30d039709f9465e96e1280298ae2aaf [id=23ef44470f504e7d8db27dcc8b5b75b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.333470 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ac2b5073e34cd98e0246620fc1515d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.075975 24030 tablet_service.cc:1467] Processing CreateTablet for tablet e03c13b1cb584f5994dff77692b0f442 (DEFAULT_TABLE table=default.Table_5520c58206f4499bac98e73fb4122056 [id=bd97c826b7b44c63904a8fc981480614]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.120182 24944 raft_consensus.cc:482] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:49.337894 24944 raft_consensus.cc:504] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.338070 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03c13b1cb584f5994dff77692b0f442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.338582 24944 leader_election.cc:288] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:49.340832 24174 ts_tablet_manager.cc:1398] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.460s user 0.003s sys 0.001s
I20240502 08:16:49.341068 24174 tablet_bootstrap.cc:492] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:49.068531 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 11ebacd4c41941cd8a6ec634e3aeeaf5 (DEFAULT_TABLE table=default.Table_a33cd662ce1043e3853792e92624c9be [id=64c9d98633614fe78bb291ff36dc5742]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.341778 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11ebacd4c41941cd8a6ec634e3aeeaf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.342527 24174 tablet_bootstrap.cc:654] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:49.132107 24055 raft_consensus.cc:2454] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:49.070245 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 12bd0a3bf7d24d939288c3eca83e1540 (DEFAULT_TABLE table=default.Table_d3cd93eb7315400aa078bbb480182814 [id=6af7860d646f4f5c9afaf744baf025c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.071645 23752 tablet_service.cc:1467] Processing CreateTablet for tablet 8a06fc827d204d02ab81d5eeb44f7421 (DEFAULT_TABLE table=default.Table_adc39e4ac3924f079897f27ed4f1981f [id=02010c592d4a48c2a862b5c294d90d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.346078 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12bd0a3bf7d24d939288c3eca83e1540. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.073041 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 70b8cd6a22804ad79681a2921f5c12c9 (DEFAULT_TABLE table=default.Table_a662f92a7fe845a990eb07796fc9d8a6 [id=0ec249e5ea9842d5ba593c3585897b66]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:49.133441 24031 consensus_meta.cc:287] T 89a17261fbe0490fbe73ffe199ebb7be P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent flushing consensus metadata: real 0.811s user 0.000s sys 0.003s
I20240502 08:16:49.346385 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b8cd6a22804ad79681a2921f5c12c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.346732 24031 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 758) took 1165 ms. Trace:
I20240502 08:16:49.346813 24031 rpcz_store.cc:276] 0502 08:16:48.181197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.182154 (+ 957us) service_pool.cc:225] Handling call
0502 08:16:48.182589 (+ 435us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:48.182622 (+ 33us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:48.321857 (+139235us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.346542 (+1024685us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:49.346708 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:49.347396 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 0902a73da6e34e8e9cd59bb53a36ec0e (DEFAULT_TABLE table=default.Table_722b7dc3bd6f4bdcb2ffc44d2b77e0a1 [id=3f8d5d0f6bf04fc0a5f354c3bb24f146]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.347731 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0902a73da6e34e8e9cd59bb53a36ec0e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:49.131866 23786 consensus_meta.cc:287] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2: Time spent flushing consensus metadata: real 0.571s user 0.003s sys 0.000s
I20240502 08:16:49.349756 24175 tablet_bootstrap.cc:492] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
W20240502 08:16:49.139309 24037 consensus_meta.cc:287] T 4eed19c1aa7c455eb23bad90c3df0b46 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent flushing consensus metadata: real 0.786s user 0.002s sys 0.000s
I20240502 08:16:49.349702 23786 raft_consensus.cc:2454] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:49.349880 24175 ts_tablet_manager.cc:1367] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.005s
I20240502 08:16:49.350064 24037 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 757) took 1168 ms. Trace:
W20240502 08:16:49.133288 23767 consensus_meta.cc:287] T 46e577132e5249b28426bfe38a28635a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent flushing consensus metadata: real 0.852s user 0.002s sys 0.000s
I20240502 08:16:49.350143 24037 rpcz_store.cc:276] 0502 08:16:48.181100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.181432 (+ 332us) service_pool.cc:225] Handling call
0502 08:16:48.181937 (+ 505us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:48.181988 (+ 51us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:48.352385 (+170397us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.349913 (+997528us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:49.350046 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:49.350452 23767 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 777) took 1305 ms. Trace:
I20240502 08:16:49.350535 23767 rpcz_store.cc:276] 0502 08:16:48.045201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.045312 (+ 111us) service_pool.cc:225] Handling call
0502 08:16:48.046067 (+ 755us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:48.046098 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:48.281215 (+235117us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.350274 (+1069059us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:49.350433 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:49.350579 24037 tablet_service.cc:1467] Processing CreateTablet for tablet fe82f5f26d7f423496858f889d65c387 (DEFAULT_TABLE table=default.Table_31208fdb293844fc838187320c87cdf9 [id=1e0c30a309b14de39df424fc9b28c7db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.350747 24175 raft_consensus.cc:348] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.350874 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe82f5f26d7f423496858f889d65c387. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.350916 24175 raft_consensus.cc:374] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:49.350982 24175 raft_consensus.cc:729] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:49.350970 23767 tablet_service.cc:1467] Processing CreateTablet for tablet e03c13b1cb584f5994dff77692b0f442 (DEFAULT_TABLE table=default.Table_5520c58206f4499bac98e73fb4122056 [id=bd97c826b7b44c63904a8fc981480614]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.351297 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03c13b1cb584f5994dff77692b0f442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.351203 24175 consensus_queue.cc:260] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.352782 24175 ts_tablet_manager.cc:1398] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240502 08:16:49.143060 24027 consensus_meta.cc:287] T 2da25b412cec4b0a90bce47f977cafd1 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent flushing consensus metadata: real 0.635s user 0.005s sys 0.000s
I20240502 08:16:49.354075 24027 tablet_service.cc:1467] Processing CreateTablet for tablet d609a55e47ad420098d279835e9aa090 (DEFAULT_TABLE table=default.Table_346dec12764b4564b56f28b373f24b0b [id=78d93afb51c7474bacdf062b5049f5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.354357 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d609a55e47ad420098d279835e9aa090. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:49.148100 24023 consensus_meta.cc:287] T 1638e78b6f544209bd8d6da6556fbce4 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent flushing consensus metadata: real 0.625s user 0.002s sys 0.000s
W20240502 08:16:49.163568 23722 service_pool.cc:135] CreateTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:43544 dropped due to backpressure. The service queue is full; it has 50 items.
I20240502 08:16:49.357623 24023 tablet_service.cc:1467] Processing CreateTablet for tablet c96c93bb20744ee78a609742b2cbab2a (DEFAULT_TABLE table=default.Table_a2055a6e13d8495b83033c1f6596b54f [id=344c30da8e9e4615ba703b4a879ad979]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.358204 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c96c93bb20744ee78a609742b2cbab2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.357848 23722 service_pool.cc:138] CreateTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:43544 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 828)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 829)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 830)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 831)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 832)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 833)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 834)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 1W20240502 08:16:49.150555 24039 consensus_meta.cc:287] T 4bea89eccaec4d049c0ec3900ed600c2 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent flushing consensus metadata: real 0.624s user 0.002s sys 0.000s
W20240502 08:16:49.151469 24036 consensus_meta.cc:287] T c6f15b4b87a04707a3b4f37538e3b64a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent flushing consensus metadata: real 0.618s user 0.000s sys 0.002s
I20240502 08:16:49.362075 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 02c50800505247cf967607bbfcca6302 (DEFAULT_TABLE table=default.Table_bbc5d6ae8e7246129746580755d1a812 [id=6cf60892656a4fe4bda6505a8de87742]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.362182 24036 tablet_service.cc:1467] Processing CreateTablet for tablet d5eead3736fb4b34b25850b248d0d62b (DEFAULT_TABLE table=default.Table_8dbc74ff11934c76b732768528c3ed39 [id=b7a577f1b904421fb27312ba8e350d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.362397 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c50800505247cf967607bbfcca6302. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:49.152885 24038 consensus_meta.cc:287] T 462f77a57ebf45948cbd3dc20c271650 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent flushing consensus metadata: real 0.613s user 0.001s sys 0.000s
I20240502 08:16:49.362990 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 9db504396f7546a89e74b6f4530b70cc (DEFAULT_TABLE table=default.Table_4b0a59aad44148b6b3656ff66dfca548 [id=46be1f5fe512486f8e8d9d6e13165f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.363317 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db504396f7546a89e74b6f4530b70cc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:49.153522 24021 consensus_meta.cc:287] T ee3d91d55ffd49ceb20d2c33408f5a5a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent flushing consensus metadata: real 0.634s user 0.003s sys 0.000s
I20240502 08:16:49.366061 24021 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0ff1c3818a49d2b13f57ea6a81c3d6 (DEFAULT_TABLE table=default.Table_ddf5aa41088d4de4bcf2768d82c7d1fb [id=e9a2a5d8e81e4163b64dc4da67ed09c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.366369 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a0ff1c3818a49d2b13f57ea6a81c3d6. 1 dirs total, 0 dirs full, 0 dirs failed
27.0.0.1:43544 (request call id 835)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 836)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 837)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 838)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 839)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 840)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 841)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 842)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 843)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 844)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 845)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 846)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 847)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 848)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 849)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 850)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 851)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 852)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 853)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 854)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 855)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 856)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 857)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 858)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 859)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 860)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 861)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 862)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 863)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 864)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 865)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 866)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 867)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 868)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 869)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 870)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 871)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 872)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 873)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 874)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 875)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 876)
W20240502 08:16:49.155301 24035 consensus_meta.cc:287] T 21d2db2158d4447f92eedc5780b46056 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent flushing consensus metadata: real 0.614s user 0.001s sys 0.000s
I20240502 08:16:49.388974 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 87a3df4fbb224dd0b4ec6141584d374b (DEFAULT_TABLE table=default.Table_cce64bbdca4946bfb1cdbd52d1d2d59c [id=8d575e6ea894499a848dc9e39a3e1b23]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.388654 23722 service_pool.cc:138] CreateTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:43544 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 830)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 831)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 832)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 833)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 834)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 835)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 836)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 837)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 838)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 839)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 840)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 841)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 842)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 843)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 844)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 845)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 846)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 847)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 848)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 849)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 850)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 851)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 852)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 853)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 854)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 855)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 856)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 857)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 858)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 859)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 860)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 861)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 862)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 863)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 864)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 865)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 866)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 867)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 868)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 869)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 870)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 871)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 872)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 873)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 874)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 875)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 876)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 878)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 879)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 880)
W20240502 08:16:49.157783 24034 consensus_meta.cc:287] T 82c49bba3df548558123344a13c946ee P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent flushing consensus metadata: real 0.615s user 0.003s sys 0.000s
I20240502 08:16:49.158386 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8eccf0ea5e43819f0b8fe364c4749d (DEFAULT_TABLE table=default.Table_a92dfc1659cf4b24a49358808fd9663b [id=44cf976481e048f79a382ed36b30446b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.367219 23918 raft_consensus.cc:2379] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a297a6c7c2504ca68e87b24f6691ffb2 in current term 1: Already voted for candidate 5b89a1f0485a44b7ac1710e2369a809b in this term.
I20240502 08:16:49.160440 24040 tablet_service.cc:1467] Processing CreateTablet for tablet bf9feb5c42cd4d8b9d265742813f798c (DEFAULT_TABLE table=default.Table_9e3aabcb11df4be58f87a1f242912dec [id=c06de80716064c6b80ab6283e4385713]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.163820 25029 raft_consensus.cc:482] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:16:49.166580 23994 service_pool.cc:135] CreateTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:51782 dropped due to backpressure. The service queue is full; it has 50 items.
I20240502 08:16:49.384145 23429 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2vVFYb/build/asan/bin/kudu as pid 24715
I20240502 08:16:49.389870 25029 raft_consensus.cc:504] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.389468 23722 service_pool.cc:138] CreateTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:43544 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 830)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 831)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 832)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 833)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 834)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 835)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 836)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 837)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 838)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 839)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 840)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 841)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 842)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 843)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 844)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 845)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 846)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 847)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 848)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 849)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 850)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 851)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 852)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 853)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 854)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 855)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 856)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 857)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 858)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 859)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 860)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 861)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 862)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 863)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 864)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 865)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 866)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 867)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 868)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 869)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 870)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 871)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 872)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 873)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 874)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 875)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 876)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 878)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 879)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 880)
W20240502 08:16:49.390484 23996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 439173us
I20240502 08:16:49.390321 23722 service_pool.cc:138] CreateTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:43544 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 830)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 831)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 832)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 833)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 834)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 835)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 836)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 837)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 838)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 839)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 840)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 841)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 842)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 843)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 844)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 845)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 846)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 847)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 848)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 849)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 850)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 851)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 852)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 853)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 854)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 855)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 856)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 857)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 858)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 859)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 860)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 861)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 862)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 863)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 864)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 865)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 866)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 867)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 868)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 869)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 870)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 871)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 872)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 873)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 874)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 875)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 876)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 878)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 879)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 880)
I20240502 08:16:49.391193 23722 service_pool.cc:138] CreateTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:43544 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 830)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 831)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 832)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 833)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 834)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 835)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 836)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 837)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 838)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 839)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 840)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 841)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 842)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 843)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 844)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 845)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 846)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 847)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 848)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 849)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 850)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 851)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 852)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 853)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 854)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 855)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 856)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 857)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 858)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 859)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 860)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 861)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 862)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 863)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 864)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 865)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 866)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 867)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 868)I20240502 08:16:49.430418 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1fa44e7d541041d3a86bed4be93b5a03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 869)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 870)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 871)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 872)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 873)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 874)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 875)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 876)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 878)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 879)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 880)
I20240502 08:16:49.383603 24715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-1/data/info.pb
W20240502 08:16:49.258358 23982 kernel_stack_watchdog.cc:198] Thread 23996 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 270ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f9187faf980 (unknown) at ??:0
@ 0x7f9180dd1255 __write at ??:0
@ 0x7f9180d4c28d _IO_file_write at ??:0
@ 0x7f9180d4cbff _IO_file_xsputn at ??:0
@ 0x7f9180d40a57 fwrite at ??:0
@ 0x8793e7 __interceptor_fwrite.part.56 at ??:?
@ 0x7f9181f338a0 google::LogMessage::SendToLog() at ??:0
@ 0x7f9181f2f7cc google::LogMessage::Flush() at ??:0
@ 0x7f9181f2fa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f91903d53f0 kudu::consensus::LeaderElection::CheckForDecision() at ??:0
@ 0x7f91903d6782 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f91903d912a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f91903d8eee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f918b86f8cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f918b86f4ae std::_Function_handler<>::_M_invoke() at ??:0
I20240502 08:16:49.369727 25076 raft_consensus.cc:482] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:49.667102 25076 raft_consensus.cc:504] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.666642 23722 service_pool.cc:138] DeleteTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:43544 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 834)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 835)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 836)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 837)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 838)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 839)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 840)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 841)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 842)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 843)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 844)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 845)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 846)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 847)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 848)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 849)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 850)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 851)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 852)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 853)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 854)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 855)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 856)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 857)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 858)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 859)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 860)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 861)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 862)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 863)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 864)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 865)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 866)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 867)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 868)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 869)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 870)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 871)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 872)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 873)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 874)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 875)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 876)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 878)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 879)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 880)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 885)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 886)
Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:43544 (request call id 887)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 888)
I20240502 08:16:49.667946 25076 leader_election.cc:288] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:49.431264 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ff75818c766d43569443cacbfcb972c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:49.667804 23722 service_pool.cc:138] DeleteTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:43544 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 834)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 835)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 836)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 837)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 838)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 839)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 840)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 841)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 842)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 843)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 844)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 845)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 846)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 847)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 848)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 849)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 850)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 851)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 852)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 853)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 854)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 855)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 856)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 857)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 858)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 859)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 860)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 861)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 862)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 863)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 864)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 865)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 866)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 867)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 868)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 869)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 870)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 871)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 872)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 873)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 874)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 875)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 876)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 878)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 879)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 880)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 885)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 886)
Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:43544 (request call id 887)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 888)
W20240502 08:16:49.668620 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff75818c766d43569443cacbfcb972c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:49.668860 23722 service_pool.cc:138] CreateTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:43544 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 834)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 835)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 836)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 837)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 838)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 839)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 840)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 841)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 842)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 843)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 844)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 845)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 846)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 847)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 848)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 849)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 850)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 851)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 852)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 853)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 854)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 855)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 856)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 857)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 858)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 859)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 860)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 861)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 862)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 863)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 864)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 865)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 866)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 867)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 868)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 869)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 870)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 871)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 872)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 873)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 874)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 875)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 876)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 878)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 879)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 880)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 885)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 886)
Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:43544 (request call id 887)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 888)
I20240502 08:16:49.220857 25021 raft_consensus.cc:504] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:49.224859 23764 consensus_meta.cc:287] T c44ce35c463f416794edccc7149aa0cf P a297a6c7c2504ca68e87b24f6691ffb2: Time spent flushing consensus metadata: real 0.768s user 0.003s sys 0.001s
I20240502 08:16:49.670210 23764 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 799) took 1221 ms. Trace:
I20240502 08:16:49.669857 23722 service_pool.cc:138] DeleteTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:43544 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 834)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 835)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 836)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 837)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 838)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 839)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 840)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 841)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 842)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 843)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 844)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 845)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 846)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 847)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 848)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 849)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 850)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 851)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 852)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 853)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 854)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 855)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 856)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 857)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 858)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 859)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 860)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 861)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 862)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 863)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 864)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 865)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 866)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 867)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 868)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 869)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 870)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 871)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 872)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 873)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 874)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 875)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 876)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 878)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 879)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 880)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 885)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 886)
Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:43544 (request call id 887)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 888)
I20240502 08:16:49.670393 25021 leader_election.cc:288] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:49.670306 23764 rpcz_store.cc:276] 0502 08:16:48.449131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.449314 (+ 183us) service_pool.cc:225] Handling call
0502 08:16:48.453766 (+ 4452us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:48.453792 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:48.456878 (+ 3086us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.670036 (+1213158us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:49.670192 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:49.369459 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5eead3736fb4b34b25850b248d0d62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.670768 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 02c50800505247cf967607bbfcca6302 (DEFAULT_TABLE table=default.Table_bbc5d6ae8e7246129746580755d1a812 [id=6cf60892656a4fe4bda6505a8de87742]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:49.235579 23763 consensus_meta.cc:287] T 462f77a57ebf45948cbd3dc20c271650 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent flushing consensus metadata: real 0.707s user 0.006s sys 0.000s
I20240502 08:16:49.671319 23763 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 808) took 1146 ms. Trace:
I20240502 08:16:49.671384 23763 rpcz_store.cc:276] 0502 08:16:48.524420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.525312 (+ 892us) service_pool.cc:225] Handling call
0502 08:16:48.525695 (+ 383us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:48.525726 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:48.528904 (+ 3178us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.671180 (+1142276us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:49.671303 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:16:49.235569 23755 consensus_meta.cc:287] T 82c49bba3df548558123344a13c946ee P a297a6c7c2504ca68e87b24f6691ffb2: Time spent flushing consensus metadata: real 0.706s user 0.004s sys 0.000s
I20240502 08:16:49.672286 23755 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 809) took 1146 ms. Trace:
I20240502 08:16:49.672365 23755 rpcz_store.cc:276] 0502 08:16:48.525467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.525848 (+ 381us) service_pool.cc:225] Handling call
0502 08:16:48.526221 (+ 373us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:48.526247 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:48.529335 (+ 3088us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.672161 (+1142826us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:49.672272 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:16:49.243841 23758 consensus_meta.cc:287] T e5de31a3db014615989d9e018521bfc9 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent flushing consensus metadata: real 0.713s user 0.002s sys 0.000s
I20240502 08:16:49.672842 23758 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 811) took 1144 ms. Trace:
I20240502 08:16:49.672909 23758 rpcz_store.cc:276] 0502 08:16:48.528035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.528084 (+ 49us) service_pool.cc:225] Handling call
0502 08:16:48.528450 (+ 366us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:48.528471 (+ 21us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:48.531140 (+ 2669us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.672707 (+1141567us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:49.672827 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:16:49.221637 23753 consensus_meta.cc:287] T c84b27ed8cf7429ca720e4cb8d396342 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent flushing consensus metadata: real 1.097s user 0.003s sys 0.000s
I20240502 08:16:49.673346 23753 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 770) took 1742 ms. Trace:
I20240502 08:16:49.673405 23753 rpcz_store.cc:276] 0502 08:16:47.931189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.941523 (+ 10334us) service_pool.cc:225] Handling call
0502 08:16:47.949776 (+ 8253us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:47.949807 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:48.117204 (+167397us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.673235 (+1556031us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:49.673333 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:49.378187 23861 leader_election.cc:302] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:49.358803 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a06fc827d204d02ab81d5eeb44f7421. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:49.673784 23861 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 295609us
I20240502 08:16:49.674301 25159 raft_consensus.cc:2781] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:49.674430 25159 raft_consensus.cc:686] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:16:49.674676 25159 consensus_queue.cc:237] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.670693 23722 service_pool.cc:138] DeleteTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:43544 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 835)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 836)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 837)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 838)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 839)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 840)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 841)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 842)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 843)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 844)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 845)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 846)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 847)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 848)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 849)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 850)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 851)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 852)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 853)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 854)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 855)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 856)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 857)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 858)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 859)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 860)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 861)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 862)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 863)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 864)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 865)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 866)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 867)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 868)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 869)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 870)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 871)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 872)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 873)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 874)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 875)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 876)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 878)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 879)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 880)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 885)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 886)
Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:43544 (request call id 887)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 888)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 893)
I20240502 08:16:49.676544 23722 service_pool.cc:138] CreateTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:43544 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 835)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 836)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 837)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 838)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 839)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 840)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 841)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 842)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 843)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 844)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 845)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 846)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 847)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 848)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 849)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 850)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 851)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 852)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 853)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 854)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 855)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 856)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 857)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 858)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 859)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 860)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 861)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 862)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 863)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 864)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 865)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 866)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 867)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 868)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 869)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 870)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 871)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 872)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 873)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 874)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 875)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 876)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 878)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 879)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 880)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 885)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 886)
Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:43544 (request call id 887)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 888)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 893)
W20240502 08:16:49.233458 23749 consensus_meta.cc:287] T c6f15b4b87a04707a3b4f37538e3b64a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent flushing consensus metadata: real 0.710s user 0.003s sys 0.000s
I20240502 08:16:49.433198 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5b4e617d1c8949daa34aa358c1528f88" } modify_external_catalogs: true
I20240502 08:16:49.677845 23749 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 807) took 1158 ms. Trace:
I20240502 08:16:49.677908 23749 rpcz_store.cc:276] 0502 08:16:48.519604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.519657 (+ 53us) service_pool.cc:225] Handling call
0502 08:16:48.520087 (+ 430us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:48.520115 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:48.523070 (+ 2955us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.677689 (+1154619us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:49.677827 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:49.378472 23993 leader_election.cc:302] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
W20240502 08:16:49.231436 23750 consensus_meta.cc:287] T 1638e78b6f544209bd8d6da6556fbce4 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent flushing consensus metadata: real 0.716s user 0.002s sys 0.000s
I20240502 08:16:49.678921 23755 tablet_service.cc:1467] Processing CreateTablet for tablet d5eead3736fb4b34b25850b248d0d62b (DEFAULT_TABLE table=default.Table_8dbc74ff11934c76b732768528c3ed39 [id=b7a577f1b904421fb27312ba8e350d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.679001 23750 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 806) took 1167 ms. Trace:
I20240502 08:16:49.679067 23750 rpcz_store.cc:276] 0502 08:16:48.511168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.511218 (+ 50us) service_pool.cc:225] Handling call
0502 08:16:48.511609 (+ 391us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:48.511631 (+ 22us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:48.514924 (+ 3293us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.678879 (+1163955us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:49.678986 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:16:49.679114 23993 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 300665us
I20240502 08:16:49.679273 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5eead3736fb4b34b25850b248d0d62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.679482 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 9db504396f7546a89e74b6f4530b70cc (DEFAULT_TABLE table=default.Table_4b0a59aad44148b6b3656ff66dfca548 [id=46be1f5fe512486f8e8d9d6e13165f8e]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:49.423328 23922 consensus_meta.cc:287] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent flushing consensus metadata: real 0.511s user 0.003s sys 0.000s
I20240502 08:16:49.679816 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db504396f7546a89e74b6f4530b70cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.679889 23922 raft_consensus.cc:2454] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:49.680510 23918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797d3c6bdc4493398954fbbf9943971" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:49.680799 25018 raft_consensus.cc:482] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:49.680898 25018 raft_consensus.cc:504] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.681510 25018 leader_election.cc:288] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:49.223330 24891 raft_consensus.cc:504] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.681855 23788 raft_consensus.cc:2379] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:16:49.682065 24891 leader_election.cc:288] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:49.682255 25154 raft_consensus.cc:2781] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:49.682878 24944 raft_consensus.cc:482] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:49.433640 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_f14ecd9121174b1db356ecd80bbac528"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:49.173280 23709 kernel_stack_watchdog.cc:198] Thread 23723 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 113ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f106bbe6980 (unknown) at ??:0
@ 0x7f1064a08255 __write at ??:0
@ 0x7f106498328d _IO_file_write at ??:0
@ 0x7f1064983bff _IO_file_xsputn at ??:0
@ 0x7f1064977a57 fwrite at ??:0
@ 0x8793e7 __interceptor_fwrite.part.56 at ??:?
@ 0x7f1065b6a8a0 google::LogMessage::SendToLog() at ??:0
@ 0x7f1065b667cc google::LogMessage::Flush() at ??:0
@ 0x7f1065b66a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f107400c3f0 kudu::consensus::LeaderElection::CheckForDecision() at ??:0
@ 0x7f107400d782 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f107401012a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f107400feee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f106f4a68cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f106f4a64ae std::_Function_handler<>::_M_invoke() at ??:0
W20240502 08:16:49.229336 23751 consensus_meta.cc:287] T 2da25b412cec4b0a90bce47f977cafd1 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent flushing consensus metadata: real 0.732s user 0.003s sys 0.000s
W20240502 08:16:49.425837 23858 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 119924us
I20240502 08:16:49.699084 23751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 802) took 1205 ms. Trace:
W20240502 08:16:49.699154 23858 leader_election.cc:341] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 4f1736b496354e2dadd844242d3867f8
I20240502 08:16:49.699159 23751 rpcz_store.cc:276] 0502 08:16:48.493296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.493352 (+ 56us) service_pool.cc:225] Handling call
0502 08:16:48.493800 (+ 448us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:48.493826 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:48.497161 (+ 3335us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.698929 (+1201768us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:49.699066 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:49.699270 23858 leader_election.cc:302] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:49.699530 25076 raft_consensus.cc:2726] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:49.699560 23751 tablet_service.cc:1467] Processing CreateTablet for tablet e4a87654d97249418e5df3a99b23c557 (DEFAULT_TABLE table=default.Table_74a4ec7454a14f74ad2c1dc97fab2080 [id=ab048bd406f141769d88fcc99b5307e0]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:49.699659 23858 leader_election.cc:341] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: b8b1fa17dad143f0a7f1751e651eb81a
I20240502 08:16:49.218286 23787 raft_consensus.cc:2454] T 7a97427e819b4826b1ed89e6d021c8d4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:49.451606 24175 tablet_bootstrap.cc:492] T 1ba1baa2279045b3adc58d187a12cbf3 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:49.700110 23858 leader_election.cc:341] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: e686db13bb7541c69ae88f0fb7e27061
I20240502 08:16:49.700203 23858 leader_election.cc:302] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:49.373345 23785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c377e037894728ac084b6500ad20de" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:49.797224 25184 sys_catalog.cc:263] Verifying existing consensus state
I20240502 08:16:49.700512 23858 leader_election.cc:302] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:49.854674 23858 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 154159us
I20240502 08:16:49.854977 23858 leader_election.cc:302] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:49.855502 23858 leader_election.cc:302] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:49.682992 24944 raft_consensus.cc:504] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.856473 24944 leader_election.cc:288] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:49.683478 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f14ecd9121174b1db356ecd80bbac528 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:49.237735 23754 consensus_meta.cc:287] T 21d2db2158d4447f92eedc5780b46056 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent flushing consensus metadata: real 0.708s user 0.000s sys 0.002s
I20240502 08:16:49.858650 23754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 810) took 1332 ms. Trace:
I20240502 08:16:49.858742 23754 rpcz_store.cc:276] 0502 08:16:48.526373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.526427 (+ 54us) service_pool.cc:225] Handling call
0502 08:16:48.526792 (+ 365us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:48.526811 (+ 19us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:48.529701 (+ 2890us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.858420 (+1328719us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:49.858620 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:49.859282 23754 tablet_service.cc:1467] Processing CreateTablet for tablet a5b5880bf09c47da9f5e19d86ef976d4 (DEFAULT_TABLE table=default.Table_664d4018b9074906bd16eb6782673acc [id=9097856ad6c24397a1385fa39e5da478]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:49.520484 23898 consensus_meta.cc:287] T 598cba0f8b9f4c8dae9e34872c4212fa P 5b89a1f0485a44b7ac1710e2369a809b: Time spent flushing consensus metadata: real 0.608s user 0.003s sys 0.000s
I20240502 08:16:49.860129 25159 raft_consensus.cc:2726] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:16:49.860270 25159 raft_consensus.cc:2726] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:49.860378 25159 raft_consensus.cc:2726] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:49.380613 24056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fcb895c8f449deb0190de75af18c3c" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:16:49.442270 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fba8f6e017e848d59239ae98432b1837 consensus state has no leader
I20240502 08:16:49.863777 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d20a3b8ecb6449a5a4cc9837354619b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:49.864356 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d20a3b8ecb6449a5a4cc9837354619b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:49.864939 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a4104f4cb64889ad66810788ad6b29" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:49.865154 23787 raft_consensus.cc:2454] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
W20240502 08:16:49.227165 23762 consensus_meta.cc:287] T 69246bf9f6be42b9a0671ae68a633255 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent flushing consensus metadata: real 0.765s user 0.002s sys 0.000s
I20240502 08:16:49.865697 23762 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 800) took 1416 ms. Trace:
I20240502 08:16:49.865775 23762 rpcz_store.cc:276] 0502 08:16:48.449263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.449549 (+ 286us) service_pool.cc:225] Handling call
0502 08:16:48.450586 (+ 1037us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:48.450621 (+ 35us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:48.462115 (+ 11494us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.865497 (+1403382us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:49.865679 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:49.866204 23762 tablet_service.cc:1467] Processing CreateTablet for tablet e20e126b1c9546e4a194bf9842f444a5 (DEFAULT_TABLE table=default.Table_769de68591d64bdb9b80f7bb337da552 [id=f8b12b3bb4904c18be60b567cb05b739]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.614969 25091 raft_consensus.cc:504] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.866973 23993 leader_election.cc:302] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:49.867329 24944 raft_consensus.cc:2781] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:49.867408 24944 raft_consensus.cc:482] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:49.867471 24944 raft_consensus.cc:3037] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:49.867607 25091 leader_election.cc:288] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:49.389793 23994 service_pool.cc:138] CreateTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:51782 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 807)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 808)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 809)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 810)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 811)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 812)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 813)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 814)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 815)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 816)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 817)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 818)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 819)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 820)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 821)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 822)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 823)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 824)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 825)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 826)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 827)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 828)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 829)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 830)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 831)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 832)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 833)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 834)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 835)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 836)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 837)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 838)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 839)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 840)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 841)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 842)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 843)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 844)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 845)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 846)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 847)
I20240502 08:16:49.443938 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet c48cea94856f43b6861c3ba360414888
I20240502 08:16:49.870061 23609 catalog_manager.cc:6554] ReplaceTablet: tablet c48cea94856f43b6861c3ba360414888 deleted and replaced by tablet e34873a3f08043098f2740641b3dc41f
I20240502 08:16:49.870163 23913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a4104f4cb64889ad66810788ad6b29" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:49.870514 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1014ad7e82a245ea801b1866b929b1c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:49.871145 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1014ad7e82a245ea801b1866b929b1c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:49.498831 24037 tablet_service.cc:1467] Processing CreateTablet for tablet e0bd774640464d2a92b2e71126c83883 (DEFAULT_TABLE table=default.Table_c456616beb454b6ea33638f85512f97e [id=aa67c143e5cb44078846d98c0cf1345e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.468302 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 1fe597986ce945aba8f2bb19d46bb7ad (DEFAULT_TABLE table=default.Table_e20e3a08238f428fa6c14fa84e67444a [id=0c2788ca8ee44843b047cd3c8786e6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.373132 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797d3c6bdc4493398954fbbf9943971" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:49.488193 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 336f0eb5085546dba763e0d5c5dd5324 (DEFAULT_TABLE table=default.Table_cda7da2a134f43a8a3c79a6609c3e52e [id=07050bd496e84a97a86fdc4f0b7811ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.872519 23786 raft_consensus.cc:3037] T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:49.693711 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c377e037894728ac084b6500ad20de" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:49.873142 24053 raft_consensus.cc:2454] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:49.424774 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 725) took 2327 ms. Trace:
I20240502 08:16:49.696691 24107 heartbeater.cc:347] Connected to a master server at 127.22.225.125:36189
I20240502 08:16:49.620398 25012 ts_tablet_manager.cc:1888] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:49.874987 24181 rpcz_store.cc:276] 0502 08:16:47.097260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.097319 (+ 59us) service_pool.cc:225] Handling call
0502 08:16:48.810385 (+1713066us) ts_tablet_manager.cc:1120] Deleting tablet b8b1fa17dad143f0a7f1751e651eb81a
0502 08:16:48.810398 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:48.941752 (+131354us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.038591 (+ 96839us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.424748 (+386157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1712862}
W20240502 08:16:49.233222 23759 consensus_meta.cc:287] T ee3d91d55ffd49ceb20d2c33408f5a5a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent flushing consensus metadata: real 0.729s user 0.003s sys 0.000s
I20240502 08:16:49.893644 24181 tablet_replica.cc:330] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:49.696990 25157 consensus_queue.cc:1035] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:49.893939 23759 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 803) took 1394 ms. Trace:
I20240502 08:16:49.894021 23759 rpcz_store.cc:276] 0502 08:16:48.499113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.499171 (+ 58us) service_pool.cc:225] Handling call
0502 08:16:48.499582 (+ 411us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:48.499609 (+ 27us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:48.503778 (+ 4169us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.893753 (+1389975us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:49.893915 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:49.894490 23759 tablet_service.cc:1467] Processing CreateTablet for tablet e546c1bb78a848ad8125e8e8b08434db (DEFAULT_TABLE table=default.Table_843f461368e94ea9b6b9244506423221 [id=687f28ef29324002821616ba0044d1e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.405367 25029 leader_election.cc:288] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:49.406477 25066 raft_consensus.cc:482] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:49.894894 25066 raft_consensus.cc:504] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.895606 25066 leader_election.cc:288] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:49.228412 23760 consensus_meta.cc:287] T 4bea89eccaec4d049c0ec3900ed600c2 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent flushing consensus metadata: real 0.765s user 0.001s sys 0.000s
I20240502 08:16:49.896047 23760 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 798) took 1450 ms. Trace:
I20240502 08:16:49.896134 23760 rpcz_store.cc:276] 0502 08:16:48.445203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.445265 (+ 62us) service_pool.cc:225] Handling call
0502 08:16:48.445832 (+ 567us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:48.445864 (+ 32us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:48.462768 (+ 16904us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.895860 (+1433092us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:49.896027 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:49.896626 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 336f0eb5085546dba763e0d5c5dd5324 (DEFAULT_TABLE table=default.Table_cda7da2a134f43a8a3c79a6609c3e52e [id=07050bd496e84a97a86fdc4f0b7811ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.379669 25070 raft_consensus.cc:482] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:49.896991 25070 raft_consensus.cc:504] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.897648 25070 leader_election.cc:288] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:49.374536 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8eccf0ea5e43819f0b8fe364c4749d (DEFAULT_TABLE table=default.Table_a92dfc1659cf4b24a49358808fd9663b [id=44cf976481e048f79a382ed36b30446b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.376004 23767 tablet_service.cc:1467] Processing CreateTablet for tablet bf9feb5c42cd4d8b9d265742813f798c (DEFAULT_TABLE table=default.Table_9e3aabcb11df4be58f87a1f242912dec [id=c06de80716064c6b80ab6283e4385713]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.458438 24973 raft_consensus.cc:504] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.899164 24973 leader_election.cc:288] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:49.459398 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 783) took 1297 ms. Trace:
I20240502 08:16:49.899315 24180 rpcz_store.cc:276] 0502 08:16:48.161802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.161854 (+ 52us) service_pool.cc:225] Handling call
0502 08:16:48.841518 (+679664us) ts_tablet_manager.cc:1120] Deleting tablet 5bcc7169796e4ee0910f5e7084849012
0502 08:16:48.841527 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:48.938807 (+ 97280us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.032779 (+ 93972us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.459366 (+426587us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":679472}
I20240502 08:16:49.899682 24180 tablet_replica.cc:330] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:49.899869 24180 raft_consensus.cc:2229] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:49.899994 24180 raft_consensus.cc:2258] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:49.900540 24180 ts_tablet_manager.cc:1875] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:49.464478 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 0902a73da6e34e8e9cd59bb53a36ec0e (DEFAULT_TABLE table=default.Table_722b7dc3bd6f4bdcb2ffc44d2b77e0a1 [id=3f8d5d0f6bf04fc0a5f354c3bb24f146]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.467305 23756 tablet_service.cc:1467] Processing CreateTablet for tablet fe82f5f26d7f423496858f889d65c387 (DEFAULT_TABLE table=default.Table_31208fdb293844fc838187320c87cdf9 [id=1e0c30a309b14de39df424fc9b28c7db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.470553 23766 tablet_service.cc:1467] Processing CreateTablet for tablet c96c93bb20744ee78a609742b2cbab2a (DEFAULT_TABLE table=default.Table_a2055a6e13d8495b83033c1f6596b54f [id=344c30da8e9e4615ba703b4a879ad979]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.470676 23757 tablet_service.cc:1467] Processing CreateTablet for tablet d609a55e47ad420098d279835e9aa090 (DEFAULT_TABLE table=default.Table_346dec12764b4564b56f28b373f24b0b [id=78d93afb51c7474bacdf062b5049f5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.471422 24778 raft_consensus.cc:482] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:49.509439 25158 raft_consensus.cc:482] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:49.541301 24174 tablet_bootstrap.cc:492] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:49.900841 24778 raft_consensus.cc:504] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.901011 24174 ts_tablet_manager.cc:1367] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.560s user 0.025s sys 0.033s
I20240502 08:16:49.900895 25158 raft_consensus.cc:504] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.901377 24778 leader_election.cc:288] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:49.901501 25158 leader_election.cc:288] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:49.545646 23835 heartbeater.cc:347] Connected to a master server at 127.22.225.125:36189
I20240502 08:16:49.901686 24174 raft_consensus.cc:348] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.901937 24174 raft_consensus.cc:374] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:49.901997 24174 raft_consensus.cc:729] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:49.902143 24174 consensus_queue.cc:260] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.902572 24174 ts_tablet_manager.cc:1398] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:16:49.902812 24174 tablet_bootstrap.cc:492] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:49.595100 24774 raft_consensus.cc:482] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:49.903265 24774 raft_consensus.cc:504] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:49.903792 24774 leader_election.cc:288] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:49.683285 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0ff1c3818a49d2b13f57ea6a81c3d6 (DEFAULT_TABLE table=default.Table_ddf5aa41088d4de4bcf2768d82c7d1fb [id=e9a2a5d8e81e4163b64dc4da67ed09c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.683743 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 87a3df4fbb224dd0b4ec6141584d374b (DEFAULT_TABLE table=default.Table_cce64bbdca4946bfb1cdbd52d1d2d59c [id=8d575e6ea894499a848dc9e39a3e1b23]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.688881 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 4566bc3dbdde463e9392f5ea4fa97741 (DEFAULT_TABLE table=default.Table_bebe57a661bf4fa1b75922c7847aa294 [id=750b978922cd4f3ebd3e453cdbba9d15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.689322 23763 tablet_service.cc:1467] Processing CreateTablet for tablet abcc0296b3c941fbad86bafa8f7965fc (DEFAULT_TABLE table=default.Table_56944338c8ef4b8a98aff14eef9fa611 [id=cdae6a1f994b46a181df44fb2c497ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.690091 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654ce33d1e3d472eb6ae6a726eb803bb" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:49.690318 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c50800505247cf967607bbfcca6302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.696099 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 0dae6a96776c477fb0ada8226d68ea25 (DEFAULT_TABLE table=default.Table_ed62fe1188a546ac8cf56b8f8adcdb62 [id=34a088b1f2654da4b015a4a9d9b71e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.904542 23788 raft_consensus.cc:2454] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:49.451324 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fabad8a6e72a4724893dfa4c0877fdb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:49.452220 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ff8f3ce7d85c49fabe0e3ef763bd7517" } new_table_name: "default.Table_35d118eaf50b48edaa15a45c6cef67b8" modify_external_catalogs: true
I20240502 08:16:49.904767 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dae6a96776c477fb0ada8226d68ea25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.905038 23993 leader_election.cc:302] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
W20240502 08:16:49.905131 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fabad8a6e72a4724893dfa4c0877fdb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:49.481791 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d67c17e0dfd94a088bd9991625f3c31a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:49.905404 25029 raft_consensus.cc:2781] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:49.905476 25029 raft_consensus.cc:482] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:49.905534 25029 raft_consensus.cc:3037] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:49.437000 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_850349fb9fa64ccb884825b10bbec754" } modify_external_catalogs: true
W20240502 08:16:49.905606 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d67c17e0dfd94a088bd9991625f3c31a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:49.685539 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fa44e7d541041d3a86bed4be93b5a03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:49.690629 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e2126bf2d4ab44628b52481dc2ec3a99
I20240502 08:16:49.907500 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe82f5f26d7f423496858f889d65c387. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.962059 25184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1: Bootstrap starting.
I20240502 08:16:49.907191 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_01d1b1f9058e48178e6bfab9c6981d97" } new_table_name: "default.Table_53ed03e87b794949a6d4d86f2a617612" modify_external_catalogs: true
W20240502 08:16:49.740950 23597 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 88bf97d0a0a54e828c3b0902976e0500 (table default.Table_a10aba559706443aae94d509cbb5eefa [id=bec84f8866b44454a38303786e974759], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 88bf97d0a0a54e828c3b0902976e0500 consensus state has no leader
I20240502 08:16:49.499056 24031 tablet_service.cc:1467] Processing CreateTablet for tablet bcb8f7e83e3a456585ba2533e96b76b9 (DEFAULT_TABLE table=default.Table_2c0098a6f7e3460e8a2bed24e0ecfb77 [id=87ecc0bcaedf4181a30dd61d8aef95b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.875278 25012 log.cc:1196] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/2bfb532c9d4c41afb7acaa4831841306
I20240502 08:16:50.723747 25012 ts_tablet_manager.cc:1909] T 2bfb532c9d4c41afb7acaa4831841306 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
W20240502 08:16:49.597364 23594 consensus_peers.cc:485] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 -> Peer 84f3c07a31414c4f8fb61c44dc9f16e1 (127.22.225.125:36189): Couldn't send request to peer 84f3c07a31414c4f8fb61c44dc9f16e1. Status: Remote error: Service unavailable: Systable not yet initialized. This is attempt 16: this message will repeat every 5th retry.
W20240502 08:16:49.846401 23583 kernel_stack_watchdog.cc:198] Thread 23594 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 158ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f7e3d37f980 (unknown) at ??:0
@ 0x7f7e3d3799c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f7e372ff904 google::LogMessage::Flush() at ??:0
@ 0x7f7e372ffa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f7e4573c9ee kudu::consensus::Peer::ProcessResponseErrorUnlocked() at ??:0
@ 0x7f7e4573b69f kudu::consensus::Peer::ProcessResponse() at ??:0
@ 0x7f7e45742061 kudu::consensus::Peer::SendNextRequest()::$_3::operator()() at ??:0
@ 0x7f7e45741e6e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f7e40c2b4e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f7e40c2c365 kudu::rpc::OutboundCall::SetFailed() at ??:0
@ 0x7f7e40c2bc13 kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f7e40bc851a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f7e40bc6f45 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7f7e40bd92c5 ev::base<>::method_thunk<>() at ??:0
@ 0x7f7e387fc699 ev_invoke_pending at ??:0
I20240502 08:16:50.724117 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 711) took 3017 ms. Trace:
W20240502 08:16:50.724093 23594 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.125:36189, user_credentials={real_user=slave}} blocked reactor thread for 1.12664e+06us
I20240502 08:16:50.724208 25012 rpcz_store.cc:276] 0502 08:16:47.706320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.706408 (+ 88us) service_pool.cc:225] Handling call
0502 08:16:49.299737 (+1593329us) ts_tablet_manager.cc:1120] Deleting tablet 2bfb532c9d4c41afb7acaa4831841306
0502 08:16:49.299749 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:49.523220 (+223471us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:49.620235 (+ 97015us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:50.724090 (+1103855us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1593171}
I20240502 08:16:50.724602 25012 tablet_replica.cc:330] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:50.724805 25012 raft_consensus.cc:2229] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:50.724992 25012 raft_consensus.cc:2258] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:50.725688 25012 ts_tablet_manager.cc:1875] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:49.677515 23722 service_pool.cc:138] CreateTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:43544 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 835)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 836)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 837)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 838)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 839)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 840)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 841)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 842)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 843)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 844)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 845)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 846)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 847)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 848)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 849)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 850)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 851)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 852)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 853)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 854)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 855)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 856)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 857)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 858)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 859)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 860)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 861)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 862)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 863)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 864)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 865)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 866)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 867)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 868)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 869)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 870)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 871)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 872)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 873)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 874)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 875)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 876)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 878)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 879)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 880)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 885)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 886)
Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:43544 (request call id 887)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 888)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 893)
I20240502 08:16:49.407245 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 4566bc3dbdde463e9392f5ea4fa97741 (DEFAULT_TABLE table=default.Table_bebe57a661bf4fa1b75922c7847aa294 [id=750b978922cd4f3ebd3e453cdbba9d15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.682847 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654ce33d1e3d472eb6ae6a726eb803bb" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:49.425237 25078 raft_consensus.cc:482] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:16:49.520466 23917 consensus_meta.cc:287] T 27093862fef54c5c85d810ebeff9819c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent flushing consensus metadata: real 0.607s user 0.003s sys 0.000s
I20240502 08:16:50.730149 23914 raft_consensus.cc:2454] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:50.730319 23914 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.225.67:34945 (request call id 114) took 1055 ms. Trace:
I20240502 08:16:50.730412 23914 rpcz_store.cc:276] 0502 08:16:49.674603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.682773 (+ 8170us) service_pool.cc:225] Handling call
0502 08:16:50.730279 (+1047506us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:49.731539 23608 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 7ed919909289410e867881baab841f53 (table default.Table_2edad641920b44ce9a4d5b7503f0279a [id=ee3912407a934444ad2c62f7b83d93ac]). Expected version 2 got 0
I20240502 08:16:49.700949 25076 raft_consensus.cc:2726] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:49.714633 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 1fe597986ce945aba8f2bb19d46bb7ad (DEFAULT_TABLE table=default.Table_e20e3a08238f428fa6c14fa84e67444a [id=0c2788ca8ee44843b047cd3c8786e6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.693692 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87a3df4fbb224dd0b4ec6141584d374b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.733374 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8eccf0ea5e43819f0b8fe364c4749d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.920424 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2e579720e5df47e9875f86b878b88260" } modify_external_catalogs: true
I20240502 08:16:50.733623 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fe597986ce945aba8f2bb19d46bb7ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.733702 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2e579720e5df47e9875f86b878b88260" } modify_external_catalogs: true
I20240502 08:16:49.701539 24175 tablet_bootstrap.cc:654] T 1ba1baa2279045b3adc58d187a12cbf3 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:49.714743 23752 tablet_service.cc:1467] Processing CreateTablet for tablet e99c6ea531ff4e9f92e130f9e21f0ff6 (DEFAULT_TABLE table=default.Table_ab1b6d7ec1d44a4f8559d75bb38fb8a1 [id=8dd96f61050642c9b2cf12841d8140d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.908272 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d609a55e47ad420098d279835e9aa090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.737753 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 336f0eb5085546dba763e0d5c5dd5324. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.418264 24058 raft_consensus.cc:2454] T 96fcb895c8f449deb0190de75af18c3c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:50.737846 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b5880bf09c47da9f5e19d86ef976d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.910578 23785 raft_consensus.cc:2454] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:50.737977 24058 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.225.65:32967 (request call id 251) took 1782 ms. Trace:
I20240502 08:16:50.738078 24058 rpcz_store.cc:276] 0502 08:16:48.955925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.955978 (+ 53us) service_pool.cc:225] Handling call
0502 08:16:50.737915 (+1781937us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:49.939200 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e2cd64637b7843539c015901a34d3258" } modify_external_catalogs: true
I20240502 08:16:50.738219 23785 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.225.67:50405 (request call id 119) took 1365 ms. Trace:
I20240502 08:16:50.738315 23785 rpcz_store.cc:276] 0502 08:16:49.373096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.373275 (+ 179us) service_pool.cc:225] Handling call
0502 08:16:50.738125 (+1364850us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x616000057d54. stack: 00007f1068831756 00007f1068831510 00007f106786448d 0000000000955804 00000000009557cd 000000000095519a 00007f10740906b5 00007f1077d61d24 00007f107225c9b6 00007f107225c6e5 00007f106f53904e 00007f106f537f2e 00007f106f53e405 00007f106f53ed2f 00007f106f53eb58 0000000000a60bcc
0502 08:16:50.738168 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":480341376}
I20240502 08:16:50.738371 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e2cd64637b7843539c015901a34d3258" } modify_external_catalogs: true
I20240502 08:16:49.953581 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_94d88b6d81024599a0005647fa02c965" } new_table_name: "default.Table_81825c4d3ce8457ba17891e473aacddc" modify_external_catalogs: true
I20240502 08:16:49.718334 25108 raft_consensus.cc:504] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:50.739503 24174 tablet_bootstrap.cc:654] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:50.740072 25108 leader_election.cc:288] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:49.473399 24026 tablet_service.cc:1467] Processing CreateTablet for tablet a5b5880bf09c47da9f5e19d86ef976d4 (DEFAULT_TABLE table=default.Table_664d4018b9074906bd16eb6782673acc [id=9097856ad6c24397a1385fa39e5da478]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:50.740298 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4566bc3dbdde463e9392f5ea4fa97741. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:49.953954 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c49f172372448c2be4909ae36371267 consensus state has no leader
I20240502 08:16:50.740499 23610 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36668 (request call id 1024) took 1638 ms. Trace:
I20240502 08:16:50.740576 23610 rpcz_store.cc:276] 0502 08:16:49.102232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.452030 (+349798us) service_pool.cc:225] Handling call
0502 08:16:49.904829 (+452799us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0502 08:16:49.905087 (+ 258us) catalog_manager.cc:3613] Apply alter schema
0502 08:16:49.906571 (+ 1484us) catalog_manager.cc:3636] Acquired catalog manager lock
0502 08:16:49.906647 (+ 76us) catalog_manager.cc:3783] Updating metadata on disk
0502 08:16:49.953741 (+ 47094us) catalog_manager.cc:3814] Committing alterations to in-memory state
0502 08:16:50.740449 (+786708us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:49.924309 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:49.924394 (+ 85us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:49.924408 (+ 14us) write_op.cc:427] Acquired schema lock
0502 08:16:49.924412 (+ 4us) tablet.cc:660] Decoding operations
0502 08:16:49.925465 (+ 1053us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:49.925503 (+ 38us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:49.925512 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0502 08:16:49.925952 (+ 440us) tablet.cc:699] Row locks acquired
0502 08:16:49.925958 (+ 6us) write_op.cc:260] PREPARE: finished
0502 08:16:49.926041 (+ 83us) write_op.cc:270] Start()
0502 08:16:49.926182 (+ 141us) write_op.cc:276] Timestamp: P: 1714637809926004 usec, L: 0
0502 08:16:49.926189 (+ 7us) op_driver.cc:331] REPLICATION: starting
0502 08:16:49.935923 (+ 9734us) log.cc:842] Serialized 405 byte log entry
0502 08:16:49.953152 (+ 17229us) op_driver.cc:444] REPLICATION: finished
0502 08:16:49.953258 (+ 106us) write_op.cc:301] APPLY: starting
0502 08:16:49.953275 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:49.953318 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:49.953371 (+ 53us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:49.953425 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:49.953454 (+ 29us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0502 08:16:49.953460 (+ 6us) write_op.cc:312] APPLY: finished
0502 08:16:49.953526 (+ 66us) log.cc:842] Serialized 23 byte log entry
0502 08:16:49.953587 (+ 61us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:49.953600 (+ 13us) write_op.cc:446] Released schema lock
0502 08:16:49.953606 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:49.953650 (+ 44us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:49.953653 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":52,"apply.run_cpu_time_us":440,"apply.run_wall_time_us":440,"num_ops":1,"prepare.queue_time_us":17335,"prepare.run_cpu_time_us":1440,"prepare.run_wall_time_us":15569,"replication_time_us":26947}]]}
I20240502 08:16:50.743211 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_33c693cfb89641a2921c7d39872b5bbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:50.743753 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33c693cfb89641a2921c7d39872b5bbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:49.681721 23919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c377e037894728ac084b6500ad20de" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:50.744763 23919 raft_consensus.cc:2454] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:50.744910 23919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.225.67:34945 (request call id 112) took 1070 ms. Trace:
I20240502 08:16:50.745013 23919 rpcz_store.cc:276] 0502 08:16:49.674466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.681648 (+ 7182us) service_pool.cc:225] Handling call
0502 08:16:50.744881 (+1063233us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:50.745677 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf9feb5c42cd4d8b9d265742813f798c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.684381 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654ce33d1e3d472eb6ae6a726eb803bb" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:49.927511 23786 raft_consensus.cc:2454] T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:49.940485 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 296b1444c5bb4d319ab9c69373fff84e (DEFAULT_TABLE table=default.Table_7227bf354dec4869bd921dd397b17de3 [id=40dbc13e0508467380cc488dfd713ac6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.954039 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_dd5271576d9746048fa96f022073b8fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:49.945211 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 1626382a260044eb9100b8048332abb0 (DEFAULT_TABLE table=default.Table_c9c50670737d40c58a305ff256334951 [id=034e49e9b5cc435cad83782e0f1d9ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.957028 23755 tablet_service.cc:1467] Processing CreateTablet for tablet bcb8f7e83e3a456585ba2533e96b76b9 (DEFAULT_TABLE table=default.Table_2c0098a6f7e3460e8a2bed24e0ecfb77 [id=87ecc0bcaedf4181a30dd61d8aef95b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.883656 23723 leader_election.cc:302] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
W20240502 08:16:50.747002 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd5271576d9746048fa96f022073b8fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:49.919705 25158 raft_consensus.cc:482] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:49.470752 24021 tablet_service.cc:1467] Processing CreateTablet for tablet e99c6ea531ff4e9f92e130f9e21f0ff6 (DEFAULT_TABLE table=default.Table_ab1b6d7ec1d44a4f8559d75bb38fb8a1 [id=8dd96f61050642c9b2cf12841d8140d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.484432 24023 tablet_service.cc:1467] Processing CreateTablet for tablet e546c1bb78a848ad8125e8e8b08434db (DEFAULT_TABLE table=default.Table_843f461368e94ea9b6b9244506423221 [id=687f28ef29324002821616ba0044d1e3]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:49.970654 23858 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 115181us
I20240502 08:16:49.735503 24036 tablet_service.cc:1467] Processing CreateTablet for tablet e9a35b9486da4aac8f3d3febc93b7816 (DEFAULT_TABLE table=default.Table_47fe484d653742f582516ba50ff30131 [id=10b3982b52054f8cb38f5b9f32739110]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.496755 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 1626382a260044eb9100b8048332abb0 (DEFAULT_TABLE table=default.Table_c9c50670737d40c58a305ff256334951 [id=034e49e9b5cc435cad83782e0f1d9ecb]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:50.748265 23858 leader_election.cc:341] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 98a96292b28a4b0aad88fe0a96730a68
I20240502 08:16:49.499538 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 548ab3f9d0d84527b1092ae715ec2bdd (DEFAULT_TABLE table=default.Table_59fc9761e46d4ff0bad5029d2b0b25fc [id=54be5ee518c94ed697af4b2a222bde65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:50.748407 23858 leader_election.cc:302] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:49.494534 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 296b1444c5bb4d319ab9c69373fff84e (DEFAULT_TABLE table=default.Table_7227bf354dec4869bd921dd397b17de3 [id=40dbc13e0508467380cc488dfd713ac6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.500491 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 3af208d6231a4cf18823e9fe367ca46d (DEFAULT_TABLE table=default.Table_872fcb31253f47e1976fbed00a641287 [id=30bc0d639d5d46f39aa4da2528842969]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.478232 24028 tablet_service.cc:1467] Processing CreateTablet for tablet e20e126b1c9546e4a194bf9842f444a5 (DEFAULT_TABLE table=default.Table_769de68591d64bdb9b80f7bb337da552 [id=f8b12b3bb4904c18be60b567cb05b739]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.428305 24173 ts_tablet_manager.cc:1398] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.128s user 0.000s sys 0.004s
W20240502 08:16:50.748834 24173 ts_tablet_manager.cc:1430] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5: Tablet startup took 1448ms
I20240502 08:16:50.748842 23858 leader_election.cc:302] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:50.748898 24173 ts_tablet_manager.cc:1432] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5: Trace:
0502 08:16:38.616393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.616842 (+ 449us) service_pool.cc:225] Handling call
0502 08:16:38.623269 (+ 6427us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:38.626080 (+ 2811us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:38.692245 (+ 66165us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:38.856815 (+164570us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:38.859109 (+ 2294us) inbound_call.cc:173] Queueing success response
0502 08:16:48.945648 (+10086539us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0502 08:16:49.299950 (+354302us) ts_tablet_manager.cc:1399] Starting tablet replica
0502 08:16:49.300150 (+ 200us) tablet_replica.cc:259] Starting instrumentation
0502 08:16:49.300340 (+ 190us) tablet_replica.cc:278] Starting consensus
Metrics: {"raft.queue_time_us":37467,"raft.run_cpu_time_us":28,"raft.run_wall_time_us":28,"tablet-open.queue_time_us":10086567,"thread_start_us":126542,"threads_started":1}
I20240502 08:16:50.749464 24173 tablet_bootstrap.cc:492] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:50.750943 24173 tablet_bootstrap.cc:654] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:49.464238 24022 tablet_service.cc:1467] Processing CreateTablet for tablet e4a87654d97249418e5df3a99b23c557 (DEFAULT_TABLE table=default.Table_74a4ec7454a14f74ad2c1dc97fab2080 [id=ab048bd406f141769d88fcc99b5307e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.954216 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c1886293aee349b2b9e7177c1cfd6d26" } modify_external_catalogs: true
I20240502 08:16:50.752940 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c1886293aee349b2b9e7177c1cfd6d26" } modify_external_catalogs: true
I20240502 08:16:49.888453 23861 leader_election.cc:302] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
W20240502 08:16:50.753464 23861 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 864999us
W20240502 08:16:50.754094 23861 leader_election.cc:341] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 9c072eb45e1b41eeb5aafa6d5a786944
I20240502 08:16:49.454743 24033 tablet_service.cc:1467] Processing CreateTablet for tablet abcc0296b3c941fbad86bafa8f7965fc (DEFAULT_TABLE table=default.Table_56944338c8ef4b8a98aff14eef9fa611 [id=cdae6a1f994b46a181df44fb2c497ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:50.290361 23607 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36668 (request call id 1026) took 1138 ms. Trace:
I20240502 08:16:49.730944 25154 raft_consensus.cc:686] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:50.755205 25154 consensus_queue.cc:237] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:50.754945 23607 rpcz_store.cc:276] 0502 08:16:49.151998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.863529 (+711531us) service_pool.cc:225] Handling call
0502 08:16:49.864465 (+ 936us) catalog_manager.cc:2072] Acquired catalog manager lock
0502 08:16:49.916492 (+ 52027us) catalog_manager.cc:2118] Created new table and tablet info
0502 08:16:50.290022 (+373530us) catalog_manager.cc:2178] Wrote table and tablets to system table
0502 08:16:50.290166 (+ 144us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0502 08:16:50.290291 (+ 125us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:49.942846 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:49.946919 (+ 4073us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:49.946926 (+ 7us) write_op.cc:427] Acquired schema lock
0502 08:16:49.947248 (+ 322us) tablet.cc:660] Decoding operations
0502 08:16:49.951994 (+ 4746us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:49.952220 (+ 226us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:49.952372 (+ 152us) tablet.cc:683] Acquiring locks for 2 operations
0502 08:16:49.957942 (+ 5570us) tablet.cc:699] Row locks acquired
0502 08:16:49.957948 (+ 6us) write_op.cc:260] PREPARE: finished
0502 08:16:49.959601 (+ 1653us) write_op.cc:270] Start()
0502 08:16:49.960721 (+ 1120us) write_op.cc:276] Timestamp: P: 1714637809958768 usec, L: 0
0502 08:16:49.960731 (+ 10us) op_driver.cc:331] REPLICATION: starting
0502 08:16:49.967331 (+ 6600us) log.cc:842] Serialized 433 byte log entry
0502 08:16:50.288863 (+321532us) op_driver.cc:444] REPLICATION: finished
0502 08:16:50.289007 (+ 144us) write_op.cc:301] APPLY: starting
0502 08:16:50.289031 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:50.289090 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:50.289134 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:50.289277 (+ 143us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:50.289326 (+ 49us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0502 08:16:50.289336 (+ 10us) write_op.cc:312] APPLY: finished
0502 08:16:50.289448 (+ 112us) log.cc:842] Serialized 29 byte log entry
0502 08:16:50.289527 (+ 79us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:50.289556 (+ 29us) write_op.cc:446] Released schema lock
0502 08:16:50.289815 (+ 259us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:50.289879 (+ 64us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:50.289886 (+ 7us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":48,"apply.run_cpu_time_us":730,"apply.run_wall_time_us":959,"num_ops":2,"prepare.queue_time_us":2759,"prepare.run_cpu_time_us":2964,"prepare.run_wall_time_us":28876,"raft.queue_time_us":166,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":283,"replication_time_us":328094,"spinlock_wait_cycles":14080}]]}
I20240502 08:16:50.755828 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8eddc9a260a04c339e7bb130c9822c62" } new_table_name: "default.Table_f5396c0d857746a2bd53a2336eb5164c" modify_external_catalogs: true
I20240502 08:16:49.380089 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd8338500f54604abcd6d42702c2eca" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:50.756484 24055 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.225.66:45417 (request call id 76) took 1376 ms. Trace:
I20240502 08:16:50.756574 24055 rpcz_store.cc:276] 0502 08:16:49.379809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.380003 (+ 194us) service_pool.cc:225] Handling call
0502 08:16:50.756432 (+1376429us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:49.460995 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 0dae6a96776c477fb0ada8226d68ea25 (DEFAULT_TABLE table=default.Table_ed62fe1188a546ac8cf56b8f8adcdb62 [id=34a088b1f2654da4b015a4a9d9b71e66]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:50.242435 23847 kernel_stack_watchdog.cc:198] Thread 23861 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 256ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fb1f89c6980 (unknown) at ??:0
@ 0x7fb1f89c09c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7fb1f2946904 google::LogMessage::Flush() at ??:0
@ 0x7fb1f2946a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fb200dec3f0 kudu::consensus::LeaderElection::CheckForDecision() at ??:0
@ 0x7fb200ded782 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7fb200df012a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7fb200defeee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fb1fc2868cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7fb1fc2864ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fb1fc2724e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fb1fc272a2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fb1fc20f51a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fb1fc20df45 kudu::rpc::Connection::ReadHandler() at ??:0
I20240502 08:16:49.698055 25031 consensus_queue.cc:1035] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:49.912659 24181 raft_consensus.cc:2229] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:49.932708 24944 raft_consensus.cc:504] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:50.757661 24944 leader_election.cc:288] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:50.758402 25076 raft_consensus.cc:2726] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:50.758563 25076 raft_consensus.cc:2726] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:50.758671 25076 raft_consensus.cc:2781] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:50.758731 25076 raft_consensus.cc:482] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:50.758797 25076 raft_consensus.cc:3037] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:49.945420 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a4104f4cb64889ad66810788ad6b29" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:50.761544 23594 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 6c49f172372448c2be4909ae36371267 (table default.Table_35d118eaf50b48edaa15a45c6cef67b8 [id=ff9209b28981436aaadedaf4f29c7afe], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6c49f172372448c2be4909ae36371267 consensus state has no leader
I20240502 08:16:50.761710 24057 raft_consensus.cc:2379] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a297a6c7c2504ca68e87b24f6691ffb2 in current term 1: Already voted for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in this term.
I20240502 08:16:49.946004 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1e208733844829b13222e00a432439" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:50.762213 24054 raft_consensus.cc:3037] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:49.946128 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e549cad1b3be4b28abb0bf5af720eb52" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:50.745504 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf9feb5c42cd4d8b9d265742813f798c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.745219 23757 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 832) took 1664 ms. Trace:
I20240502 08:16:50.763677 24053 raft_consensus.cc:3037] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.763886 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb8f7e83e3a456585ba2533e96b76b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.764302 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1626382a260044eb9100b8048332abb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.747437 25158 raft_consensus.cc:504] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:50.764938 25158 leader_election.cc:288] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:49.947973 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ccd43e78574f3288b199fa14bcc0e8" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:50.765091 25158 raft_consensus.cc:2781] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:50.779623 24051 raft_consensus.cc:3037] T f5ccd43e78574f3288b199fa14bcc0e8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.746608 23786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.225.67:50405 (request call id 118) took 1373 ms. Trace:
I20240502 08:16:49.948416 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4a31e5aa834adeb9f59267247c6980" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:50.779898 23786 rpcz_store.cc:276] 0502 08:16:49.372993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.373045 (+ 52us) service_pool.cc:225] Handling call
0502 08:16:50.746564 (+1373519us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:50.780069 24052 raft_consensus.cc:2379] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a297a6c7c2504ca68e87b24f6691ffb2 in current term 1: Already voted for candidate 5b89a1f0485a44b7ac1710e2369a809b in this term.
W20240502 08:16:49.888836 23996 leader_election.cc:341] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:50.025456 25029 raft_consensus.cc:504] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:50.780290 23996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 891408us
I20240502 08:16:50.780769 25029 leader_election.cc:288] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:50.106552 23982 kernel_stack_watchdog.cc:198] Thread 23996 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 120ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f9187faf980 (unknown) at ??:0
@ 0x7f9187fa99c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f9181f2f904 google::LogMessage::Flush() at ??:0
@ 0x7f9181f2fa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f91903d65d3 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f91903d912a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f91903d8eee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f918b86f8cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f918b86f4ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f918b85b4e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f918b85ba2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f918b7f851a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f918b7f6f45 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7f918b8092c5 ev::base<>::method_thunk<>() at ??:0
I20240502 08:16:50.675338 25222 consensus_queue.cc:1035] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:50.781389 24059 raft_consensus.cc:2379] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in this term.
I20240502 08:16:50.730167 25078 raft_consensus.cc:504] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:50.767354 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8eccf0ea5e43819f0b8fe364c4749d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.688375 25219 raft_consensus.cc:482] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:50.781719 25219 raft_consensus.cc:504] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:50.766211 23993 leader_election.cc:302] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:50.781997 25078 leader_election.cc:288] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:50.782150 23918 raft_consensus.cc:3037] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.782289 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296b1444c5bb4d319ab9c69373fff84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.782387 25219 leader_election.cc:288] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:50.782550 25227 raft_consensus.cc:2781] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:50.782637 25227 raft_consensus.cc:482] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:50.782745 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87a3df4fbb224dd0b4ec6141584d374b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.782771 25227 raft_consensus.cc:3037] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.783048 24059 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.225.66:45417 (request call id 78) took 1102 ms. Trace:
I20240502 08:16:50.783120 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e546c1bb78a848ad8125e8e8b08434db. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:50.770648 23594 catalog_manager.cc:4573] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): CreateTablet RPC failed for tablet 37b499184f31454c96f66cfdf9143d14: Remote error: Service unavailable: CreateTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:43544 dropped due to backpressure. The service queue is full; it has 50 items.
W20240502 08:16:50.985771 23594 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 215126us
I20240502 08:16:50.783340 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e99c6ea531ff4e9f92e130f9e21f0ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.779045 25108 consensus_queue.cc:1035] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:50.986920 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abcc0296b3c941fbad86bafa8f7965fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.796317 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed919909289410e867881baab841f53" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:50.796504 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a4104f4cb64889ad66810788ad6b29" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:50.796674 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654ce33d1e3d472eb6ae6a726eb803bb" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:50.820763 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a1e208733844829b13222e00a432439 with delete_type TABLET_DATA_DELETED (replaced by tablet 8910bc67c6234a538c8160e7c020653c) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:50.821314 23900 tablet_service.cc:1467] Processing CreateTablet for tablet c61243a38bfa4964b225eeae0e8cc28e (DEFAULT_TABLE table=default.Table_0e7c8c39d3c4443d8b770a580c137b2a [id=80c2e205a47743e8a24ac52b79aa246d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:50.821313 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 5101590c3db24c9ebbb45ee23c974524 (DEFAULT_TABLE table=default.Table_3ee62693a3d14323a553988181673d32 [id=05bf70b65ed94236aa5582f5ceda90c9]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:50.822476 23861 consensus_peers.cc:485] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8b1fa17dad143f0a7f1751e651eb81a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:50.987951 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 5bcc7169796e4ee0910f5e7084849012 (table default.Table_bc9b016da1ce42b6a0509fbb664c8548 [id=8c0e32da69b34dd0b2a3c68e4ba7a166]) successfully deleted
I20240502 08:16:50.851683 23899 tablet_service.cc:1467] Processing CreateTablet for tablet e4ac2b5073e34cd98e0246620fc1515d (DEFAULT_TABLE table=default.Table_f30d039709f9465e96e1280298ae2aaf [id=23ef44470f504e7d8db27dcc8b5b75b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:50.852586 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 11ebacd4c41941cd8a6ec634e3aeeaf5 (DEFAULT_TABLE table=default.Table_a33cd662ce1043e3853792e92624c9be [id=64c9d98633614fe78bb291ff36dc5742]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:50.783142 24059 rpcz_store.cc:276] 0502 08:16:49.680350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.684278 (+ 3928us) service_pool.cc:225] Handling call
0502 08:16:50.782951 (+1098673us) spinlock_profiling.cc:243] Waited 34.9 ms on lock 0x616000117a54. stack: 00007f9184bfa756 00007f9184bfa510 00007f9183c2d48d 0000000000955804 00000000009557cd 000000000095519a 00007f91904596b5 00007f919412ad24 00007f918e6259b6 00007f918e6256e5 00007f918b90204e 00007f918b900f2e 00007f918b907405 00007f918b907d2f 00007f918b907b58 0000000000a60bcc
0502 08:16:50.782996 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":80313088}
I20240502 08:16:50.767203 24181 pending_rounds.cc:62] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5: Trying to abort 1 pending ops.
I20240502 08:16:50.989230 24181 pending_rounds.cc:68] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156463881158656 op_type: NO_OP noop_request { }
I20240502 08:16:50.989403 24181 raft_consensus.cc:2866] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:50.745219 23753 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 839) took 1656 ms. Trace:
I20240502 08:16:50.989537 24181 raft_consensus.cc:2258] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:50.771394 23755 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 851) took 1669 ms. Trace:
I20240502 08:16:50.772718 23785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a4104f4cb64889ad66810788ad6b29" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:50.763754 23757 rpcz_store.cc:276] 0502 08:16:49.080354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.469461 (+389107us) service_pool.cc:225] Handling call
0502 08:16:49.901500 (+432039us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.901523 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:50.741942 (+840419us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:50.744975 (+ 3033us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:50.745161 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27,"spinlock_wait_cycles":896}
I20240502 08:16:50.784811 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed919909289410e867881baab841f53" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:50.990242 24181 ts_tablet_manager.cc:1875] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:50.785493 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654ce33d1e3d472eb6ae6a726eb803bb" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:50.768519 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4566bc3dbdde463e9392f5ea4fa97741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.798255 25158 raft_consensus.cc:686] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:50.775954 24038 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 811) took 1681 ms. Trace:
I20240502 08:16:50.989617 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fe597986ce945aba8f2bb19d46bb7ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.990903 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0bd774640464d2a92b2e71126c83883. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.990864 24038 rpcz_store.cc:276] 0502 08:16:49.093491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.468085 (+374594us) service_pool.cc:225] Handling call
0502 08:16:49.872232 (+404147us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.872256 (+ 24us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:50.736800 (+864544us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:50.763555 (+ 26755us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:50.775329 (+ 11774us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:50.991428 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 4e32319fa9244ed2a323b2b3f5940748 (DEFAULT_TABLE table=default.Table_41a2401b3f4c4748b1d3eb964d8053d5 [id=959f40776ede4f2e8dfa0b1b849f3f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:50.778857 24173 tablet_bootstrap.cc:492] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:50.991755 24173 ts_tablet_manager.cc:1367] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.242s user 0.003s sys 0.001s
I20240502 08:16:50.992609 24173 raft_consensus.cc:348] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:50.992825 24173 raft_consensus.cc:374] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:50.992926 24173 raft_consensus.cc:729] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:50.993155 24173 consensus_queue.cc:260] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:50.993654 24173 ts_tablet_manager.cc:1398] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:50.781186 25029 raft_consensus.cc:482] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a297a6c7c2504ca68e87b24f6691ffb2)
I20240502 08:16:50.993841 25029 raft_consensus.cc:504] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:50.994457 25029 leader_election.cc:288] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:50.766479 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319fabf91458438ba5e40ab68b323055" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:50.994951 24055 raft_consensus.cc:3037] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.995211 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b5880bf09c47da9f5e19d86ef976d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.995985 23763 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 840) took 1905 ms. Trace:
I20240502 08:16:50.996090 23763 rpcz_store.cc:276] 0502 08:16:49.090010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.689127 (+599117us) service_pool.cc:225] Handling call
0502 08:16:49.904369 (+215242us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.904391 (+ 22us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:50.989872 (+1085481us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:50.995287 (+ 5415us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:50.995934 (+ 647us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:50.996603 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 548ab3f9d0d84527b1092ae715ec2bdd (DEFAULT_TABLE table=default.Table_59fc9761e46d4ff0bad5029d2b0b25fc [id=54be5ee518c94ed697af4b2a222bde65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:50.996989 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548ab3f9d0d84527b1092ae715ec2bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.783776 23993 leader_election.cc:302] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
W20240502 08:16:50.997539 23993 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 213746us
I20240502 08:16:50.997694 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e20e126b1c9546e4a194bf9842f444a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.998082 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c96c93bb20744ee78a609742b2cbab2a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:50.998113 23993 leader_election.cc:341] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:50.797204 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797d3c6bdc4493398954fbbf9943971" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:50.814848 25222 consensus_queue.cc:1035] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:50.998375 24052 raft_consensus.cc:2379] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in this term.
I20240502 08:16:50.998721 25222 raft_consensus.cc:2781] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:50.998844 25222 raft_consensus.cc:686] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:50.999106 25222 consensus_queue.cc:237] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:50.999869 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb8f7e83e3a456585ba2533e96b76b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.989646 23753 rpcz_store.cc:276] 0502 08:16:49.088505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.688649 (+600144us) service_pool.cc:225] Handling call
0502 08:16:49.904208 (+215559us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.904227 (+ 19us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:50.742780 (+838553us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:50.744966 (+ 2186us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:50.745161 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.000701 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 3af208d6231a4cf18823e9fe367ca46d (DEFAULT_TABLE table=default.Table_872fcb31253f47e1976fbed00a641287 [id=30bc0d639d5d46f39aa4da2528842969]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:51.000855 23993 leader_election.cc:341] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 27346a7eae6a488fa3415646262a4b88
I20240502 08:16:50.989801 23785 raft_consensus.cc:3037] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:51.001762 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e32319fa9244ed2a323b2b3f5940748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.990242 23757 tablet_service.cc:1467] Processing CreateTablet for tablet e0bd774640464d2a92b2e71126c83883 (DEFAULT_TABLE table=default.Table_c456616beb454b6ea33638f85512f97e [id=aa67c143e5cb44078846d98c0cf1345e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:50.990577 23786 raft_consensus.cc:3037] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.990763 25158 consensus_queue.cc:237] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:50.989706 23755 rpcz_store.cc:276] 0502 08:16:49.101717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.955842 (+854125us) service_pool.cc:225] Handling call
0502 08:16:50.746978 (+791136us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.746996 (+ 18us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:50.767581 (+ 20585us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:50.770633 (+ 3052us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:50.771295 (+ 662us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.005662 23755 tablet_service.cc:1467] Processing CreateTablet for tablet e9a35b9486da4aac8f3d3febc93b7816 (DEFAULT_TABLE table=default.Table_47fe484d653742f582516ba50ff30131 [id=10b3982b52054f8cb38f5b9f32739110]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.007100 24173 tablet_bootstrap.cc:492] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:50.795681 23754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 845) took 1700 ms. Trace:
W20240502 08:16:50.802563 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 918852us
I20240502 08:16:51.008285 23754 rpcz_store.cc:276] 0502 08:16:49.095114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.858982 (+763868us) service_pool.cc:225] Handling call
0502 08:16:49.859511 (+ 529us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.859544 (+ 33us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:50.740827 (+881283us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:50.793889 (+ 53062us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:50.795531 (+ 1642us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.008414 23723 leader_election.cc:302] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:50.814669 25154 consensus_queue.cc:1035] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240502 08:16:51.008632 23723 raft_consensus.cc:2670] T 5bcc7169796e4ee0910f5e7084849012 P a297a6c7c2504ca68e87b24f6691ffb2: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:16:51.008751 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 4e32319fa9244ed2a323b2b3f5940748 (DEFAULT_TABLE table=default.Table_41a2401b3f4c4748b1d3eb964d8053d5 [id=959f40776ede4f2e8dfa0b1b849f3f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:50.899391 24054 raft_consensus.cc:2454] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:50.963690 24053 raft_consensus.cc:2454] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:50.807739 24174 tablet_bootstrap.cc:492] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:51.008939 25241 consensus_queue.cc:1035] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:51.009153 24174 ts_tablet_manager.cc:1367] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 1.106s user 0.025s sys 0.038s
I20240502 08:16:51.009371 25242 consensus_queue.cc:1035] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:50.864706 23756 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 850) took 1764 ms. Trace:
I20240502 08:16:51.009757 23756 rpcz_store.cc:276] 0502 08:16:49.100605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.944971 (+844366us) service_pool.cc:225] Handling call
0502 08:16:50.746833 (+801862us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.746849 (+ 16us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:50.861350 (+114501us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:50.864439 (+ 3089us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:50.864643 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.010012 24174 raft_consensus.cc:348] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.010200 24174 raft_consensus.cc:374] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:51.010286 24174 raft_consensus.cc:729] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:51.010241 23756 tablet_service.cc:1467] Processing CreateTablet for tablet e6e633f26b0e4be38e93af3d65e2e636 (DEFAULT_TABLE table=default.Table_db2ecaab8195464388e2ba94695fce03 [id=490733fbe9e84df1b842fa52fd197e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.010496 24174 consensus_queue.cc:260] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.011011 24174 ts_tablet_manager.cc:1398] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:51.011276 24174 tablet_bootstrap.cc:492] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:51.012862 24174 tablet_bootstrap.cc:654] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:50.870249 23749 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 838) took 1782 ms. Trace:
I20240502 08:16:50.870347 23764 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 849) took 1770 ms. Trace:
I20240502 08:16:50.918726 23767 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 829) took 1841 ms. Trace:
I20240502 08:16:51.013674 23764 rpcz_store.cc:276] 0502 08:16:49.099586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.940220 (+840634us) service_pool.cc:225] Handling call
0502 08:16:50.746662 (+806442us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.746688 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:50.867335 (+120647us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:50.870173 (+ 2838us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:50.870305 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.013634 23749 rpcz_store.cc:276] 0502 08:16:49.087308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.683550 (+596242us) service_pool.cc:225] Handling call
0502 08:16:49.904105 (+220555us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.904134 (+ 29us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:50.867329 (+963195us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:50.869757 (+ 2428us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:50.870121 (+ 364us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.013712 23767 rpcz_store.cc:276] 0502 08:16:49.077045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.374755 (+297710us) service_pool.cc:225] Handling call
0502 08:16:49.898563 (+523808us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.898594 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:50.864452 (+965858us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:50.918506 (+ 54054us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:50.918672 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.014122 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 8d44b485936845b992315645891a5004 (DEFAULT_TABLE table=default.Table_ba6d528fd464438eb4c79eeade694e24 [id=65a77f5506414a849458777eab928d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.014200 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 98f4f2475d4848018414c3025a54488e (DEFAULT_TABLE table=default.Table_282e8b7fa027438c9e38f997527d7406 [id=760e39ce1ac54ca0b347324eb9127630]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.014122 23764 tablet_service.cc:1467] Processing CreateTablet for tablet d5c82820278b4a0c8f7d586f87104bf0 (DEFAULT_TABLE table=default.Table_722c9cabaa9a4f43a7c8aeab8a17dc05 [id=98df2952710847e9af30703324a08d66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.014601 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98f4f2475d4848018414c3025a54488e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.014799 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c82820278b4a0c8f7d586f87104bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.921386 23765 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 828) took 1845 ms. Trace:
I20240502 08:16:51.015029 23765 rpcz_store.cc:276] 0502 08:16:49.075939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.372588 (+296649us) service_pool.cc:225] Handling call
0502 08:16:49.897830 (+525242us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.897857 (+ 27us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:50.868329 (+970472us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:50.921154 (+ 52825us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:50.921331 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.015478 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 1cebfc5ffc184de9a2e703aa4e333c3a (DEFAULT_TABLE table=default.Table_18f465f6e9d048ac8bb97321715d2a6e [id=6be0266503de408a97643c9a8b255096]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.015812 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cebfc5ffc184de9a2e703aa4e333c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.020336 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e633f26b0e4be38e93af3d65e2e636. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.020583 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0bd774640464d2a92b2e71126c83883. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.021376 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a0ff1c3818a49d2b13f57ea6a81c3d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.021771 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a87654d97249418e5df3a99b23c557. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.022372 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af208d6231a4cf18823e9fe367ca46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.022547 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 336f0eb5085546dba763e0d5c5dd5324. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.022753 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a35b9486da4aac8f3d3febc93b7816. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.023000 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0902a73da6e34e8e9cd59bb53a36ec0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.023276 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d44b485936845b992315645891a5004. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.025265 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e32319fa9244ed2a323b2b3f5940748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.029510 24053 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.225.65:32967 (request call id 257) took 1063 ms. Trace:
I20240502 08:16:51.030339 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296b1444c5bb4d319ab9c69373fff84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.046974 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e99c6ea531ff4e9f92e130f9e21f0ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.045512 24053 rpcz_store.cc:276] 0502 08:16:49.945543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.945858 (+ 315us) service_pool.cc:225] Handling call
0502 08:16:51.009016 (+1063158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3072}
I20240502 08:16:51.049861 24054 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.225.65:32967 (request call id 258) took 1063 ms. Trace:
I20240502 08:16:51.050226 24173 tablet_bootstrap.cc:654] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:51.050618 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a35b9486da4aac8f3d3febc93b7816. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:51.055395 23597 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 331561us
I20240502 08:16:51.057421 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet b8b1fa17dad143f0a7f1751e651eb81a (table default.Table_d4bd37f8c6e54ecf82cd247846aa06d0 [id=57722a7341c541b19efa213bcf48eb53]) successfully deleted
W20240502 08:16:51.057744 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 48612.4us
I20240502 08:16:51.058307 25241 consensus_queue.cc:1035] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:51.085752 23886 tablet_service.cc:1467] Processing CreateTablet for tablet fe82f5f26d7f423496858f889d65c387 (DEFAULT_TABLE table=default.Table_31208fdb293844fc838187320c87cdf9 [id=1e0c30a309b14de39df424fc9b28c7db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.170838 25252 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f87df6a45ad94428a9aa6b2796a0fd25)
I20240502 08:16:51.085773 23893 tablet_service.cc:1467] Processing CreateTablet for tablet bf9feb5c42cd4d8b9d265742813f798c (DEFAULT_TABLE table=default.Table_9e3aabcb11df4be58f87a1f242912dec [id=c06de80716064c6b80ab6283e4385713]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.250591 25252 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:51.060559 23785 raft_consensus.cc:2454] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:51.069350 24174 tablet_bootstrap.cc:492] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:51.087504 23894 tablet_service.cc:1467] Processing CreateTablet for tablet e03c13b1cb584f5994dff77692b0f442 (DEFAULT_TABLE table=default.Table_5520c58206f4499bac98e73fb4122056 [id=bd97c826b7b44c63904a8fc981480614]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.251426 24174 ts_tablet_manager.cc:1367] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.240s user 0.010s sys 0.001s
I20240502 08:16:51.087770 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5101590c3db24c9ebbb45ee23c974524. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.252079 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf9feb5c42cd4d8b9d265742813f798c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:51.087836 23861 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 165680us
I20240502 08:16:51.252341 24174 raft_consensus.cc:348] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.252552 24174 raft_consensus.cc:374] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:51.252606 24174 raft_consensus.cc:729] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:51.252802 24174 consensus_queue.cc:260] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.088140 23916 raft_consensus.cc:3037] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:51.089171 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 70b8cd6a22804ad79681a2921f5c12c9 (DEFAULT_TABLE table=default.Table_a662f92a7fe845a990eb07796fc9d8a6 [id=0ec249e5ea9842d5ba593c3585897b66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.253319 24174 ts_tablet_manager.cc:1398] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:51.253432 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b8cd6a22804ad79681a2921f5c12c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.642343 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe82f5f26d7f423496858f889d65c387. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.643131 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ac2b5073e34cd98e0246620fc1515d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.645776 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c61243a38bfa4964b225eeae0e8cc28e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:51.173902 23597 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 116321us
I20240502 08:16:51.649420 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 27346a7eae6a488fa3415646262a4b88 (table default.Table_5d5022ea23c34e4f960c366d5ac96c39 [id=999c7344afec4d3c858dc29a986a8426]) successfully deleted
I20240502 08:16:51.275144 25252 leader_election.cc:288] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84f3c07a31414c4f8fb61c44dc9f16e1 (127.22.225.125:36189), f87df6a45ad94428a9aa6b2796a0fd25 (127.22.225.124:41745)
I20240502 08:16:51.117138 23752 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 844) took 2023 ms. Trace:
I20240502 08:16:51.649735 23752 rpcz_store.cc:276] 0502 08:16:49.094060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.713493 (+619433us) service_pool.cc:225] Handling call
0502 08:16:50.737712 (+1024219us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.737751 (+ 39us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:50.989911 (+252160us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.116904 (+126993us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.117085 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.650321 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11ebacd4c41941cd8a6ec634e3aeeaf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.650307 23752 tablet_service.cc:1467] Processing CreateTablet for tablet 506644a1ead54a1faeaa8583ef96b449 (DEFAULT_TABLE table=default.Table_2e5b16fa5b3b4be3af4d6843606d91f6 [id=362af6ee46a740b28b0fa3e9ae3391fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.650576 23886 tablet_service.cc:1467] Processing CreateTablet for tablet e4a87654d97249418e5df3a99b23c557 (DEFAULT_TABLE table=default.Table_74a4ec7454a14f74ad2c1dc97fab2080 [id=ab048bd406f141769d88fcc99b5307e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.650770 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 506644a1ead54a1faeaa8583ef96b449. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.099692 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dae6a96776c477fb0ada8226d68ea25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.653779 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a87654d97249418e5df3a99b23c557. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.089555 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 12bd0a3bf7d24d939288c3eca83e1540 (DEFAULT_TABLE table=default.Table_d3cd93eb7315400aa078bbb480182814 [id=6af7860d646f4f5c9afaf744baf025c6]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:51.194700 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ae2b20ae3a014c85af33d5ee51b704e1 consensus state has no leader
I20240502 08:16:51.654278 23606 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36668 (request call id 1028) took 2150 ms. Trace:
I20240502 08:16:51.654394 23606 rpcz_store.cc:276] 0502 08:16:49.503431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.906980 (+403549us) service_pool.cc:225] Handling call
0502 08:16:50.721753 (+814773us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0502 08:16:50.722036 (+ 283us) catalog_manager.cc:3613] Apply alter schema
0502 08:16:50.722275 (+ 239us) catalog_manager.cc:3636] Acquired catalog manager lock
0502 08:16:50.722355 (+ 80us) catalog_manager.cc:3783] Updating metadata on disk
0502 08:16:51.194409 (+472054us) catalog_manager.cc:3814] Committing alterations to in-memory state
0502 08:16:51.654213 (+459804us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:50.937829 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:50.937979 (+ 150us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:50.937988 (+ 9us) write_op.cc:427] Acquired schema lock
0502 08:16:50.937992 (+ 4us) tablet.cc:660] Decoding operations
0502 08:16:50.938161 (+ 169us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:50.938180 (+ 19us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:50.938189 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0502 08:16:50.938435 (+ 246us) tablet.cc:699] Row locks acquired
0502 08:16:50.938448 (+ 13us) write_op.cc:260] PREPARE: finished
0502 08:16:50.938499 (+ 51us) write_op.cc:270] Start()
0502 08:16:50.938541 (+ 42us) write_op.cc:276] Timestamp: P: 1714637810938411 usec, L: 0
0502 08:16:50.938547 (+ 6us) op_driver.cc:331] REPLICATION: starting
0502 08:16:50.938789 (+ 242us) log.cc:842] Serialized 405 byte log entry
0502 08:16:51.160482 (+221693us) op_driver.cc:444] REPLICATION: finished
0502 08:16:51.186831 (+ 26349us) write_op.cc:301] APPLY: starting
0502 08:16:51.186859 (+ 28us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:51.187610 (+ 751us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:51.190124 (+ 2514us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:51.191056 (+ 932us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:51.191541 (+ 485us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0502 08:16:51.191552 (+ 11us) write_op.cc:312] APPLY: finished
0502 08:16:51.192565 (+ 1013us) log.cc:842] Serialized 23 byte log entry
0502 08:16:51.193538 (+ 973us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.193883 (+ 345us) write_op.cc:446] Released schema lock
0502 08:16:51.193895 (+ 12us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:51.195252 (+ 1357us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.195264 (+ 12us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":25648,"apply.run_cpu_time_us":1142,"apply.run_wall_time_us":9380,"num_ops":1,"prepare.queue_time_us":214830,"prepare.run_cpu_time_us":11150,"prepare.run_wall_time_us":193893,"raft.queue_time_us":82885,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":329,"replication_time_us":221902,"thread_start_us":435974,"threads_started":3}]]}
I20240502 08:16:51.119755 23759 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 847) took 2022 ms. Trace:
I20240502 08:16:51.656545 23759 rpcz_store.cc:276] 0502 08:16:49.097201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.894242 (+797041us) service_pool.cc:225] Handling call
0502 08:16:49.894683 (+ 441us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.894713 (+ 30us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.014519 (+1119806us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.119536 (+105017us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.119709 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.656590 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 1fe597986ce945aba8f2bb19d46bb7ad (DEFAULT_TABLE table=default.Table_e20e3a08238f428fa6c14fa84e67444a [id=0c2788ca8ee44843b047cd3c8786e6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.656937 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a87654d97249418e5df3a99b23c557. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.657038 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 88bf97d0a0a54e828c3b0902976e0500 (DEFAULT_TABLE table=default.Table_a10aba559706443aae94d509cbb5eefa [id=bec84f8866b44454a38303786e974759]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.657390 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88bf97d0a0a54e828c3b0902976e0500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.276099 23620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" candidate_term: 2 candidate_status { last_received { term: 1 index: 1056 } } ignore_live_leader: false dest_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" is_pre_election: true
I20240502 08:16:51.358971 23608 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418] alter complete (version 2)
W20240502 08:16:51.658334 23608 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.225.67:46497 (request call id 148) took 2121 ms (client timeout 1000 ms). Trace:
W20240502 08:16:51.658453 23608 rpcz_store.cc:269] 0502 08:16:49.537150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.693495 (+156345us) service_pool.cc:225] Handling call
0502 08:16:51.658297 (+1964802us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:51.132285 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:51.133352 (+ 1067us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:51.133358 (+ 6us) write_op.cc:427] Acquired schema lock
0502 08:16:51.133360 (+ 2us) tablet.cc:660] Decoding operations
0502 08:16:51.133978 (+ 618us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:51.133993 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:51.134187 (+ 194us) tablet.cc:683] Acquiring locks for 1 operations
0502 08:16:51.134650 (+ 463us) tablet.cc:699] Row locks acquired
0502 08:16:51.134653 (+ 3us) write_op.cc:260] PREPARE: finished
0502 08:16:51.135452 (+ 799us) write_op.cc:270] Start()
0502 08:16:51.135741 (+ 289us) write_op.cc:276] Timestamp: P: 1714637811134919 usec, L: 0
0502 08:16:51.135875 (+ 134us) op_driver.cc:331] REPLICATION: starting
0502 08:16:51.137988 (+ 2113us) log.cc:842] Serialized 356 byte log entry
0502 08:16:51.357354 (+219366us) op_driver.cc:444] REPLICATION: finished
0502 08:16:51.357655 (+ 301us) write_op.cc:301] APPLY: starting
0502 08:16:51.357693 (+ 38us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:51.357745 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:51.357779 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:51.357850 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:51.357890 (+ 40us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0502 08:16:51.357902 (+ 12us) write_op.cc:312] APPLY: finished
0502 08:16:51.357999 (+ 97us) log.cc:842] Serialized 23 byte log entry
0502 08:16:51.358077 (+ 78us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.358097 (+ 20us) write_op.cc:446] Released schema lock
0502 08:16:51.358105 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:51.358167 (+ 62us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.358174 (+ 7us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":229,"apply.run_cpu_time_us":579,"apply.run_wall_time_us":578,"num_ops":1,"prepare.queue_time_us":399024,"prepare.run_cpu_time_us":1656,"prepare.run_wall_time_us":7189,"replication_time_us":221447}]]}
I20240502 08:16:51.660530 23608 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 27093862fef54c5c85d810ebeff9819c (table default.Table_aded763332144f979b418931264d1eb5 [id=0035c98028aa49f7a7dcea576bb23351]). Expected version 1 got 0
I20240502 08:16:51.661422 24022 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 810) took 2568 ms. Trace:
I20240502 08:16:51.100487 24054 rpcz_store.cc:276] 0502 08:16:49.945650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.945858 (+ 208us) service_pool.cc:225] Handling call
0502 08:16:51.008966 (+1063108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3328}
I20240502 08:16:51.089783 23889 tablet_service.cc:1467] Processing CreateTablet for tablet d5eead3736fb4b34b25850b248d0d62b (DEFAULT_TABLE table=default.Table_8dbc74ff11934c76b732768528c3ed39 [id=b7a577f1b904421fb27312ba8e350d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.661556 24022 rpcz_store.cc:276] 0502 08:16:49.092463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.463979 (+371516us) service_pool.cc:225] Handling call
0502 08:16:50.751782 (+1287803us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.751803 (+ 21us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.656944 (+905141us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.661151 (+ 4207us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.661376 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.662045 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5eead3736fb4b34b25850b248d0d62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.663690 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e20e126b1c9546e4a194bf9842f444a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.664076 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1626382a260044eb9100b8048332abb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.124917 23762 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 846) took 2028 ms. Trace:
I20240502 08:16:51.665666 23762 rpcz_store.cc:276] 0502 08:16:49.096132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.865969 (+769837us) service_pool.cc:225] Handling call
0502 08:16:49.866403 (+ 434us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.866438 (+ 35us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.017363 (+1150925us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.124669 (+107306us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.124869 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.666011 23900 tablet_service.cc:1467] Processing CreateTablet for tablet e99c6ea531ff4e9f92e130f9e21f0ff6 (DEFAULT_TABLE table=default.Table_ab1b6d7ec1d44a4f8559d75bb38fb8a1 [id=8dd96f61050642c9b2cf12841d8140d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.666230 23762 tablet_service.cc:1467] Processing CreateTablet for tablet dfd897ba97204b6788fd2d99cf2a0303 (DEFAULT_TABLE table=default.Table_97534a2075bc4b7297b36916b7a5f77c [id=5823995f324846088c24d9ffd5d742c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.666671 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd897ba97204b6788fd2d99cf2a0303. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.666692 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e99c6ea531ff4e9f92e130f9e21f0ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.359103 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet f685cbe705b149e69f9536e270a74a42
I20240502 08:16:51.413954 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet fe5e08ad6af2442bb0a8fcc80a99b532
I20240502 08:16:51.669482 23899 tablet_service.cc:1467] Processing CreateTablet for tablet a5b5880bf09c47da9f5e19d86ef976d4 (DEFAULT_TABLE table=default.Table_664d4018b9074906bd16eb6782673acc [id=9097856ad6c24397a1385fa39e5da478]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.669999 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b5880bf09c47da9f5e19d86ef976d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.106117 24051 raft_consensus.cc:2454] T f5ccd43e78574f3288b199fa14bcc0e8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
W20240502 08:16:51.415024 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 981b7a7292884b1fb48af99db6e762c0 consensus state has no leader
I20240502 08:16:51.107774 25154 raft_consensus.cc:482] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:51.165144 24173 tablet_bootstrap.cc:492] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:51.166092 25227 raft_consensus.cc:504] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.670243 23602 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36668 (request call id 1032) took 2035 ms. Trace:
I20240502 08:16:51.670225 24051 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.225.65:32967 (request call id 264) took 1722 ms. Trace:
I20240502 08:16:51.090085 23905 tablet_service.cc:1467] Processing CreateTablet for tablet d609a55e47ad420098d279835e9aa090 (DEFAULT_TABLE table=default.Table_346dec12764b4564b56f28b373f24b0b [id=78d93afb51c7474bacdf062b5049f5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.670347 24173 ts_tablet_manager.cc:1367] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.663s user 0.011s sys 0.007s
I20240502 08:16:51.129130 25266 raft_consensus.cc:482] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:51.670346 23602 rpcz_store.cc:276] 0502 08:16:49.634688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.942111 (+307423us) service_pool.cc:225] Handling call
0502 08:16:49.953498 (+ 11387us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x616000021d54. stack: 00007f7e39fca756 00007f7e39fca510 00007f7e38ffd48d 0000000000955804 00000000009557cd 00007f7e48b0915d 00007f7e48b09253 00007f7e4583bd41 00007f7e4d42734d 00007f7e4d5b5f46 00007f7e4439fc26 00007f7e4439f955 00007f7e40cd204e 00007f7e40cd0f2e 00007f7e40cd7405 00007f7e40cd7d2f
0502 08:16:50.738655 (+785157us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0502 08:16:50.738883 (+ 228us) catalog_manager.cc:3613] Apply alter schema
0502 08:16:50.739052 (+ 169us) catalog_manager.cc:3636] Acquired catalog manager lock
0502 08:16:50.739107 (+ 55us) catalog_manager.cc:3783] Updating metadata on disk
0502 08:16:51.414728 (+675621us) catalog_manager.cc:3814] Committing alterations to in-memory state
0502 08:16:51.670203 (+255475us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:51.225744 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:51.225952 (+ 208us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:51.225958 (+ 6us) write_op.cc:427] Acquired schema lock
0502 08:16:51.225960 (+ 2us) tablet.cc:660] Decoding operations
0502 08:16:51.226077 (+ 117us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:51.226093 (+ 16us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:51.226100 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0502 08:16:51.226168 (+ 68us) tablet.cc:699] Row locks acquired
0502 08:16:51.226172 (+ 4us) write_op.cc:260] PREPARE: finished
0502 08:16:51.226206 (+ 34us) write_op.cc:270] Start()
0502 08:16:51.226237 (+ 31us) write_op.cc:276] Timestamp: P: 1714637811226102 usec, L: 0
0502 08:16:51.226241 (+ 4us) op_driver.cc:331] REPLICATION: starting
0502 08:16:51.226426 (+ 185us) log.cc:842] Serialized 405 byte log entry
0502 08:16:51.412779 (+186353us) op_driver.cc:444] REPLICATION: finished
0502 08:16:51.414128 (+ 1349us) write_op.cc:301] APPLY: starting
0502 08:16:51.414148 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:51.414212 (+ 64us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:51.414247 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:51.414304 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:51.414342 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0502 08:16:51.414354 (+ 12us) write_op.cc:312] APPLY: finished
0502 08:16:51.414457 (+ 103us) log.cc:842] Serialized 23 byte log entry
0502 08:16:51.414508 (+ 51us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.414528 (+ 20us) write_op.cc:446] Released schema lock
0502 08:16:51.414536 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:51.414602 (+ 66us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.414608 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":25451776,"child_traces":[["op",{"apply.queue_time_us":1294,"apply.run_cpu_time_us":537,"apply.run_wall_time_us":537,"num_ops":1,"prepare.queue_time_us":486214,"prepare.run_cpu_time_us":720,"prepare.run_wall_time_us":1049,"replication_time_us":186480}]]}
I20240502 08:16:51.670657 25266 raft_consensus.cc:504] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.670809 25227 leader_election.cc:288] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:51.671032 24173 raft_consensus.cc:348] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.671214 24173 raft_consensus.cc:374] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:51.671258 24173 raft_consensus.cc:729] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:51.671169 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_80ca7b4c29584de0b0fddf31e64d9f0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:51.671306 25266 leader_election.cc:288] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:51.671422 24173 consensus_queue.cc:260] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:51.416735 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d446aae7dec4734b112490ed77f26a1 consensus state has no leader
I20240502 08:16:51.091122 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 02c50800505247cf967607bbfcca6302 (DEFAULT_TABLE table=default.Table_bbc5d6ae8e7246129746580755d1a812 [id=6cf60892656a4fe4bda6505a8de87742]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.671849 23607 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36668 (request call id 1035) took 2036 ms. Trace:
I20240502 08:16:51.122109 23766 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 833) took 2040 ms. Trace:
I20240502 08:16:51.671913 24173 ts_tablet_manager.cc:1398] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:16:51.672010 23766 rpcz_store.cc:276] 0502 08:16:49.081332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.470415 (+389083us) service_pool.cc:225] Handling call
0502 08:16:49.900765 (+430350us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.900800 (+ 35us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.019816 (+1119016us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.121896 (+102080us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.122065 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.672179 24173 tablet_bootstrap.cc:492] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:51.671931 23607 rpcz_store.cc:276] 0502 08:16:49.634976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:50.755632 (+1120656us) service_pool.cc:225] Handling call
0502 08:16:50.755999 (+ 367us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0502 08:16:50.756230 (+ 231us) catalog_manager.cc:3613] Apply alter schema
0502 08:16:50.756431 (+ 201us) catalog_manager.cc:3636] Acquired catalog manager lock
0502 08:16:50.756523 (+ 92us) catalog_manager.cc:3783] Updating metadata on disk
0502 08:16:51.416516 (+659993us) catalog_manager.cc:3814] Committing alterations to in-memory state
0502 08:16:51.671799 (+255283us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:51.227527 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:51.227630 (+ 103us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:51.227638 (+ 8us) write_op.cc:427] Acquired schema lock
0502 08:16:51.227641 (+ 3us) tablet.cc:660] Decoding operations
0502 08:16:51.227729 (+ 88us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:51.227737 (+ 8us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:51.227744 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0502 08:16:51.227796 (+ 52us) tablet.cc:699] Row locks acquired
0502 08:16:51.227801 (+ 5us) write_op.cc:260] PREPARE: finished
0502 08:16:51.227830 (+ 29us) write_op.cc:270] Start()
0502 08:16:51.227858 (+ 28us) write_op.cc:276] Timestamp: P: 1714637811227725 usec, L: 0
0502 08:16:51.227863 (+ 5us) op_driver.cc:331] REPLICATION: starting
0502 08:16:51.228006 (+ 143us) log.cc:842] Serialized 405 byte log entry
0502 08:16:51.415766 (+187760us) op_driver.cc:444] REPLICATION: finished
0502 08:16:51.415966 (+ 200us) write_op.cc:301] APPLY: starting
0502 08:16:51.415982 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:51.416015 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:51.416049 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:51.416096 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:51.416134 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0502 08:16:51.416145 (+ 11us) write_op.cc:312] APPLY: finished
0502 08:16:51.416220 (+ 75us) log.cc:842] Serialized 23 byte log entry
0502 08:16:51.416268 (+ 48us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.416290 (+ 22us) write_op.cc:446] Released schema lock
0502 08:16:51.416300 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:51.416373 (+ 73us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.416409 (+ 36us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":132,"apply.run_cpu_time_us":504,"apply.run_wall_time_us":504,"num_ops":1,"prepare.queue_time_us":470594,"prepare.run_cpu_time_us":548,"prepare.run_wall_time_us":547,"replication_time_us":187783}]]}
I20240502 08:16:51.672482 23766 tablet_service.cc:1467] Processing CreateTablet for tablet d87820f2490f4631b455e8dc942f7147 (DEFAULT_TABLE table=default.Table_fc2d835b22c1417bad53bb6f66bde481 [id=d361511883cd45d28d4965e0a718e0f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.672847 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d87820f2490f4631b455e8dc942f7147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.672713 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9ff1ae77ef32484abb840fd10fcb88e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:51.168807 24038 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 824) took 2060 ms. Trace:
I20240502 08:16:51.673611 24038 rpcz_store.cc:276] 0502 08:16:49.108458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:50.991170 (+1882712us) service_pool.cc:225] Handling call
0502 08:16:51.001493 (+ 10323us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.001527 (+ 34us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.066246 (+ 64719us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.168596 (+102350us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.168757 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.674113 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 8d44b485936845b992315645891a5004 (DEFAULT_TABLE table=default.Table_ba6d528fd464438eb4c79eeade694e24 [id=65a77f5506414a849458777eab928d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.674453 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d44b485936845b992315645891a5004. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.171687 24025 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 817) took 2071 ms. Trace:
I20240502 08:16:51.418118 23610 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36668 (request call id 1034) took 1783 ms. Trace:
I20240502 08:16:51.677608 24025 rpcz_store.cc:276] 0502 08:16:49.100144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.494299 (+394155us) service_pool.cc:225] Handling call
0502 08:16:50.748556 (+1254257us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.748571 (+ 15us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.142232 (+393661us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.171528 (+ 29296us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.171669 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.678004 24025 tablet_service.cc:1467] Processing CreateTablet for tablet d5c82820278b4a0c8f7d586f87104bf0 (DEFAULT_TABLE table=default.Table_722c9cabaa9a4f43a7c8aeab8a17dc05 [id=98df2952710847e9af30703324a08d66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.677670 23610 rpcz_store.cc:276] 0502 08:16:49.634892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:50.742997 (+1108105us) service_pool.cc:225] Handling call
0502 08:16:50.743854 (+ 857us) catalog_manager.cc:2072] Acquired catalog manager lock
0502 08:16:50.744301 (+ 447us) catalog_manager.cc:2118] Created new table and tablet info
0502 08:16:51.417809 (+673508us) catalog_manager.cc:2178] Wrote table and tablets to system table
0502 08:16:51.417936 (+ 127us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0502 08:16:51.418051 (+ 115us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:51.228111 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:51.228217 (+ 106us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:51.228223 (+ 6us) write_op.cc:427] Acquired schema lock
0502 08:16:51.228228 (+ 5us) tablet.cc:660] Decoding operations
0502 08:16:51.228313 (+ 85us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:51.228323 (+ 10us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:51.228330 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0502 08:16:51.228416 (+ 86us) tablet.cc:699] Row locks acquired
0502 08:16:51.228419 (+ 3us) write_op.cc:260] PREPARE: finished
0502 08:16:51.228447 (+ 28us) write_op.cc:270] Start()
0502 08:16:51.228472 (+ 25us) write_op.cc:276] Timestamp: P: 1714637811228342 usec, L: 0
0502 08:16:51.228477 (+ 5us) op_driver.cc:331] REPLICATION: starting
0502 08:16:51.228646 (+ 169us) log.cc:842] Serialized 433 byte log entry
0502 08:16:51.416980 (+188334us) op_driver.cc:444] REPLICATION: finished
0502 08:16:51.417251 (+ 271us) write_op.cc:301] APPLY: starting
0502 08:16:51.417268 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:51.417307 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:51.417338 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:51.417488 (+ 150us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:51.417526 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0502 08:16:51.417538 (+ 12us) write_op.cc:312] APPLY: finished
0502 08:16:51.417629 (+ 91us) log.cc:842] Serialized 29 byte log entry
0502 08:16:51.417686 (+ 57us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.417701 (+ 15us) write_op.cc:446] Released schema lock
0502 08:16:51.417707 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:51.417766 (+ 59us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.417774 (+ 8us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":183,"apply.run_cpu_time_us":582,"apply.run_wall_time_us":582,"num_ops":2,"prepare.queue_time_us":392768,"prepare.run_cpu_time_us":604,"prepare.run_wall_time_us":603,"replication_time_us":188378}]]}
I20240502 08:16:51.678336 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c82820278b4a0c8f7d586f87104bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.678557 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_06aacf94362b40b9afb9c267bdcef6a3" } new_table_name: "default.Table_3f83bc6549c54d4aaba15d41ffb66020" modify_external_catalogs: true
I20240502 08:16:51.161485 23786 raft_consensus.cc:2454] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:51.419328 23601 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36668 (request call id 1033) took 1784 ms. Trace:
I20240502 08:16:51.172673 24031 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 819) took 2070 ms. Trace:
I20240502 08:16:51.255574 24055 raft_consensus.cc:2454] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:51.687390 24031 rpcz_store.cc:276] 0502 08:16:49.102386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.498467 (+396081us) service_pool.cc:225] Handling call
0502 08:16:50.723148 (+1224681us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.723223 (+ 75us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.078474 (+355251us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.172465 (+ 93991us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.172627 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.255872 24037 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 820) took 2152 ms. Trace:
I20240502 08:16:51.687783 24037 rpcz_store.cc:276] 0502 08:16:49.103440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.498611 (+395171us) service_pool.cc:225] Handling call
0502 08:16:49.871857 (+373246us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.871887 (+ 30us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:50.998125 (+1126238us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.255656 (+257531us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.255846 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.687924 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 98f4f2475d4848018414c3025a54488e (DEFAULT_TABLE table=default.Table_282e8b7fa027438c9e38f997527d7406 [id=760e39ce1ac54ca0b347324eb9127630]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.256387 24034 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 807) took 2166 ms. Trace:
I20240502 08:16:51.168043 23756 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 857) took 2059 ms. Trace:
I20240502 08:16:51.169025 24180 ts_tablet_manager.cc:1888] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:51.688195 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 1cebfc5ffc184de9a2e703aa4e333c3a (DEFAULT_TABLE table=default.Table_18f465f6e9d048ac8bb97321715d2a6e [id=6be0266503de408a97643c9a8b255096]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.181341 23754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 856) took 2074 ms. Trace:
I20240502 08:16:51.091352 25076 raft_consensus.cc:504] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.688133 24034 rpcz_store.cc:276] 0502 08:16:49.089448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.406926 (+317478us) service_pool.cc:225] Handling call
0502 08:16:50.729862 (+1322936us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.729896 (+ 34us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:50.997110 (+267214us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.256184 (+259074us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.256325 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20}
I20240502 08:16:51.688408 23754 rpcz_store.cc:276] 0502 08:16:49.107399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.008501 (+1901102us) service_pool.cc:225] Handling call
0502 08:16:51.008908 (+ 407us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.008947 (+ 39us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.027865 (+ 18918us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.181117 (+153252us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.181315 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.688724 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 506644a1ead54a1faeaa8583ef96b449 (DEFAULT_TABLE table=default.Table_2e5b16fa5b3b4be3af4d6843606d91f6 [id=362af6ee46a740b28b0fa3e9ae3391fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.688930 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2c637e208044d8aaf1774f448e0527 (DEFAULT_TABLE table=default.Table_0f6ad9ecb09e4a03b995f7350e0a3af0 [id=0b630609c7294606bff513bbcd28c4a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.256690 24040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 797) took 2179 ms. Trace:
I20240502 08:16:51.689126 25076 leader_election.cc:288] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:51.090320 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 87a3df4fbb224dd0b4ec6141584d374b (DEFAULT_TABLE table=default.Table_cce64bbdca4946bfb1cdbd52d1d2d59c [id=8d575e6ea894499a848dc9e39a3e1b23]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.689163 24040 rpcz_store.cc:276] 0502 08:16:49.077663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.160214 (+ 82551us) service_pool.cc:225] Handling call
0502 08:16:49.393035 (+232821us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.393062 (+ 27us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:50.848389 (+1455327us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.256537 (+408148us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.256672 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.689328 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2c637e208044d8aaf1774f448e0527. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.092077 23897 tablet_service.cc:1467] Processing CreateTablet for tablet 4566bc3dbdde463e9392f5ea4fa97741 (DEFAULT_TABLE table=default.Table_bebe57a661bf4fa1b75922c7847aa294 [id=750b978922cd4f3ebd3e453cdbba9d15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.689499 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 88bf97d0a0a54e828c3b0902976e0500 (DEFAULT_TABLE table=default.Table_a10aba559706443aae94d509cbb5eefa [id=bec84f8866b44454a38303786e974759]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.093015 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 8a06fc827d204d02ab81d5eeb44f7421 (DEFAULT_TABLE table=default.Table_adc39e4ac3924f079897f27ed4f1981f [id=02010c592d4a48c2a862b5c294d90d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.257030 24024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 796) took 2180 ms. Trace:
I20240502 08:16:51.259092 24035 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 806) took 2171 ms. Trace:
I20240502 08:16:51.689792 24035 rpcz_store.cc:276] 0502 08:16:49.087852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.388732 (+300880us) service_pool.cc:225] Handling call
0502 08:16:49.389166 (+ 434us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.389189 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.110493 (+1721304us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.258938 (+148445us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.259074 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.689769 24024 rpcz_store.cc:276] 0502 08:16:49.076517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.158121 (+ 81604us) service_pool.cc:225] Handling call
0502 08:16:49.389366 (+231245us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.389386 (+ 20us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:50.841946 (+1452560us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.256182 (+414236us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.257014 (+ 832us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":550}
I20240502 08:16:51.259527 24039 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 816) took 2160 ms. Trace:
I20240502 08:16:51.690137 23861 leader_election.cc:302] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:51.690128 24035 tablet_service.cc:1467] Processing CreateTablet for tablet dfd897ba97204b6788fd2d99cf2a0303 (DEFAULT_TABLE table=default.Table_97534a2075bc4b7297b36916b7a5f77c [id=5823995f324846088c24d9ffd5d742c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.690161 24039 rpcz_store.cc:276] 0502 08:16:49.099373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.487972 (+388599us) service_pool.cc:225] Handling call
0502 08:16:49.872654 (+384682us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.872707 (+ 53us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:50.845206 (+972499us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.259390 (+414184us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.259510 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.688302 23756 rpcz_store.cc:276] 0502 08:16:49.108728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.009978 (+1901250us) service_pool.cc:225] Handling call
0502 08:16:51.010387 (+ 409us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.010408 (+ 21us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.027787 (+ 17379us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.167835 (+140048us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.168000 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.690455 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2c637e208044d8aaf1774f448e0527 (DEFAULT_TABLE table=default.Table_0f6ad9ecb09e4a03b995f7350e0a3af0 [id=0b630609c7294606bff513bbcd28c4a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.261494 24026 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 813) took 2165 ms. Trace:
I20240502 08:16:51.251924 23993 leader_election.cc:302] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:51.690688 24026 rpcz_store.cc:276] 0502 08:16:49.095634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.473170 (+377536us) service_pool.cc:225] Handling call
0502 08:16:50.740267 (+1267097us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.740293 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.113532 (+373239us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.261342 (+147810us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.261449 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.690766 23756 tablet_service.cc:1467] Processing CreateTablet for tablet d1218fd56b8a4e88827f275f952ad782 (DEFAULT_TABLE table=default.Table_f17a7a3f2d4547799bdbea376e3771b2 [id=8a6b12ac10854ce986dc1ad7c67b3153]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:51.690825 23993 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 438962us
I20240502 08:16:51.691011 24026 tablet_service.cc:1467] Processing CreateTablet for tablet d1218fd56b8a4e88827f275f952ad782 (DEFAULT_TABLE table=default.Table_f17a7a3f2d4547799bdbea376e3771b2 [id=8a6b12ac10854ce986dc1ad7c67b3153]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.691072 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1218fd56b8a4e88827f275f952ad782. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.691210 23993 leader_election.cc:302] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:51.481071 25219 raft_consensus.cc:482] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:51.691506 25219 raft_consensus.cc:504] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.692039 25219 leader_election.cc:288] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:51.692171 25219 raft_consensus.cc:2781] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:51.692270 25219 raft_consensus.cc:686] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:51.690128 24024 tablet_service.cc:1467] Processing CreateTablet for tablet d87820f2490f4631b455e8dc942f7147 (DEFAULT_TABLE table=default.Table_fc2d835b22c1417bad53bb6f66bde481 [id=d361511883cd45d28d4965e0a718e0f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.692492 25219 consensus_queue.cc:237] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.184300 23763 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 853) took 2080 ms. Trace:
I20240502 08:16:51.693682 25031 raft_consensus.cc:2781] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:51.693615 23763 rpcz_store.cc:276] 0502 08:16:49.103852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:50.996299 (+1892447us) service_pool.cc:225] Handling call
0502 08:16:50.996786 (+ 487us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.996814 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.070414 (+ 73600us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.184079 (+113665us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.184241 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.693769 25031 raft_consensus.cc:686] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:51.670221 25154 raft_consensus.cc:504] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.694021 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 1f42627a877c4f83a75320f1b5ab539f (DEFAULT_TABLE table=default.Table_5fe628d3da624233941f440e3743da88 [id=ca161ce6780b446fa5743d3974cfa27b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.693982 25031 consensus_queue.cc:237] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.694243 25154 leader_election.cc:288] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:51.694305 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f42627a877c4f83a75320f1b5ab539f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.695170 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c377e037894728ac084b6500ad20de" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:51.695353 23786 raft_consensus.cc:3037] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:51.093556 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8eccf0ea5e43819f0b8fe364c4749d (DEFAULT_TABLE table=default.Table_a92dfc1659cf4b24a49358808fd9663b [id=44cf976481e048f79a382ed36b30446b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.187109 23768 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 824) took 2117 ms. Trace:
I20240502 08:16:51.697809 23768 rpcz_store.cc:276] 0502 08:16:49.069987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.070039 (+ 52us) service_pool.cc:225] Handling call
0502 08:16:49.345920 (+275881us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.345945 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.070236 (+1724291us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.186932 (+116696us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.187084 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.698232 23768 tablet_service.cc:1467] Processing CreateTablet for tablet ed545e81a6934cf8972ff84181048156 (DEFAULT_TABLE table=default.Table_33297782b5e64503b4186fe4a347f8e9 [id=d68c0a6c1e6441d18a6c08e4b1373ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.698551 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed545e81a6934cf8972ff84181048156. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.680089 23601 rpcz_store.cc:276] 0502 08:16:49.634813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.953810 (+318997us) service_pool.cc:225] Handling call
0502 08:16:50.747099 (+793289us) catalog_manager.cc:2072] Acquired catalog manager lock
0502 08:16:50.747633 (+ 534us) catalog_manager.cc:2118] Created new table and tablet info
0502 08:16:51.419044 (+671411us) catalog_manager.cc:2178] Wrote table and tablets to system table
0502 08:16:51.419163 (+ 119us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0502 08:16:51.419292 (+ 129us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:51.226825 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:51.226964 (+ 139us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:51.226971 (+ 7us) write_op.cc:427] Acquired schema lock
0502 08:16:51.226975 (+ 4us) tablet.cc:660] Decoding operations
0502 08:16:51.227066 (+ 91us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:51.227078 (+ 12us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:51.227086 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0502 08:16:51.227178 (+ 92us) tablet.cc:699] Row locks acquired
0502 08:16:51.227182 (+ 4us) write_op.cc:260] PREPARE: finished
0502 08:16:51.227213 (+ 31us) write_op.cc:270] Start()
0502 08:16:51.227249 (+ 36us) write_op.cc:276] Timestamp: P: 1714637811227108 usec, L: 0
0502 08:16:51.227255 (+ 6us) op_driver.cc:331] REPLICATION: starting
0502 08:16:51.227417 (+ 162us) log.cc:842] Serialized 433 byte log entry
0502 08:16:51.415307 (+187890us) op_driver.cc:444] REPLICATION: finished
0502 08:16:51.415515 (+ 208us) write_op.cc:301] APPLY: starting
0502 08:16:51.415532 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:51.415696 (+ 164us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:51.415875 (+ 179us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:51.418605 (+ 2730us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:51.418633 (+ 28us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0502 08:16:51.418641 (+ 8us) write_op.cc:312] APPLY: finished
0502 08:16:51.418761 (+ 120us) log.cc:842] Serialized 29 byte log entry
0502 08:16:51.418887 (+ 126us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.418904 (+ 17us) write_op.cc:446] Released schema lock
0502 08:16:51.418910 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:51.419037 (+ 127us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.419044 (+ 7us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":73,"apply.run_cpu_time_us":881,"apply.run_wall_time_us":3614,"num_ops":2,"prepare.queue_time_us":478737,"prepare.run_cpu_time_us":664,"prepare.run_wall_time_us":662,"replication_time_us":187943}]]}
I20240502 08:16:51.563931 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 303a0a3748ee422faab856e1c4bff66d
I20240502 08:16:51.571493 23576 master_runner.cc:186] Error getting master registration for 127.22.225.125:36189: OK, instance_id { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" instance_seqno: 1714637809148659 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240502 08:16:51.700618 23576 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240502 08:16:51.563696 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 29f6c3cea4b14364b6d073a92171119d
I20240502 08:16:51.093736 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 0902a73da6e34e8e9cd59bb53a36ec0e (DEFAULT_TABLE table=default.Table_722b7dc3bd6f4bdcb2ffc44d2b77e0a1 [id=3f8d5d0f6bf04fc0a5f354c3bb24f146]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.189982 23765 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 861) took 2077 ms. Trace:
I20240502 08:16:51.196702 23750 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 843) took 2103 ms. Trace:
I20240502 08:16:51.701714 23765 rpcz_store.cc:276] 0502 08:16:49.112642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.015260 (+1902618us) service_pool.cc:225] Handling call
0502 08:16:51.015640 (+ 380us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.015662 (+ 22us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.053681 (+ 38019us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.189752 (+136071us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.189937 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.701866 23750 rpcz_store.cc:276] 0502 08:16:49.092967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.712593 (+619626us) service_pool.cc:225] Handling call
0502 08:16:50.732999 (+1020406us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.733034 (+ 35us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.031487 (+298453us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.196525 (+165038us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.196674 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.702133 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 7ecde716a0c643cdb0698a37cde1c906 (DEFAULT_TABLE table=default.Table_e00f744a28c6496e98f875d47bdccf95 [id=bdc6f08aa8d144b88f0612bcd8bd8ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.093822 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 9db504396f7546a89e74b6f4530b70cc (DEFAULT_TABLE table=default.Table_4b0a59aad44148b6b3656ff66dfca548 [id=46be1f5fe512486f8e8d9d6e13165f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.682830 24173 tablet_bootstrap.cc:654] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:51.702451 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ecde716a0c643cdb0698a37cde1c906. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.203083 23753 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 854) took 2098 ms. Trace:
I20240502 08:16:51.702536 23753 rpcz_store.cc:276] 0502 08:16:49.104983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.000492 (+1895509us) service_pool.cc:225] Handling call
0502 08:16:51.000859 (+ 367us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.000883 (+ 24us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.075494 (+ 74611us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.202873 (+127379us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.203058 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.702863 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 36dece738afd42b99f3f38d7fd11a0af (DEFAULT_TABLE table=default.Table_b1d809964c99404891db2d83cb16887f [id=6851bfc588bd45d097ed07ee8ad1b81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.703140 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36dece738afd42b99f3f38d7fd11a0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.200259 23749 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 858) took 2090 ms. Trace:
I20240502 08:16:51.705170 23749 rpcz_store.cc:276] 0502 08:16:49.109706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.013908 (+1904202us) service_pool.cc:225] Handling call
0502 08:16:51.014304 (+ 396us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.014331 (+ 27us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.072626 (+ 58295us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.200038 (+127412us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.200214 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.097610 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0ff1c3818a49d2b13f57ea6a81c3d6 (DEFAULT_TABLE table=default.Table_ddf5aa41088d4de4bcf2768d82c7d1fb [id=e9a2a5d8e81e4163b64dc4da67ed09c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.705543 23749 tablet_service.cc:1467] Processing CreateTablet for tablet af5d13f956354acca041913011923840 (DEFAULT_TABLE table=default.Table_69664f97f411484e8277f8a0a02b6f90 [id=562481e7fd0c4211a090d3e382bfd64d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:51.102393 23858 service_pool.cc:135] DeleteTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:33620 dropped due to backpressure. The service queue is full; it has 50 items.
I20240502 08:16:51.705855 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af5d13f956354acca041913011923840. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.706207 23858 service_pool.cc:138] DeleteTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:33620 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 808)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 809)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 810)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 811)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 812)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 813)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 814)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 815)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 816)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 817)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 818)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 819)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 820)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 821)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 822)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 823)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 824)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 825)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 826)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 827)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 828)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 829)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 830)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 831)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 832)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 833)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 834)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 835)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 836)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 837)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 838)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 839)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 840)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 841)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 842)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 843)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 844)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 845)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 846)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 847)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 848)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 849)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 850)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 851)
I20240502 08:16:51.707751 23858 service_pool.cc:138] CreateTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:33620 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 808)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 809)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 810)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 811)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 812)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 813)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 814)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 815)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 816)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 817)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 818)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 819)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 820)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 821)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 822)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 823)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 824)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 825)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 826)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 827)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 828)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 829)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 830)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 831)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 832)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 833)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 834)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 835)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 836)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 837)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 838)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 839)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 840)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 841)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 842)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 843)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 844)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 845)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 846)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 847)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 848)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 849)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 850)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 851)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 853)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 854)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 855)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 856)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 857)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 858)
I20240502 08:16:51.207643 23757 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 852) took 2104 ms. Trace:
I20240502 08:16:51.708719 23757 rpcz_store.cc:276] 0502 08:16:49.102769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:50.989936 (+1887167us) service_pool.cc:225] Handling call
0502 08:16:51.003537 (+ 13601us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.003560 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.069724 (+ 66164us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.207436 (+137712us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.207615 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.708626 23858 service_pool.cc:138] DeleteTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:33620 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 808)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 809)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 810)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 811)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 812)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 813)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 814)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 815)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 816)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 817)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 818)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 819)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 820)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 821)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 822)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 823)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 824)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 825)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 826)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 827)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 828)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 829)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 830)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 831)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 832)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 833)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 834)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 835)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 836)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 837)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 838)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 839)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 840)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 841)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 842)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 843)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 844)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 845)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 846)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 847)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 848)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 849)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 850)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 851)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 853)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 854)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 855)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 856)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 857)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 858)
I20240502 08:16:51.709065 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3570787d8242338d1ce01fb0299eb2 (DEFAULT_TABLE table=default.Table_d03c41e82d684920af0c093caa7cd1e6 [id=1a0ddcb26d794ee99068fa11c3b345e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.709340 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3570787d8242338d1ce01fb0299eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.118677 24175 tablet_bootstrap.cc:492] T 1ba1baa2279045b3adc58d187a12cbf3 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:51.709757 24175 ts_tablet_manager.cc:1367] T 1ba1baa2279045b3adc58d187a12cbf3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 2.259s user 0.009s sys 0.015s
I20240502 08:16:51.710487 24175 raft_consensus.cc:348] T 1ba1baa2279045b3adc58d187a12cbf3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.710698 24175 raft_consensus.cc:374] T 1ba1baa2279045b3adc58d187a12cbf3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:51.710755 24175 raft_consensus.cc:729] T 1ba1baa2279045b3adc58d187a12cbf3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:51.710973 24175 consensus_queue.cc:260] T 1ba1baa2279045b3adc58d187a12cbf3 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.711443 24175 ts_tablet_manager.cc:1398] T 1ba1baa2279045b3adc58d187a12cbf3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:51.711781 24175 tablet_bootstrap.cc:492] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:51.125963 25245 sasl_common.cc:323] Time spent creating new SASL kudu client: real 0.291s user 0.001s sys 0.000s
I20240502 08:16:51.161190 23918 raft_consensus.cc:2454] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:51.712096 23918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.22.225.67:34945 (request call id 111) took 2037 ms. Trace:
I20240502 08:16:51.207930 23755 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 855) took 2101 ms. Trace:
I20240502 08:16:51.712267 25285 raft_consensus.cc:2781] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
W20240502 08:16:51.712975 23996 leader_election.cc:341] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:51.720252 23996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 467793us
I20240502 08:16:51.712313 23755 rpcz_store.cc:276] 0502 08:16:49.106065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.005400 (+1899335us) service_pool.cc:225] Handling call
0502 08:16:51.005829 (+ 429us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.005858 (+ 29us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.078033 (+ 72175us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.207759 (+129726us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.207895 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:16:51.720880 23996 leader_election.cc:341] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:51.720978 23996 leader_election.cc:302] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:51.721118 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad53633ce57478194437f95135d81de (DEFAULT_TABLE table=default.Table_9522d368c4c84f25bce3eb822a9bcad8 [id=e07e5990e2244361bcda7aa9108994b1]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:51.721313 23996 leader_election.cc:341] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Not found: Tablet not found: 27346a7eae6a488fa3415646262a4b88
I20240502 08:16:51.721447 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad53633ce57478194437f95135d81de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.721417 23996 leader_election.cc:302] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:51.712386 25285 raft_consensus.cc:686] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:16:51.721884 25222 raft_consensus.cc:2726] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:51.722038 25222 raft_consensus.cc:2726] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:51.722136 25285 consensus_queue.cc:237] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.722481 24022 tablet_service.cc:1467] Processing CreateTablet for tablet e6e633f26b0e4be38e93af3d65e2e636 (DEFAULT_TABLE table=default.Table_db2ecaab8195464388e2ba94695fce03 [id=490733fbe9e84df1b842fa52fd197e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.235643 25274 tablet.cc:1722] T 27093862fef54c5c85d810ebeff9819c P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:51.089471 23903 tablet_service.cc:1467] Processing CreateTablet for tablet c96c93bb20744ee78a609742b2cbab2a (DEFAULT_TABLE table=default.Table_a2055a6e13d8495b83033c1f6596b54f [id=344c30da8e9e4615ba703b4a879ad979]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.255537 23916 raft_consensus.cc:2454] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:51.262082 23893 tablet_service.cc:1467] Processing CreateTablet for tablet abcc0296b3c941fbad86bafa8f7965fc (DEFAULT_TABLE table=default.Table_56944338c8ef4b8a98aff14eef9fa611 [id=cdae6a1f994b46a181df44fb2c497ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.276268 23917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.225.65:36563 (request call id 228) took 2266 ms. Trace:
I20240502 08:16:51.331009 25078 raft_consensus.cc:482] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:51.723817 25078 raft_consensus.cc:504] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.723752 23917 rpcz_store.cc:276] 0502 08:16:48.912902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.912954 (+ 52us) service_pool.cc:225] Handling call
0502 08:16:48.913084 (+ 130us) raft_consensus.cc:1457] Updating replica for 2 ops
0502 08:16:50.730316 (+1817232us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:16:50.730322 (+ 6us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0502 08:16:50.730535 (+ 213us) alter_schema_op.cc:119] START. Timestamp: 1714637807365946
0502 08:16:51.109172 (+378637us) log.cc:842] Serialized 199 byte log entry
0502 08:16:51.131531 (+ 22359us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:16:51.132608 (+ 1077us) log.cc:842] Serialized 14 byte log entry
0502 08:16:51.153899 (+ 21291us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:16:51.160515 (+ 6616us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:16:51.175848 (+ 15333us) raft_consensus.cc:1671] finished
0502 08:16:51.175854 (+ 6us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:16:51.178909 (+ 3055us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:51.093508 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0502 08:16:51.093636 (+ 128us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0502 08:16:51.093647 (+ 11us) alter_schema_op.cc:59] Acquired schema lock
0502 08:16:51.093655 (+ 8us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0502 08:16:51.133288 (+ 39633us) op_driver.cc:444] REPLICATION: finished
0502 08:16:51.221706 (+ 88418us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
Metrics: {"prepare.queue_time_us":192,"prepare.run_cpu_time_us":13,"prepare.run_wall_time_us":13,"raft.queue_time_us":735,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":111,"thread_start_us":20741,"threads_started":1,"wal-append.queue_time_us":20867,"child_traces":[["op",{"apply.queue_time_us":20642,"prepare.queue_time_us":362851,"prepare.run_cpu_time_us":170,"prepare.run_wall_time_us":171,"replication_time_us":402706,"thread_start_us":381124,"threads_started":2}]]}
I20240502 08:16:51.724403 25078 leader_election.cc:288] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:51.166642 25012 ts_tablet_manager.cc:1888] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:51.244561 23767 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 860) took 2132 ms. Trace:
I20240502 08:16:51.724527 23767 rpcz_store.cc:276] 0502 08:16:49.111721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.013948 (+1902227us) service_pool.cc:225] Handling call
0502 08:16:51.014386 (+ 438us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.014411 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.079769 (+ 65358us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.244316 (+164547us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.244510 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.712213 23918 rpcz_store.cc:276] 0502 08:16:49.674375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.680416 (+ 6041us) service_pool.cc:225] Handling call
0502 08:16:51.712024 (+2031608us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x616000151654. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e706b5 00007fb204b41d24 00007fb1ff03c9b6 00007fb1ff03c6e5 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:51.712054 (+ 30us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2533169408}
I20240502 08:16:51.724572 25012 log.cc:1196] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/a3043c5ed98c401b82c6e967ca3f3613
I20240502 08:16:51.709295 23858 service_pool.cc:138] DeleteTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:33620 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 808)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 809)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 810)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 811)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 812)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 813)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 814)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 815)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 816)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 817)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 818)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 819)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 820)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 821)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 822)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 823)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 824)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 825)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 826)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 827)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 828)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 829)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 830)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 831)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 832)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 833)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 834)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 835)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 836)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 837)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 838)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 839)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 840)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 841)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 842)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 843)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 844)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 845)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 846)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 847)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 848)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 849)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 850)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 851)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 853)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 854)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 855)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 856)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 857)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 858)
I20240502 08:16:51.724931 23767 tablet_service.cc:1467] Processing CreateTablet for tablet d6e589e879f245378cff543fe430be5d (DEFAULT_TABLE table=default.Table_5fb8fa42fdb945729f5026cab15051b0 [id=323ecf6fa9a1474f869f2452d7b63da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.725008 25012 ts_tablet_manager.cc:1909] T a3043c5ed98c401b82c6e967ca3f3613 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:51.725265 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e589e879f245378cff543fe430be5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.725365 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 712) took 4018 ms. Trace:
I20240502 08:16:51.725490 25012 rpcz_store.cc:276] 0502 08:16:47.706411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.706609 (+ 198us) service_pool.cc:225] Handling call
0502 08:16:50.724537 (+3017928us) ts_tablet_manager.cc:1120] Deleting tablet a3043c5ed98c401b82c6e967ca3f3613
0502 08:16:50.724546 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:50.855806 (+131260us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.166496 (+310690us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.725318 (+558822us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3017784}
I20240502 08:16:51.725894 25012 tablet_replica.cc:330] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:51.726172 25012 raft_consensus.cc:2229] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:51.726337 25012 raft_consensus.cc:2258] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:51.726969 25012 ts_tablet_manager.cc:1875] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:51.724541 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 0dae6a96776c477fb0ada8226d68ea25 (DEFAULT_TABLE table=default.Table_ed62fe1188a546ac8cf56b8f8adcdb62 [id=34a088b1f2654da4b015a4a9d9b71e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.245038 23751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 842) took 2153 ms. Trace:
I20240502 08:16:51.728643 23751 rpcz_store.cc:276] 0502 08:16:49.091864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.699356 (+607492us) service_pool.cc:225] Handling call
0502 08:16:49.699731 (+ 375us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.699756 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.073470 (+1373714us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.244895 (+171425us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.245019 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.729100 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 37b499184f31454c96f66cfdf9143d14 (DEFAULT_TABLE table=default.Table_7a0267df98fe45a583a32684e35f9416 [id=953199c936cc4d859bc7ff49dc0c2ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.729431 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b499184f31454c96f66cfdf9143d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.730254 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c96c93bb20744ee78a609742b2cbab2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.730456 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abcc0296b3c941fbad86bafa8f7965fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.118533 23723 leader_election.cc:302] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:51.264106 24174 tablet_bootstrap.cc:492] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
W20240502 08:16:51.732460 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 655089us
I20240502 08:16:51.732780 23723 leader_election.cc:302] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:51.732992 23723 leader_election.cc:302] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:51.733803 24174 tablet_bootstrap.cc:654] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:51.311637 25158 raft_consensus.cc:482] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:51.734584 25158 raft_consensus.cc:504] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.735251 25158 leader_election.cc:288] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:51.735399 25158 raft_consensus.cc:2781] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:51.735503 25158 raft_consensus.cc:686] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:51.735769 25158 consensus_queue.cc:237] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.736177 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b811b81ddf2b4dc9b5e5f80331bff294" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:51.412923 23764 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 859) took 2301 ms. Trace:
I20240502 08:16:51.736758 23764 rpcz_store.cc:276] 0502 08:16:49.110770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.013908 (+1903138us) service_pool.cc:225] Handling call
0502 08:16:51.014484 (+ 576us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.014506 (+ 22us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.017346 (+ 2840us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.410691 (+393345us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.412638 (+ 1947us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.737206 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 6a7ab5fc54834f5783f71e615bfc283a (DEFAULT_TABLE table=default.Table_e97c69e9998149d98b2954d150b8d220 [id=025b29b37a4f40539680de19b892cd15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.737370 25266 raft_consensus.cc:2781] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:51.456161 23761 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 830) took 2377 ms. Trace:
I20240502 08:16:51.480160 25241 raft_consensus.cc:482] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:51.737483 23761 rpcz_store.cc:276] 0502 08:16:49.078317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.464213 (+385896us) service_pool.cc:225] Handling call
0502 08:16:49.900715 (+436502us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.900753 (+ 38us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.171442 (+1270689us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.435726 (+264284us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.456110 (+ 20384us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1408}
I20240502 08:16:51.737679 25241 raft_consensus.cc:504] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.737934 23761 tablet_service.cc:1467] Processing CreateTablet for tablet ce8dc203c36149cc91cc56176e929da6 (DEFAULT_TABLE table=default.Table_effefe18d3cd42f7855fed709d87de8c [id=596b181baa6140c0b44a9f900e9cbb5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.738238 25241 leader_election.cc:288] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:51.738361 25241 raft_consensus.cc:2781] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:51.738438 25241 raft_consensus.cc:686] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:51.738651 25241 consensus_queue.cc:237] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.635959 25244 raft_consensus.cc:482] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:51.739552 25244 raft_consensus.cc:504] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.740132 25244 leader_election.cc:288] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:51.214536 23758 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 837) took 2128 ms. Trace:
I20240502 08:16:51.740706 23758 rpcz_store.cc:276] 0502 08:16:49.086084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.682999 (+596915us) service_pool.cc:225] Handling call
0502 08:16:49.904042 (+221043us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.904064 (+ 22us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.169821 (+1265757us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.214290 (+ 44469us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.214485 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.741171 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 39cc4ef1174a4358bad777b0471c5f76 (DEFAULT_TABLE table=default.Table_7952191ae36044e9aef62593bbe0bb27 [id=7125223862ae4d04b11a726f81b81073]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.688352 24180 log.cc:1196] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/9c072eb45e1b41eeb5aafa6d5a786944
I20240502 08:16:51.741729 24180 ts_tablet_manager.cc:1909] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:51.702133 23765 tablet_service.cc:1467] Processing CreateTablet for tablet d86d534538de4613bd148228a78e5e9a (DEFAULT_TABLE table=default.Table_b4905971ad3f4250a6b2e8f12c6ecd96 [id=5722267c6c5047fd99430504a3a5e06c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.235488 23760 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 848) took 2137 ms. Trace:
I20240502 08:16:51.742195 23760 rpcz_store.cc:276] 0502 08:16:49.098306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.896386 (+798080us) service_pool.cc:225] Handling call
0502 08:16:49.896809 (+ 423us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:49.896830 (+ 21us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.169395 (+1272565us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.235266 (+ 65871us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.235436 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.742503 23786 raft_consensus.cc:2454] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:51.742569 23760 tablet_service.cc:1467] Processing CreateTablet for tablet b7339f1207924812b4ef2ac7ccffc894 (DEFAULT_TABLE table=default.Table_375988d4437044bc890604e6d5ab8af2 [id=6c14f79d3913425e9c0a0d0b524c1c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.744694 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7ab5fc54834f5783f71e615bfc283a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.744923 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce8dc203c36149cc91cc56176e929da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.745127 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7339f1207924812b4ef2ac7ccffc894. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.745663 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39cc4ef1174a4358bad777b0471c5f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.746250 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d86d534538de4613bd148228a78e5e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.727881 23858 service_pool.cc:138] DeleteTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:33620 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 808)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 809)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 810)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 811)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 812)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 813)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 814)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 815)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 816)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 817)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 818)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 819)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 820)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 821)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 822)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 823)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 824)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 825)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 826)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 827)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 828)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 829)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 830)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 831)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 832)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 833)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 834)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 835)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 836)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 837)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 838)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 839)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 840)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 841)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 842)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 843)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 844)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 845)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 846)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 847)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 848)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 849)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 850)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 851)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 853)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 854)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 855)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 856)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 857)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 858)
I20240502 08:16:51.747960 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a0ff1c3818a49d2b13f57ea6a81c3d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.748225 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0902a73da6e34e8e9cd59bb53a36ec0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.684799 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548ab3f9d0d84527b1092ae715ec2bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.751762 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d87820f2490f4631b455e8dc942f7147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.751896 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db504396f7546a89e74b6f4530b70cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.686713 24038 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 826) took 2576 ms. Trace:
I20240502 08:16:51.752319 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd897ba97204b6788fd2d99cf2a0303. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.752231 24038 rpcz_store.cc:276] 0502 08:16:49.110277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.673851 (+2563574us) service_pool.cc:225] Handling call
0502 08:16:51.674284 (+ 433us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.674312 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.677365 (+ 3053us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.686546 (+ 9181us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.686693 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.670318 24051 rpcz_store.cc:276] 0502 08:16:49.947834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.947871 (+ 37us) service_pool.cc:225] Handling call
0502 08:16:51.670156 (+1722285us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.717130 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c377e037894728ac084b6500ad20de" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:51.753175 24055 raft_consensus.cc:2379] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in this term.
I20240502 08:16:51.753175 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 1f42627a877c4f83a75320f1b5ab539f (DEFAULT_TABLE table=default.Table_5fe628d3da624233941f440e3743da88 [id=ca161ce6780b446fa5743d3974cfa27b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.753836 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f42627a877c4f83a75320f1b5ab539f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.756225 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 506644a1ead54a1faeaa8583ef96b449. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.757189 24024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 833) took 2639 ms. Trace:
I20240502 08:16:51.757257 24024 rpcz_store.cc:276] 0502 08:16:49.117737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.689977 (+2572240us) service_pool.cc:225] Handling call
0502 08:16:51.692665 (+ 2688us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.692682 (+ 17us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.754568 (+ 61886us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.757039 (+ 2471us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.757172 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.757733 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1218fd56b8a4e88827f275f952ad782. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.757939 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98f4f2475d4848018414c3025a54488e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.758173 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2c637e208044d8aaf1774f448e0527. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.759032 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af208d6231a4cf18823e9fe367ca46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.774304 23918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c377e037894728ac084b6500ad20de" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:51.774529 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d658ae1f967a4e0aaedf80da510bae21 consensus state has no leader
I20240502 08:16:51.774745 23610 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36668 (request call id 1038) took 1482 ms. Trace:
I20240502 08:16:51.774840 23610 rpcz_store.cc:276] 0502 08:16:50.294090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.678311 (+1384221us) service_pool.cc:225] Handling call
0502 08:16:51.678764 (+ 453us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0502 08:16:51.679013 (+ 249us) catalog_manager.cc:3613] Apply alter schema
0502 08:16:51.679187 (+ 174us) catalog_manager.cc:3636] Acquired catalog manager lock
0502 08:16:51.679263 (+ 76us) catalog_manager.cc:3783] Updating metadata on disk
0502 08:16:51.774266 (+ 95003us) catalog_manager.cc:3814] Committing alterations to in-memory state
0502 08:16:51.774679 (+ 413us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:51.679873 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:51.679962 (+ 89us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:51.679975 (+ 13us) write_op.cc:427] Acquired schema lock
0502 08:16:51.679978 (+ 3us) tablet.cc:660] Decoding operations
0502 08:16:51.680086 (+ 108us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:51.680101 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:51.680107 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0502 08:16:51.680179 (+ 72us) tablet.cc:699] Row locks acquired
0502 08:16:51.680182 (+ 3us) write_op.cc:260] PREPARE: finished
0502 08:16:51.680223 (+ 41us) write_op.cc:270] Start()
0502 08:16:51.680259 (+ 36us) write_op.cc:276] Timestamp: P: 1714637811680171 usec, L: 0
0502 08:16:51.680264 (+ 5us) op_driver.cc:331] REPLICATION: starting
0502 08:16:51.680459 (+ 195us) log.cc:842] Serialized 405 byte log entry
0502 08:16:51.687643 (+ 7184us) op_driver.cc:444] REPLICATION: finished
0502 08:16:51.688592 (+ 949us) write_op.cc:301] APPLY: starting
0502 08:16:51.688611 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:51.688655 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:51.688692 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:51.688748 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:51.688785 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0502 08:16:51.688792 (+ 7us) write_op.cc:312] APPLY: finished
0502 08:16:51.688878 (+ 86us) log.cc:842] Serialized 23 byte log entry
0502 08:16:51.688950 (+ 72us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.688982 (+ 32us) write_op.cc:446] Released schema lock
0502 08:16:51.688990 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:51.689059 (+ 69us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.689062 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":881,"apply.run_cpu_time_us":526,"apply.run_wall_time_us":527,"num_ops":1,"prepare.queue_time_us":64,"prepare.run_cpu_time_us":717,"prepare.run_wall_time_us":722,"raft.queue_time_us":63,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":234,"replication_time_us":7347}]]}
I20240502 08:16:51.775779 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1136fadacc774d899c0461e8a0579376" } modify_external_catalogs: true
I20240502 08:16:51.775979 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1136fadacc774d899c0461e8a0579376" } modify_external_catalogs: true
I20240502 08:16:51.777217 23918 raft_consensus.cc:2379] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 149ce744feab49b5aa5bcaa14bd6aba5 in current term 1: Already voted for candidate 5b89a1f0485a44b7ac1710e2369a809b in this term.
I20240502 08:16:51.778236 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f02efa659e457ab57bc32ed3f66bb9" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:51.782066 23595 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 981b7a7292884b1fb48af99db6e762c0 (table default.Table_81825c4d3ce8457ba17891e473aacddc [id=a681d30fb3e5469aad1bc27dfdd56ef7], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 981b7a7292884b1fb48af99db6e762c0 consensus state has no leader
I20240502 08:16:51.783218 23609 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36668 (request call id 1030) took 2275 ms. Trace:
I20240502 08:16:51.783334 23609 rpcz_store.cc:276] 0502 08:16:49.507972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.926901 (+418929us) service_pool.cc:225] Handling call
0502 08:16:49.939136 (+ 12235us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x616000021d54. stack: 00007f7e39fca756 00007f7e39fca510 00007f7e38ffd48d 0000000000955804 00000000009557cd 00007f7e48b0915d 00007f7e48b09253 00007f7e4583bd41 00007f7e4d42734d 00007f7e4d5b4ca6 00007f7e4439df66 00007f7e4439dc95 00007f7e40cd204e 00007f7e40cd0f2e 00007f7e40cd7405 00007f7e40cd7d2f
0502 08:16:50.738503 (+799367us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0502 08:16:50.738626 (+ 123us) catalog_manager.cc:2749] Modifying in-memory table state
0502 08:16:50.738678 (+ 52us) catalog_manager.cc:2758] Locking tablets
0502 08:16:50.738832 (+ 154us) catalog_manager.cc:2772] Removing table and tablets from system table
0502 08:16:51.413646 (+674814us) catalog_manager.cc:2790] Removing table from by-name map
0502 08:16:51.413785 (+ 139us) catalog_manager.cc:2806] Committing in-memory state
0502 08:16:51.413871 (+ 86us) catalog_manager.cc:2810] Aborting table tasks
0502 08:16:51.783162 (+369291us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:51.149867 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:51.151262 (+ 1395us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:51.151272 (+ 10us) write_op.cc:427] Acquired schema lock
0502 08:16:51.151275 (+ 3us) tablet.cc:660] Decoding operations
0502 08:16:51.153167 (+ 1892us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:51.153181 (+ 14us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:51.153367 (+ 186us) tablet.cc:683] Acquiring locks for 2 operations
0502 08:16:51.224653 (+ 71286us) tablet.cc:699] Row locks acquired
0502 08:16:51.224659 (+ 6us) write_op.cc:260] PREPARE: finished
0502 08:16:51.224748 (+ 89us) write_op.cc:270] Start()
0502 08:16:51.224800 (+ 52us) write_op.cc:276] Timestamp: P: 1714637811224635 usec, L: 0
0502 08:16:51.224807 (+ 7us) op_driver.cc:331] REPLICATION: starting
0502 08:16:51.225099 (+ 292us) log.cc:842] Serialized 762 byte log entry
0502 08:16:51.412612 (+187513us) op_driver.cc:444] REPLICATION: finished
0502 08:16:51.412899 (+ 287us) write_op.cc:301] APPLY: starting
0502 08:16:51.412922 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:51.412979 (+ 57us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:51.413019 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:51.413113 (+ 94us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:51.413157 (+ 44us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0502 08:16:51.413167 (+ 10us) write_op.cc:312] APPLY: finished
0502 08:16:51.413271 (+ 104us) log.cc:842] Serialized 29 byte log entry
0502 08:16:51.413358 (+ 87us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.413400 (+ 42us) write_op.cc:446] Released schema lock
0502 08:16:51.413413 (+ 13us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:51.413492 (+ 79us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.413499 (+ 7us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":27096320,"child_traces":[["op",{"apply.queue_time_us":195,"apply.run_cpu_time_us":679,"apply.run_wall_time_us":679,"num_ops":2,"prepare.queue_time_us":409517,"prepare.run_cpu_time_us":1732,"prepare.run_wall_time_us":75971,"replication_time_us":187771}]]}
I20240502 08:16:51.784659 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_447f4a9a45334a60a8f4fbb914b1e546"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:51.791584 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_447f4a9a45334a60a8f4fbb914b1e546 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:51.786480 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80ca7b4c29584de0b0fddf31e64d9f0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:51.787098 23604 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36668 (request call id 1031) took 2276 ms. Trace:
I20240502 08:16:51.793895 23604 rpcz_store.cc:276] 0502 08:16:49.511052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.941854 (+430802us) service_pool.cc:225] Handling call
0502 08:16:49.954151 (+ 12297us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x616000021d54. stack: 00007f7e39fca756 00007f7e39fca510 00007f7e38ffd48d 0000000000955804 00000000009557cd 00007f7e48b0915d 00007f7e48b09253 00007f7e4583bd41 00007f7e4d42734d 00007f7e4d5b4ca6 00007f7e4439df66 00007f7e4439dc95 00007f7e40cd204e 00007f7e40cd0f2e 00007f7e40cd7405 00007f7e40cd7d2f
0502 08:16:50.753077 (+798926us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0502 08:16:50.753227 (+ 150us) catalog_manager.cc:2749] Modifying in-memory table state
0502 08:16:50.753287 (+ 60us) catalog_manager.cc:2758] Locking tablets
0502 08:16:51.498677 (+745390us) catalog_manager.cc:2772] Removing table and tablets from system table
0502 08:16:51.563297 (+ 64620us) catalog_manager.cc:2790] Removing table from by-name map
0502 08:16:51.563529 (+ 232us) catalog_manager.cc:2806] Committing in-memory state
0502 08:16:51.563595 (+ 66us) catalog_manager.cc:2810] Aborting table tasks
0502 08:16:51.787055 (+223460us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:51.501862 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:51.501999 (+ 137us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:51.502005 (+ 6us) write_op.cc:427] Acquired schema lock
0502 08:16:51.502008 (+ 3us) tablet.cc:660] Decoding operations
0502 08:16:51.502140 (+ 132us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:51.502156 (+ 16us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:51.502162 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0502 08:16:51.502290 (+ 128us) tablet.cc:699] Row locks acquired
0502 08:16:51.502294 (+ 4us) write_op.cc:260] PREPARE: finished
0502 08:16:51.502330 (+ 36us) write_op.cc:270] Start()
0502 08:16:51.502366 (+ 36us) write_op.cc:276] Timestamp: P: 1714637811502293 usec, L: 0
0502 08:16:51.502371 (+ 5us) op_driver.cc:331] REPLICATION: starting
0502 08:16:51.502554 (+ 183us) log.cc:842] Serialized 731 byte log entry
0502 08:16:51.562465 (+ 59911us) op_driver.cc:444] REPLICATION: finished
0502 08:16:51.562719 (+ 254us) write_op.cc:301] APPLY: starting
0502 08:16:51.562741 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:51.562793 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:51.562828 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:51.562909 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:51.562966 (+ 57us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0502 08:16:51.562975 (+ 9us) write_op.cc:312] APPLY: finished
0502 08:16:51.563069 (+ 94us) log.cc:842] Serialized 29 byte log entry
0502 08:16:51.563139 (+ 70us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.563162 (+ 23us) write_op.cc:446] Released schema lock
0502 08:16:51.563169 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:51.563240 (+ 71us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.563247 (+ 7us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":27792768,"child_traces":[["op",{"apply.queue_time_us":178,"apply.run_cpu_time_us":587,"apply.run_wall_time_us":587,"num_ops":2,"prepare.queue_time_us":2427,"prepare.run_cpu_time_us":762,"prepare.run_wall_time_us":762,"replication_time_us":60062}]]}
I20240502 08:16:51.795578 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_4e432e7524d342fa8de7aedc283e9a2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:51.796810 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e432e7524d342fa8de7aedc283e9a2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:51.761555 24034 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 830) took 2647 ms. Trace:
I20240502 08:16:51.797494 24034 rpcz_store.cc:276] 0502 08:16:49.114392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.688557 (+2574165us) service_pool.cc:225] Handling call
0502 08:16:51.688862 (+ 305us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.688885 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.758818 (+ 69933us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.761370 (+ 2552us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.761536 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:16:51.787639 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ff1ae77ef32484abb840fd10fcb88e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:51.788689 23605 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36668 (request call id 1016) took 2836 ms. Trace:
W20240502 08:16:51.798909 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 65922.1us
I20240502 08:16:51.799036 23605 rpcz_store.cc:276] 0502 08:16:48.952582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.061760 (+109178us) service_pool.cc:225] Handling call
0502 08:16:49.905655 (+843895us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0502 08:16:49.905823 (+ 168us) catalog_manager.cc:2749] Modifying in-memory table state
0502 08:16:49.905890 (+ 67us) catalog_manager.cc:2758] Locking tablets
0502 08:16:51.498840 (+1592950us) catalog_manager.cc:2772] Removing table and tablets from system table
0502 08:16:51.563756 (+ 64916us) catalog_manager.cc:2790] Removing table from by-name map
0502 08:16:51.563842 (+ 86us) catalog_manager.cc:2806] Committing in-memory state
0502 08:16:51.563885 (+ 43us) catalog_manager.cc:2810] Aborting table tasks
0502 08:16:51.788619 (+224734us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:51.502645 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:51.502744 (+ 99us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:51.502750 (+ 6us) write_op.cc:427] Acquired schema lock
0502 08:16:51.502753 (+ 3us) tablet.cc:660] Decoding operations
0502 08:16:51.502883 (+ 130us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:51.502892 (+ 9us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:51.502896 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0502 08:16:51.503009 (+ 113us) tablet.cc:699] Row locks acquired
0502 08:16:51.503013 (+ 4us) write_op.cc:260] PREPARE: finished
0502 08:16:51.503045 (+ 32us) write_op.cc:270] Start()
0502 08:16:51.503075 (+ 30us) write_op.cc:276] Timestamp: P: 1714637811503009 usec, L: 0
0502 08:16:51.503081 (+ 6us) op_driver.cc:331] REPLICATION: starting
0502 08:16:51.503233 (+ 152us) log.cc:842] Serialized 731 byte log entry
0502 08:16:51.562558 (+ 59325us) op_driver.cc:444] REPLICATION: finished
0502 08:16:51.563341 (+ 783us) write_op.cc:301] APPLY: starting
0502 08:16:51.563353 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:51.563384 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:51.563405 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:51.563460 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:51.563495 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0502 08:16:51.563505 (+ 10us) write_op.cc:312] APPLY: finished
0502 08:16:51.563565 (+ 60us) log.cc:842] Serialized 29 byte log entry
0502 08:16:51.563625 (+ 60us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.563642 (+ 17us) write_op.cc:446] Released schema lock
0502 08:16:51.563647 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:51.563700 (+ 53us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.563704 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":896,"child_traces":[["op",{"apply.queue_time_us":751,"apply.run_cpu_time_us":406,"apply.run_wall_time_us":406,"num_ops":2,"prepare.queue_time_us":3070,"prepare.run_cpu_time_us":644,"prepare.run_wall_time_us":643,"replication_time_us":59460}]]}
I20240502 08:16:51.799898 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fd05e15f7383485b9966bdf3ab0ab045"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:51.800410 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd05e15f7383485b9966bdf3ab0ab045 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:51.789036 23603 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36668 (request call id 1029) took 2281 ms. Trace:
I20240502 08:16:51.801545 23603 rpcz_store.cc:276] 0502 08:16:49.507819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.920242 (+412423us) service_pool.cc:225] Handling call
0502 08:16:50.733864 (+813622us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0502 08:16:50.733993 (+ 129us) catalog_manager.cc:2749] Modifying in-memory table state
0502 08:16:50.734043 (+ 50us) catalog_manager.cc:2758] Locking tablets
0502 08:16:50.734161 (+ 118us) catalog_manager.cc:2772] Removing table and tablets from system table
0502 08:16:51.358910 (+624749us) catalog_manager.cc:2790] Removing table from by-name map
0502 08:16:51.359015 (+ 105us) catalog_manager.cc:2806] Committing in-memory state
0502 08:16:51.359056 (+ 41us) catalog_manager.cc:2810] Aborting table tasks
0502 08:16:51.788981 (+429925us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:51.139863 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:51.140871 (+ 1008us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:51.140877 (+ 6us) write_op.cc:427] Acquired schema lock
0502 08:16:51.140879 (+ 2us) tablet.cc:660] Decoding operations
0502 08:16:51.142823 (+ 1944us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:51.142836 (+ 13us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:51.142969 (+ 133us) tablet.cc:683] Acquiring locks for 2 operations
0502 08:16:51.144297 (+ 1328us) tablet.cc:699] Row locks acquired
0502 08:16:51.144302 (+ 5us) write_op.cc:260] PREPARE: finished
0502 08:16:51.145013 (+ 711us) write_op.cc:270] Start()
0502 08:16:51.145262 (+ 249us) write_op.cc:276] Timestamp: P: 1714637811144666 usec, L: 0
0502 08:16:51.145466 (+ 204us) op_driver.cc:331] REPLICATION: starting
0502 08:16:51.147762 (+ 2296us) log.cc:842] Serialized 731 byte log entry
0502 08:16:51.357451 (+209689us) op_driver.cc:444] REPLICATION: finished
0502 08:16:51.358277 (+ 826us) write_op.cc:301] APPLY: starting
0502 08:16:51.358293 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:51.358334 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:51.358371 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:51.358452 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:51.358486 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0502 08:16:51.358495 (+ 9us) write_op.cc:312] APPLY: finished
0502 08:16:51.358554 (+ 59us) log.cc:842] Serialized 29 byte log entry
0502 08:16:51.358616 (+ 62us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.358633 (+ 17us) write_op.cc:446] Released schema lock
0502 08:16:51.358639 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:51.358690 (+ 51us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.358696 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":784,"apply.run_cpu_time_us":487,"apply.run_wall_time_us":486,"num_ops":2,"prepare.queue_time_us":404923,"prepare.run_cpu_time_us":1929,"prepare.run_wall_time_us":8898,"replication_time_us":211962,"thread_start_us":54852,"threads_started":1}]]}
I20240502 08:16:51.802560 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_4610f864dbdd41378b26279a24da77f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:51.803174 25154 raft_consensus.cc:482] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a297a6c7c2504ca68e87b24f6691ffb2)
W20240502 08:16:51.803318 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4610f864dbdd41378b26279a24da77f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:51.803314 25154 raft_consensus.cc:504] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.789304 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_efbec5d24d154d4ab7354736f7ef8fe5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:51.789969 23597 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 140543us
I20240502 08:16:51.809469 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 2bfb532c9d4c41afb7acaa4831841306 (table default.Table_f48b4cdee51644e0b3c904d772f01404 [id=35259dda872249a7bdddd93783103db8]) successfully deleted
I20240502 08:16:51.815029 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 784) took 3652 ms. Trace:
I20240502 08:16:51.815099 23767 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 876) took 2686 ms. Trace:
I20240502 08:16:51.815156 23767 rpcz_store.cc:276] 0502 08:16:49.128885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.724720 (+2595835us) service_pool.cc:225] Handling call
0502 08:16:51.725091 (+ 371us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.725114 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.728323 (+ 3209us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.814961 (+ 86638us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.815088 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.815131 24180 rpcz_store.cc:276] 0502 08:16:48.162245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.162291 (+ 46us) service_pool.cc:225] Handling call
0502 08:16:49.899641 (+1737350us) ts_tablet_manager.cc:1120] Deleting tablet 9c072eb45e1b41eeb5aafa6d5a786944
0502 08:16:49.899651 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:50.861286 (+961635us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.168862 (+307576us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.814999 (+646137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1737169}
I20240502 08:16:51.815513 24180 tablet_replica.cc:330] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:51.815624 23767 tablet_service.cc:1467] Processing CreateTablet for tablet ba3b3abbb2e94c87ba19cc5695006e07 (DEFAULT_TABLE table=default.Table_39434b47dd9b421796ea5924f95292b9 [id=4c6173704b9e476fa4b836ba7fe5ff4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.816002 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3b3abbb2e94c87ba19cc5695006e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.816427 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 9c072eb45e1b41eeb5aafa6d5a786944 (table default.Table_20c5ace604d64b6cabead6037cdc04e1 [id=2e5831a5849049c8bd445667b01b1bd8]) successfully deleted
I20240502 08:16:51.818004 23757 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 874) took 2690 ms. Trace:
I20240502 08:16:51.818071 23757 rpcz_store.cc:276] 0502 08:16:49.127084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.708885 (+2581801us) service_pool.cc:225] Handling call
0502 08:16:51.709207 (+ 322us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.709225 (+ 18us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.711719 (+ 2494us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.817836 (+106117us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.817984 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.818457 23757 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:43544 (request call id 887) took 2152 ms. Trace:
I20240502 08:16:51.818516 23757 rpcz_store.cc:276] 0502 08:16:49.666173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.818249 (+2152076us) service_pool.cc:225] Handling call
0502 08:16:51.818443 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.818699 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet 642744194e22461f8316a78689fd99ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:51.818851 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet c1e746013fde4de4862f213792e235ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:51.817925 23602 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36668 (request call id 1036) took 2165 ms. Trace:
I20240502 08:16:51.819118 23757 tablet_service.cc:1467] Processing CreateTablet for tablet e34873a3f08043098f2740641b3dc41f (DEFAULT_TABLE table=default.Table_a47e8e08a0ef4797aaf83cfa8adf0019 [id=630b2f4ab433442ab82b9451c8f54f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.819417 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e34873a3f08043098f2740641b3dc41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.818948 23602 rpcz_store.cc:276] 0502 08:16:49.652666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.670916 (+2018250us) service_pool.cc:225] Handling call
0502 08:16:51.793026 (+122110us) catalog_manager.cc:2072] Acquired catalog manager lock
0502 08:16:51.793440 (+ 414us) catalog_manager.cc:2118] Created new table and tablet info
0502 08:16:51.817644 (+ 24204us) catalog_manager.cc:2178] Wrote table and tablets to system table
0502 08:16:51.817758 (+ 114us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0502 08:16:51.817875 (+ 117us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:51.805959 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:51.806050 (+ 91us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:51.806055 (+ 5us) write_op.cc:427] Acquired schema lock
0502 08:16:51.806058 (+ 3us) tablet.cc:660] Decoding operations
0502 08:16:51.806135 (+ 77us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:51.806144 (+ 9us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:51.806148 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0502 08:16:51.806212 (+ 64us) tablet.cc:699] Row locks acquired
0502 08:16:51.806214 (+ 2us) write_op.cc:260] PREPARE: finished
0502 08:16:51.806246 (+ 32us) write_op.cc:270] Start()
0502 08:16:51.806274 (+ 28us) write_op.cc:276] Timestamp: P: 1714637811806229 usec, L: 0
0502 08:16:51.806277 (+ 3us) op_driver.cc:331] REPLICATION: starting
0502 08:16:51.806454 (+ 177us) log.cc:842] Serialized 433 byte log entry
0502 08:16:51.814085 (+ 7631us) op_driver.cc:444] REPLICATION: finished
0502 08:16:51.814380 (+ 295us) write_op.cc:301] APPLY: starting
0502 08:16:51.814395 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:51.814433 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:51.814462 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:51.814563 (+ 101us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:51.814593 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0502 08:16:51.814599 (+ 6us) write_op.cc:312] APPLY: finished
0502 08:16:51.814658 (+ 59us) log.cc:842] Serialized 29 byte log entry
0502 08:16:51.814708 (+ 50us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.814724 (+ 16us) write_op.cc:446] Released schema lock
0502 08:16:51.814729 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:51.814776 (+ 47us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.814779 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":244,"apply.run_cpu_time_us":444,"apply.run_wall_time_us":444,"mutex_wait_us":11684,"num_ops":2,"prepare.queue_time_us":12091,"prepare.run_cpu_time_us":554,"prepare.run_wall_time_us":555,"replication_time_us":7784}]]}
I20240502 08:16:51.820050 23755 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 875) took 2692 ms. Trace:
I20240502 08:16:51.820118 23755 rpcz_store.cc:276] 0502 08:16:49.128032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.720873 (+2592841us) service_pool.cc:225] Handling call
0502 08:16:51.721276 (+ 403us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.721302 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.817721 (+ 96419us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.819909 (+ 2188us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.820033 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.818676 23607 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36668 (request call id 1037) took 1923 ms. Trace:
I20240502 08:16:51.820823 23607 rpcz_store.cc:276] 0502 08:16:49.895155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.672492 (+1777337us) service_pool.cc:225] Handling call
0502 08:16:51.798179 (+125687us) catalog_manager.cc:2072] Acquired catalog manager lock
0502 08:16:51.798589 (+ 410us) catalog_manager.cc:2118] Created new table and tablet info
0502 08:16:51.818441 (+ 19852us) catalog_manager.cc:2178] Wrote table and tablets to system table
0502 08:16:51.818539 (+ 98us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0502 08:16:51.818617 (+ 78us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:51.807066 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:51.807158 (+ 92us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:51.807162 (+ 4us) write_op.cc:427] Acquired schema lock
0502 08:16:51.807181 (+ 19us) tablet.cc:660] Decoding operations
0502 08:16:51.807244 (+ 63us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:51.807253 (+ 9us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:51.807260 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0502 08:16:51.807311 (+ 51us) tablet.cc:699] Row locks acquired
0502 08:16:51.807313 (+ 2us) write_op.cc:260] PREPARE: finished
0502 08:16:51.807334 (+ 21us) write_op.cc:270] Start()
0502 08:16:51.807355 (+ 21us) write_op.cc:276] Timestamp: P: 1714637811807321 usec, L: 0
0502 08:16:51.807360 (+ 5us) op_driver.cc:331] REPLICATION: starting
0502 08:16:51.807501 (+ 141us) log.cc:842] Serialized 433 byte log entry
0502 08:16:51.814187 (+ 6686us) op_driver.cc:444] REPLICATION: finished
0502 08:16:51.815279 (+ 1092us) write_op.cc:301] APPLY: starting
0502 08:16:51.815290 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:51.815339 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:51.815371 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:51.815478 (+ 107us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:51.815504 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0502 08:16:51.815512 (+ 8us) write_op.cc:312] APPLY: finished
0502 08:16:51.815561 (+ 49us) log.cc:842] Serialized 29 byte log entry
0502 08:16:51.815608 (+ 47us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.815622 (+ 14us) write_op.cc:446] Released schema lock
0502 08:16:51.815627 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:51.815670 (+ 43us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.815673 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1062,"apply.run_cpu_time_us":432,"apply.run_wall_time_us":432,"mutex_wait_us":6734,"num_ops":2,"prepare.queue_time_us":8062,"prepare.run_cpu_time_us":487,"prepare.run_wall_time_us":486,"replication_time_us":6816,"threads_started":1}]]}
I20240502 08:16:51.826027 23604 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36668 (request call id 1042) took 1452 ms. Trace:
I20240502 08:16:51.815692 24180 raft_consensus.cc:2229] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:51.827226 23609 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36668 (request call id 1040) took 1474 ms. Trace:
I20240502 08:16:51.827325 23609 rpcz_store.cc:276] 0502 08:16:50.354335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.784105 (+1429770us) service_pool.cc:225] Handling call
0502 08:16:51.791748 (+ 7643us) catalog_manager.cc:2072] Acquired catalog manager lock
0502 08:16:51.792389 (+ 641us) catalog_manager.cc:2118] Created new table and tablet info
0502 08:16:51.826924 (+ 34535us) catalog_manager.cc:2178] Wrote table and tablets to system table
0502 08:16:51.827051 (+ 127us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0502 08:16:51.827150 (+ 99us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:51.804377 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:51.804482 (+ 105us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:51.804491 (+ 9us) write_op.cc:427] Acquired schema lock
0502 08:16:51.804495 (+ 4us) tablet.cc:660] Decoding operations
0502 08:16:51.804596 (+ 101us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:51.804612 (+ 16us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:51.804618 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0502 08:16:51.804707 (+ 89us) tablet.cc:699] Row locks acquired
0502 08:16:51.804710 (+ 3us) write_op.cc:260] PREPARE: finished
0502 08:16:51.804749 (+ 39us) write_op.cc:270] Start()
0502 08:16:51.804786 (+ 37us) write_op.cc:276] Timestamp: P: 1714637811804730 usec, L: 0
0502 08:16:51.804791 (+ 5us) op_driver.cc:331] REPLICATION: starting
0502 08:16:51.804994 (+ 203us) log.cc:842] Serialized 433 byte log entry
0502 08:16:51.810127 (+ 5133us) op_driver.cc:444] REPLICATION: finished
0502 08:16:51.810245 (+ 118us) write_op.cc:301] APPLY: starting
0502 08:16:51.810260 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:51.810306 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:51.810337 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:51.810454 (+ 117us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:51.810488 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0502 08:16:51.810494 (+ 6us) write_op.cc:312] APPLY: finished
0502 08:16:51.810565 (+ 71us) log.cc:842] Serialized 29 byte log entry
0502 08:16:51.810625 (+ 60us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.810643 (+ 18us) write_op.cc:446] Released schema lock
0502 08:16:51.810648 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:51.810702 (+ 54us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.810705 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":70,"apply.run_cpu_time_us":511,"apply.run_wall_time_us":510,"num_ops":2,"prepare.queue_time_us":11387,"prepare.run_cpu_time_us":725,"prepare.run_wall_time_us":725,"raft.queue_time_us":60,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":293,"replication_time_us":5313}]]}
I20240502 08:16:51.826169 23604 rpcz_store.cc:276] 0502 08:16:50.373925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.794633 (+1420708us) service_pool.cc:225] Handling call
0502 08:16:51.796905 (+ 2272us) catalog_manager.cc:2072] Acquired catalog manager lock
0502 08:16:51.797345 (+ 440us) catalog_manager.cc:2118] Created new table and tablet info
0502 08:16:51.825185 (+ 27840us) catalog_manager.cc:2178] Wrote table and tablets to system table
0502 08:16:51.825364 (+ 179us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0502 08:16:51.825984 (+ 620us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:51.806537 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:51.806615 (+ 78us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:51.806619 (+ 4us) write_op.cc:427] Acquired schema lock
0502 08:16:51.806621 (+ 2us) tablet.cc:660] Decoding operations
0502 08:16:51.806701 (+ 80us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:51.806709 (+ 8us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:51.806713 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0502 08:16:51.806792 (+ 79us) tablet.cc:699] Row locks acquired
0502 08:16:51.806794 (+ 2us) write_op.cc:260] PREPARE: finished
0502 08:16:51.806819 (+ 25us) write_op.cc:270] Start()
0502 08:16:51.806840 (+ 21us) write_op.cc:276] Timestamp: P: 1714637811806804 usec, L: 0
0502 08:16:51.806843 (+ 3us) op_driver.cc:331] REPLICATION: starting
0502 08:16:51.806958 (+ 115us) log.cc:842] Serialized 433 byte log entry
0502 08:16:51.814147 (+ 7189us) op_driver.cc:444] REPLICATION: finished
0502 08:16:51.814851 (+ 704us) write_op.cc:301] APPLY: starting
0502 08:16:51.814860 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:51.814889 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:51.814908 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:51.815002 (+ 94us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:51.815026 (+ 24us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0502 08:16:51.815033 (+ 7us) write_op.cc:312] APPLY: finished
0502 08:16:51.815083 (+ 50us) log.cc:842] Serialized 29 byte log entry
0502 08:16:51.815119 (+ 36us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.815132 (+ 13us) write_op.cc:446] Released schema lock
0502 08:16:51.815137 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:51.815187 (+ 50us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.815190 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":674,"apply.run_cpu_time_us":398,"apply.run_wall_time_us":396,"mutex_wait_us":7884,"num_ops":2,"prepare.queue_time_us":8769,"prepare.run_cpu_time_us":501,"prepare.run_wall_time_us":500,"replication_time_us":7290,"spinlock_wait_cycles":48000}]]}
W20240502 08:16:51.831349 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_efbec5d24d154d4ab7354736f7ef8fe5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:51.772531 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e546c1bb78a848ad8125e8e8b08434db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.834120 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e633f26b0e4be38e93af3d65e2e636. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.834663 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88bf97d0a0a54e828c3b0902976e0500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.836723 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cebfc5ffc184de9a2e703aa4e333c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.769272 24024 tablet_service.cc:1467] Processing CreateTablet for tablet ed545e81a6934cf8972ff84181048156 (DEFAULT_TABLE table=default.Table_33297782b5e64503b4186fe4a347f8e9 [id=d68c0a6c1e6441d18a6c08e4b1373ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.779608 24039 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 834) took 2660 ms. Trace:
I20240502 08:16:51.844635 24039 rpcz_store.cc:276] 0502 08:16:49.118824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.690309 (+2571485us) service_pool.cc:225] Handling call
0502 08:16:51.690585 (+ 276us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.690602 (+ 17us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.765728 (+ 75126us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.779444 (+ 13716us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.779590 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.845494 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abcc0296b3c941fbad86bafa8f7965fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.782158 24036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 823) took 2675 ms. Trace:
I20240502 08:16:51.847010 24036 rpcz_store.cc:276] 0502 08:16:49.106648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.734574 (+627926us) service_pool.cc:225] Handling call
0502 08:16:50.748242 (+1013668us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.748258 (+ 16us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.682083 (+933825us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.781982 (+ 99899us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.782135 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.847715 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 36dece738afd42b99f3f38d7fd11a0af (DEFAULT_TABLE table=default.Table_b1d809964c99404891db2d83cb16887f [id=6851bfc588bd45d097ed07ee8ad1b81d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:51.778900 23996 leader_election.cc:341] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:51.848121 23996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 69233.6us
I20240502 08:16:51.792448 24028 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 814) took 2695 ms. Trace:
I20240502 08:16:51.848531 24028 rpcz_store.cc:276] 0502 08:16:49.097287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.478006 (+380719us) service_pool.cc:225] Handling call
0502 08:16:50.748722 (+1270716us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.748736 (+ 14us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.675168 (+926432us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.792238 (+117070us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.792415 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.849138 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36dece738afd42b99f3f38d7fd11a0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.861063 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c377e037894728ac084b6500ad20de" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:51.861411 24055 raft_consensus.cc:2379] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a297a6c7c2504ca68e87b24f6691ffb2 in current term 1: Already voted for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in this term.
I20240502 08:16:51.861523 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed545e81a6934cf8972ff84181048156. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.862210 24034 tablet_service.cc:1467] Processing CreateTablet for tablet d86d534538de4613bd148228a78e5e9a (DEFAULT_TABLE table=default.Table_b4905971ad3f4250a6b2e8f12c6ecd96 [id=5722267c6c5047fd99430504a3a5e06c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.863124 24028 tablet_service.cc:1467] Processing CreateTablet for tablet af5d13f956354acca041913011923840 (DEFAULT_TABLE table=default.Table_69664f97f411484e8277f8a0a02b6f90 [id=562481e7fd0c4211a090d3e382bfd64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.864557 23605 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36668 (request call id 1043) took 1490 ms. Trace:
I20240502 08:16:51.864598 25154 leader_election.cc:288] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:51.864679 23605 rpcz_store.cc:276] 0502 08:16:50.373997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.799684 (+1425687us) service_pool.cc:225] Handling call
0502 08:16:51.800492 (+ 808us) catalog_manager.cc:2072] Acquired catalog manager lock
0502 08:16:51.800864 (+ 372us) catalog_manager.cc:2118] Created new table and tablet info
0502 08:16:51.864207 (+ 63343us) catalog_manager.cc:2178] Wrote table and tablets to system table
0502 08:16:51.864379 (+ 172us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0502 08:16:51.864491 (+ 112us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:51.807586 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:51.807681 (+ 95us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:51.807686 (+ 5us) write_op.cc:427] Acquired schema lock
0502 08:16:51.807690 (+ 4us) tablet.cc:660] Decoding operations
0502 08:16:51.807758 (+ 68us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:51.807766 (+ 8us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:51.807770 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0502 08:16:51.807839 (+ 69us) tablet.cc:699] Row locks acquired
0502 08:16:51.807843 (+ 4us) write_op.cc:260] PREPARE: finished
0502 08:16:51.807868 (+ 25us) write_op.cc:270] Start()
0502 08:16:51.807894 (+ 26us) write_op.cc:276] Timestamp: P: 1714637811807854 usec, L: 0
0502 08:16:51.807898 (+ 4us) op_driver.cc:331] REPLICATION: starting
0502 08:16:51.808052 (+ 154us) log.cc:842] Serialized 433 byte log entry
0502 08:16:51.862588 (+ 54536us) op_driver.cc:444] REPLICATION: finished
0502 08:16:51.862799 (+ 211us) write_op.cc:301] APPLY: starting
0502 08:16:51.862821 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:51.862872 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:51.862907 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:51.863087 (+ 180us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:51.863127 (+ 40us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0502 08:16:51.863136 (+ 9us) write_op.cc:312] APPLY: finished
0502 08:16:51.863236 (+ 100us) log.cc:842] Serialized 29 byte log entry
0502 08:16:51.863329 (+ 93us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.863355 (+ 26us) write_op.cc:446] Released schema lock
0502 08:16:51.863364 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:51.863433 (+ 69us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.863440 (+ 7us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":129,"apply.run_cpu_time_us":704,"apply.run_wall_time_us":715,"mutex_wait_us":4578,"num_ops":2,"prepare.queue_time_us":6337,"prepare.run_cpu_time_us":525,"prepare.run_wall_time_us":524,"replication_time_us":54662}]]}
I20240502 08:16:51.845268 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 7ecde716a0c643cdb0698a37cde1c906 (DEFAULT_TABLE table=default.Table_e00f744a28c6496e98f875d47bdccf95 [id=bdc6f08aa8d144b88f0612bcd8bd8ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.866948 23603 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36668 (request call id 1044) took 1492 ms. Trace:
I20240502 08:16:51.867066 23603 rpcz_store.cc:276] 0502 08:16:50.374072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.802308 (+1428236us) service_pool.cc:225] Handling call
0502 08:16:51.803435 (+ 1127us) catalog_manager.cc:2072] Acquired catalog manager lock
0502 08:16:51.803821 (+ 386us) catalog_manager.cc:2118] Created new table and tablet info
0502 08:16:51.866611 (+ 62790us) catalog_manager.cc:2178] Wrote table and tablets to system table
0502 08:16:51.866753 (+ 142us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0502 08:16:51.866855 (+ 102us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:51.808156 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:51.808234 (+ 78us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:51.808243 (+ 9us) write_op.cc:427] Acquired schema lock
0502 08:16:51.808246 (+ 3us) tablet.cc:660] Decoding operations
0502 08:16:51.808309 (+ 63us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:51.808317 (+ 8us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:51.808321 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0502 08:16:51.808393 (+ 72us) tablet.cc:699] Row locks acquired
0502 08:16:51.808397 (+ 4us) write_op.cc:260] PREPARE: finished
0502 08:16:51.808424 (+ 27us) write_op.cc:270] Start()
0502 08:16:51.808448 (+ 24us) write_op.cc:276] Timestamp: P: 1714637811808408 usec, L: 0
0502 08:16:51.808452 (+ 4us) op_driver.cc:331] REPLICATION: starting
0502 08:16:51.808595 (+ 143us) log.cc:842] Serialized 433 byte log entry
0502 08:16:51.862676 (+ 54081us) op_driver.cc:444] REPLICATION: finished
0502 08:16:51.863544 (+ 868us) write_op.cc:301] APPLY: starting
0502 08:16:51.863557 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:51.863594 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:51.863629 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:51.863750 (+ 121us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:51.863782 (+ 32us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0502 08:16:51.863789 (+ 7us) write_op.cc:312] APPLY: finished
0502 08:16:51.863856 (+ 67us) log.cc:842] Serialized 29 byte log entry
0502 08:16:51.863946 (+ 90us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.863962 (+ 16us) write_op.cc:446] Released schema lock
0502 08:16:51.863968 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:51.864015 (+ 47us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.864022 (+ 7us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":832,"apply.run_cpu_time_us":523,"apply.run_wall_time_us":523,"mutex_wait_us":1650,"num_ops":2,"prepare.queue_time_us":3887,"prepare.run_cpu_time_us":490,"prepare.run_wall_time_us":489,"replication_time_us":54206}]]}
I20240502 08:16:51.824101 23749 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 873) took 2697 ms. Trace:
I20240502 08:16:51.821456 23755 tablet_service.cc:1467] Processing CreateTablet for tablet b1a2bcc6937c4a95b71103dbf8738702 (DEFAULT_TABLE table=default.Table_fc86e7c02217467c9a9e596e55aeee08 [id=a87a2af04b464ab283571bbc57ead567]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.871433 23749 rpcz_store.cc:276] 0502 08:16:49.126318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.705366 (+2579048us) service_pool.cc:225] Handling call
0502 08:16:51.705709 (+ 343us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.705729 (+ 20us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.708478 (+ 2749us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.823942 (+115464us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.824081 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.871320 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_201d8955331641b4afc23044b3706aa7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:51.871690 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet e2126bf2d4ab44628b52481dc2ec3a99 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:49 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:51.871990 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8eccf0ea5e43819f0b8fe364c4749d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:51.871980 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_201d8955331641b4afc23044b3706aa7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:51.872045 23749 tablet_service.cc:1467] Processing CreateTablet for tablet bb9b75d6be924d0291f94fc4a439dea6 (DEFAULT_TABLE table=default.Table_41d744e17dee458382533bb204fd1197 [id=e99a8a3a2c9343369d31c9852c31de9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.826836 24180 raft_consensus.cc:2258] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:51.873029 24180 ts_tablet_manager.cc:1875] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:51.828361 23753 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 872) took 2703 ms. Trace:
I20240502 08:16:51.873443 23753 rpcz_store.cc:276] 0502 08:16:49.125284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.702706 (+2577422us) service_pool.cc:225] Handling call
0502 08:16:51.702992 (+ 286us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.703014 (+ 22us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.716552 (+ 13538us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.828210 (+111658us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.828344 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.873869 23753 tablet_service.cc:1467] Processing CreateTablet for tablet d68448a2033e450794440fa84de6a19a (DEFAULT_TABLE table=default.Table_6a6b2f29d9aa4a679e4c41863e3096bd [id=38c7d40fae5b4b7eb187d2c8e43f04d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.826683 23768 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 869) took 2704 ms. Trace:
I20240502 08:16:51.874477 23768 rpcz_store.cc:276] 0502 08:16:49.121801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.698027 (+2576226us) service_pool.cc:225] Handling call
0502 08:16:51.698392 (+ 365us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.698416 (+ 24us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.701110 (+ 2694us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.826568 (+125458us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.826669 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.874826 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 2bb70ee9169046dd98e3f35ffcccdf3a (DEFAULT_TABLE table=default.Table_973ce52632594e5f9c980d6d7ae7371b [id=11d129fb7ef1439fad8010fee542e716]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.832392 23754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 866) took 2714 ms. Trace:
I20240502 08:16:51.875799 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c31d9edf126f4ae6a1c06bc92f31a5b9" } new_table_name: "default.Table_3964effad9bb411dab9ba4e158fe33fd" modify_external_catalogs: true
I20240502 08:16:51.875890 23754 rpcz_store.cc:276] 0502 08:16:49.118268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.688654 (+2570386us) service_pool.cc:225] Handling call
0502 08:16:51.689146 (+ 492us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.689175 (+ 29us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.692189 (+ 3014us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.832226 (+140037us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.832371 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.876295 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 858b23b8e12b467e86d2c914e63e8fa2 (DEFAULT_TABLE table=default.Table_6bab5c3f6452414eba871948cffb5714 [id=43b8fb37d8704742affd5ed4d1d4a287]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.834877 23765 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 870) took 2712 ms. Trace:
I20240502 08:16:51.878468 23765 rpcz_store.cc:276] 0502 08:16:49.122818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.701938 (+2579120us) service_pool.cc:225] Handling call
0502 08:16:51.741986 (+ 40048us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.742131 (+ 145us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.749297 (+ 7166us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.834750 (+ 85453us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.834855 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.878698 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet a0224317e65747e1add533d33472f7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:51.878871 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet ee3d91d55ffd49ceb20d2c33408f5a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:51.878980 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet e549cad1b3be4b28abb0bf5af720eb52 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:37 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:51.879094 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a1e208733844829b13222e00a432439 with delete_type TABLET_DATA_DELETED (replaced by tablet 8910bc67c6234a538c8160e7c020653c) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:51.879341 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 26c8325099af4d72ad00afb2316df885 (DEFAULT_TABLE table=default.Table_64b18a844e8d44ba8bca05606e5b34e8 [id=dbd1ab9190f94aebbdc794fd9f1fc074]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.840965 23756 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 867) took 2721 ms. Trace:
I20240502 08:16:51.880458 23756 rpcz_store.cc:276] 0502 08:16:49.119320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.690582 (+2571262us) service_pool.cc:225] Handling call
0502 08:16:51.690917 (+ 335us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.690939 (+ 22us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.797998 (+107059us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.840839 (+ 42841us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.840949 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.881172 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc2270a5461448ca0245bd875f4245a (DEFAULT_TABLE table=default.Table_e29dfece3b88466ba6e6f6db0c1a31fd [id=7c9478b76f0a46c1869bd31c27a8eddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.881271 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5a6711c7a8e74b05af6d0c8fb3cfa33f" } new_table_name: "default.Table_0f529cf8c0b04c1890a96668d3e38ecd" modify_external_catalogs: true
I20240502 08:16:51.843859 23758 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 880) took 2455 ms. Trace:
I20240502 08:16:51.881623 23758 rpcz_store.cc:276] 0502 08:16:49.388503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.740949 (+2352446us) service_pool.cc:225] Handling call
0502 08:16:51.741341 (+ 392us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.741368 (+ 27us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.767195 (+ 25827us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.843706 (+ 76511us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.843841 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.882392 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2aa20459484e4b98e0ae372418020f (DEFAULT_TABLE table=default.Table_ff75818c766d43569443cacbfcb972c1 [id=22d38df685d2423da2228076975133a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.844453 23757 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 897) took 1115 ms. Trace:
I20240502 08:16:51.882826 23757 rpcz_store.cc:276] 0502 08:16:50.729287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.818931 (+1089644us) service_pool.cc:225] Handling call
0502 08:16:51.819273 (+ 342us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.819297 (+ 24us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.837325 (+ 18028us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.844323 (+ 6998us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.844436 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.883476 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet fe5e08ad6af2442bb0a8fcc80a99b532 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:50 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:51.883651 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet 29f6c3cea4b14364b6d073a92171119d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:50 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:51.884397 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet 303a0a3748ee422faab856e1c4bff66d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:49 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:51.884550 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet f685cbe705b149e69f9536e270a74a42 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:50 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:51.846980 23764 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 878) took 2458 ms. Trace:
I20240502 08:16:51.884796 23764 rpcz_store.cc:276] 0502 08:16:49.388343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.736989 (+2348646us) service_pool.cc:225] Handling call
0502 08:16:51.737379 (+ 390us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.737407 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.756130 (+ 18723us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.846137 (+ 90007us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.846849 (+ 712us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.848333 23750 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 871) took 2724 ms. Trace:
I20240502 08:16:51.885231 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_682d60898e9545ceb4723b24bb852374"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:51.885462 23750 rpcz_store.cc:276] 0502 08:16:49.123971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.702027 (+2578056us) service_pool.cc:225] Handling call
0502 08:16:51.702310 (+ 283us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.702335 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.704941 (+ 2606us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.848209 (+143268us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.848317 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:16:51.885936 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_682d60898e9545ceb4723b24bb852374 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:51.870817 23752 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 862) took 2757 ms. Trace:
I20240502 08:16:51.887274 23752 rpcz_store.cc:276] 0502 08:16:49.113663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.650013 (+2536350us) service_pool.cc:225] Handling call
0502 08:16:51.650567 (+ 554us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.650603 (+ 36us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.808812 (+158209us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.870620 (+ 61808us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.870792 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11989}
I20240502 08:16:51.887512 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6118b7d5075e4fbb9eaab8b4a9fd1627" } new_table_name: "default.Table_d89be39e4ca74509a6b119c0b42413ca" modify_external_catalogs: true
I20240502 08:16:51.871704 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a2bcc6937c4a95b71103dbf8738702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.847250 24174 tablet_bootstrap.cc:492] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:51.889348 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc2270a5461448ca0245bd875f4245a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.891141 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bb70ee9169046dd98e3f35ffcccdf3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.891270 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4566bc3dbdde463e9392f5ea4fa97741. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.891556 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68448a2033e450794440fa84de6a19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.891896 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9b75d6be924d0291f94fc4a439dea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.892135 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b811b81ddf2b4dc9b5e5f80331bff294" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:51.892249 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858b23b8e12b467e86d2c914e63e8fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.892639 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c8325099af4d72ad00afb2316df885. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.838706 23760 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 885) took 2173 ms. Trace:
I20240502 08:16:51.893021 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5398d425143843de8fe5c9594fcf8503" } new_table_name: "default.Table_3a66619663c642f5be35fd8249184e1a" modify_external_catalogs: true
I20240502 08:16:51.893049 23760 rpcz_store.cc:276] 0502 08:16:49.665752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.742404 (+2076652us) service_pool.cc:225] Handling call
0502 08:16:51.742701 (+ 297us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.742721 (+ 20us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.752740 (+ 10019us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.838585 (+ 85845us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.838691 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.836448 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet c48cea94856f43b6861c3ba360414888 with delete_type TABLET_DATA_DELETED (replaced by tablet e34873a3f08043098f2740641b3dc41f) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:51.850553 23762 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 864) took 2734 ms. Trace:
I20240502 08:16:51.893646 23762 rpcz_store.cc:276] 0502 08:16:49.116084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.665948 (+2549864us) service_pool.cc:225] Handling call
0502 08:16:51.666496 (+ 548us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.666525 (+ 29us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.669489 (+ 2964us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.850414 (+180925us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.850532 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.853207 23759 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 863) took 2738 ms. Trace:
I20240502 08:16:51.893959 23759 rpcz_store.cc:276] 0502 08:16:49.114589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.656791 (+2542202us) service_pool.cc:225] Handling call
0502 08:16:51.657213 (+ 422us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.657238 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.660642 (+ 3404us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.853066 (+192424us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.853189 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.858762 23761 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 879) took 2470 ms. Trace:
I20240502 08:16:51.894230 23761 rpcz_store.cc:276] 0502 08:16:49.388434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.737760 (+2349326us) service_pool.cc:225] Handling call
0502 08:16:51.738093 (+ 333us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.738115 (+ 22us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.750986 (+ 12871us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.858568 (+107582us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.858723 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.860864 23766 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 865) took 2743 ms. Trace:
I20240502 08:16:51.894508 23766 rpcz_store.cc:276] 0502 08:16:49.117165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.672233 (+2555068us) service_pool.cc:225] Handling call
0502 08:16:51.672675 (+ 442us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.672709 (+ 34us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.675653 (+ 2944us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.860745 (+185092us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.860846 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:16:51.869457 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 66001.2us
I20240502 08:16:51.896136 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b2aa20459484e4b98e0ae372418020f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.856246 23763 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 868) took 2735 ms. Trace:
I20240502 08:16:51.896646 23763 rpcz_store.cc:276] 0502 08:16:49.120705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.693815 (+2573110us) service_pool.cc:225] Handling call
0502 08:16:51.694167 (+ 352us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.694190 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.715913 (+ 21723us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.856072 (+140159us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.856224 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.897466 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9402cf8abc4b4d893d42f9a3a99016" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:51.897765 23917 raft_consensus.cc:2454] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:51.898300 23996 leader_election.cc:302] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:51.898651 25154 raft_consensus.cc:2781] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:16:51.898736 25154 raft_consensus.cc:482] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:51.898795 25154 raft_consensus.cc:3037] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:16:51.899813 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4a31e5aa834adeb9f59267247c6980" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:51.900069 23786 raft_consensus.cc:2379] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:16:51.901190 25301 consensus_queue.cc:1035] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:51.901584 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d2bb4dba23ac4b4eb53b3f2d75d0442e consensus state has no leader
W20240502 08:16:51.902599 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed428ef1be8a49c19e4b6b48025b9cac consensus state has no leader
I20240502 08:16:51.903277 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd8338500f54604abcd6d42702c2eca" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:51.903504 23786 raft_consensus.cc:2379] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in this term.
I20240502 08:16:51.904067 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fcb895c8f449deb0190de75af18c3c" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:51.904017 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_562886123f0548bfab83210c1c45b552"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:51.905297 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654ce33d1e3d472eb6ae6a726eb803bb" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:51.905465 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_562886123f0548bfab83210c1c45b552 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:51.905511 23786 raft_consensus.cc:2379] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in this term.
I20240502 08:16:51.906144 23608 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36668 (request call id 1041) took 1532 ms. Trace:
I20240502 08:16:51.906625 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c377e037894728ac084b6500ad20de" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:51.906839 23786 raft_consensus.cc:2379] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in this term.
I20240502 08:16:51.908280 24029 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 818) took 2806 ms. Trace:
I20240502 08:16:51.909260 23723 leader_election.cc:302] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:51.909475 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e03c13b1cb584f5994dff77692b0f442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.909960 25301 raft_consensus.cc:2781] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:51.910089 24174 ts_tablet_manager.cc:1367] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.657s user 0.007s sys 0.007s
I20240502 08:16:51.910375 25301 raft_consensus.cc:686] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:51.910905 25301 consensus_queue.cc:237] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.911566 24174 raft_consensus.cc:348] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.912079 24174 raft_consensus.cc:374] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:51.912233 24174 raft_consensus.cc:729] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:51.775152 23858 service_pool.cc:138] CreateTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:33620 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 808)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 809)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 810)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 811)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 812)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 813)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 814)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 815)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 816)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 817)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 818)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 819)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 820)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 821)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 822)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 823)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 824)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 825)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 826)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 827)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 828)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 829)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 830)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 831)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 832)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 833)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 834)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 835)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 836)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 837)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 838)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 839)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 840)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 841)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 842)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 843)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 844)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 845)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 846)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 847)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 848)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 849)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 850)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 851)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 853)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 854)
Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 855)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 856)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 857)
Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 858)
I20240502 08:16:51.915719 25078 raft_consensus.cc:482] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a297a6c7c2504ca68e87b24f6691ffb2)
I20240502 08:16:51.916045 25076 raft_consensus.cc:482] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a297a6c7c2504ca68e87b24f6691ffb2)
I20240502 08:16:51.917526 24025 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 827) took 2806 ms. Trace:
I20240502 08:16:51.917608 24027 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 821) took 2813 ms. Trace:
I20240502 08:16:51.916829 23916 raft_consensus.cc:3037] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:51.918716 25108 raft_consensus.cc:482] T 36f2d7b6f4e442ae95b2d1002d571b6a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a297a6c7c2504ca68e87b24f6691ffb2)
I20240502 08:16:51.917716 24025 rpcz_store.cc:276] 0502 08:16:49.111484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.677815 (+2566331us) service_pool.cc:225] Handling call
0502 08:16:51.678173 (+ 358us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.678204 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.680975 (+ 2771us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.866731 (+185756us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.917492 (+ 50761us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.918891 25108 raft_consensus.cc:504] T 36f2d7b6f4e442ae95b2d1002d571b6a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:51.918152 24021 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 812) took 2823 ms. Trace:
I20240502 08:16:51.919703 25108 leader_election.cc:288] T 36f2d7b6f4e442ae95b2d1002d571b6a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:51.919778 24021 rpcz_store.cc:276] 0502 08:16:49.094690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.470570 (+375880us) service_pool.cc:225] Handling call
0502 08:16:50.747984 (+1277414us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.748013 (+ 29us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.678615 (+930602us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.909073 (+230458us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.918138 (+ 9065us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.919898 25108 raft_consensus.cc:482] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a297a6c7c2504ca68e87b24f6691ffb2)
I20240502 08:16:51.920042 25108 raft_consensus.cc:504] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.920228 24021 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad53633ce57478194437f95135d81de (DEFAULT_TABLE table=default.Table_9522d368c4c84f25bce3eb822a9bcad8 [id=e07e5990e2244361bcda7aa9108994b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.918311 24035 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 832) took 2801 ms. Trace:
I20240502 08:16:51.920627 25108 leader_election.cc:288] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:51.920645 24035 rpcz_store.cc:276] 0502 08:16:49.116642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.689970 (+2573328us) service_pool.cc:225] Handling call
0502 08:16:51.690272 (+ 302us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.690290 (+ 18us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.755766 (+ 65476us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.863850 (+108084us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.918296 (+ 54446us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.919620 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3570787d8242338d1ce01fb0299eb2 (DEFAULT_TABLE table=default.Table_d03c41e82d684920af0c093caa7cd1e6 [id=1a0ddcb26d794ee99068fa11c3b345e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.919651 24038 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 836) took 2798 ms. Trace:
I20240502 08:16:51.923400 24038 rpcz_store.cc:276] 0502 08:16:49.121596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.752683 (+2631087us) service_pool.cc:225] Handling call
0502 08:16:51.753551 (+ 868us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.753616 (+ 65us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.865209 (+111593us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.911656 (+ 46447us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.919634 (+ 7978us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.924002 25227 raft_consensus.cc:482] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:51.927772 24035 tablet_service.cc:1467] Processing CreateTablet for tablet d6e589e879f245378cff543fe430be5d (DEFAULT_TABLE table=default.Table_5fb8fa42fdb945729f5026cab15051b0 [id=323ecf6fa9a1474f869f2452d7b63da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.921268 23918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fcb895c8f449deb0190de75af18c3c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:51.908669 24029 rpcz_store.cc:276] 0502 08:16:49.101213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.496548 (+395335us) service_pool.cc:225] Handling call
0502 08:16:50.748350 (+1251802us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.748374 (+ 24us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.669433 (+921059us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.861976 (+192543us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.907940 (+ 45964us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:16:51.930037 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 462f77a57ebf45948cbd3dc20c271650 consensus state has no leader
I20240502 08:16:51.930974 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 37b499184f31454c96f66cfdf9143d14 (DEFAULT_TABLE table=default.Table_7a0267df98fe45a583a32684e35f9416 [id=953199c936cc4d859bc7ff49dc0c2ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.906819 23608 rpcz_store.cc:276] 0502 08:16:50.373800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.788839 (+1415039us) service_pool.cc:225] Handling call
0502 08:16:51.831476 (+ 42637us) catalog_manager.cc:2072] Acquired catalog manager lock
0502 08:16:51.832103 (+ 627us) catalog_manager.cc:2118] Created new table and tablet info
0502 08:16:51.905857 (+ 73754us) catalog_manager.cc:2178] Wrote table and tablets to system table
0502 08:16:51.905992 (+ 135us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0502 08:16:51.906082 (+ 90us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:51.842460 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:51.842595 (+ 135us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:51.842603 (+ 8us) write_op.cc:427] Acquired schema lock
0502 08:16:51.842606 (+ 3us) tablet.cc:660] Decoding operations
0502 08:16:51.842735 (+ 129us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:51.842753 (+ 18us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:51.842762 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0502 08:16:51.842968 (+ 206us) tablet.cc:699] Row locks acquired
0502 08:16:51.842974 (+ 6us) write_op.cc:260] PREPARE: finished
0502 08:16:51.843035 (+ 61us) write_op.cc:270] Start()
0502 08:16:51.843076 (+ 41us) write_op.cc:276] Timestamp: P: 1714637811843009 usec, L: 0
0502 08:16:51.843080 (+ 4us) op_driver.cc:331] REPLICATION: starting
0502 08:16:51.873359 (+ 30279us) log.cc:842] Serialized 433 byte log entry
0502 08:16:51.874073 (+ 714us) spinlock_profiling.cc:243] Waited 30 ms on lock 0x616000021d54. stack: 00007f7e39fca756 00007f7e39fca510 00007f7e38ffd48d 0000000000955804 00000000009557cd 000000000095519a 00007f7e4580f162 00007f7e46b77b02 00007f7e46b76629 00007f7e46b7e53b 00007f7e46b7e3b8 0000000000a60bcc 00007f7e3a010b74 00007f7e3a01419b 00007f7e3a014018 0000000000a60bcc
0502 08:16:51.878551 (+ 4478us) op_driver.cc:444] REPLICATION: finished
0502 08:16:51.878734 (+ 183us) write_op.cc:301] APPLY: starting
0502 08:16:51.878751 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:51.878810 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:51.878812 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:51.878954 (+ 142us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:51.878993 (+ 39us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0502 08:16:51.879000 (+ 7us) write_op.cc:312] APPLY: finished
0502 08:16:51.879083 (+ 83us) log.cc:842] Serialized 29 byte log entry
0502 08:16:51.879152 (+ 69us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.879171 (+ 19us) write_op.cc:446] Released schema lock
0502 08:16:51.879180 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:51.879239 (+ 59us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:51.879243 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":132,"apply.run_cpu_time_us":562,"apply.run_wall_time_us":562,"num_ops":2,"prepare.queue_time_us":9730,"prepare.run_cpu_time_us":1781,"prepare.run_wall_time_us":31719,"raft.queue_time_us":894,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":360,"replication_time_us":35447,"spinlock_wait_cycles":69096192}]]}
I20240502 08:16:51.933216 23922 raft_consensus.cc:2454] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:51.934064 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a4104f4cb64889ad66810788ad6b29" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:51.924140 25227 raft_consensus.cc:504] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:51.940999 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 6a7ab5fc54834f5783f71e615bfc283a (DEFAULT_TABLE table=default.Table_e97c69e9998149d98b2954d150b8d220 [id=025b29b37a4f40539680de19b892cd15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.934828 25227 leader_election.cc:288] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:51.974188 24032 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 809) took 2826 ms. Trace:
I20240502 08:16:51.974325 24032 rpcz_store.cc:276] 0502 08:16:49.091864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.460743 (+368879us) service_pool.cc:225] Handling call
0502 08:16:50.756779 (+1296036us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.756797 (+ 18us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.772167 (+1015370us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.901173 (+129006us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.918012 (+ 16839us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.974905 24032 tablet_service.cc:1467] Processing CreateTablet for tablet ce8dc203c36149cc91cc56176e929da6 (DEFAULT_TABLE table=default.Table_effefe18d3cd42f7855fed709d87de8c [id=596b181baa6140c0b44a9f900e9cbb5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.975414 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce8dc203c36149cc91cc56176e929da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.936830 25078 raft_consensus.cc:504] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:51.977463 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0f88db2e7c844740bcb3df0fb75508b8" } new_table_name: "default.Table_f1ae3f19a7614f28a1916c0e40c37674" modify_external_catalogs: true
I20240502 08:16:51.978495 24175 tablet_bootstrap.cc:654] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:51.979601 25076 raft_consensus.cc:504] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:51.981916 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fe597986ce945aba8f2bb19d46bb7ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.917742 24027 rpcz_store.cc:276] 0502 08:16:49.104470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.499329 (+394859us) service_pool.cc:225] Handling call
0502 08:16:50.748471 (+1249142us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.748487 (+ 16us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.866085 (+1117598us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.870276 (+ 4191us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.917580 (+ 47304us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.983685 23918 raft_consensus.cc:2379] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a297a6c7c2504ca68e87b24f6691ffb2 in current term 1: Already voted for candidate 5b89a1f0485a44b7ac1710e2369a809b in this term.
I20240502 08:16:51.983951 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4a31e5aa834adeb9f59267247c6980" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:51.984200 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d86d534538de4613bd148228a78e5e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.987623 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ecde716a0c643cdb0698a37cde1c906. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.988183 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e589e879f245378cff543fe430be5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.988359 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a44933d855bd48aa8865c11eaaa5cbe4" } modify_external_catalogs: true
I20240502 08:16:51.988577 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a44933d855bd48aa8865c11eaaa5cbe4" } modify_external_catalogs: true
W20240502 08:16:51.989210 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 79999us
I20240502 08:16:51.989312 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad53633ce57478194437f95135d81de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.992177 24181 ts_tablet_manager.cc:1888] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:51.992395 24181 log.cc:1196] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/c17e1be6ef294769a01981f4d361be21
I20240502 08:16:51.993038 24181 ts_tablet_manager.cc:1909] T c17e1be6ef294769a01981f4d361be21 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:51.994153 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e549cad1b3be4b28abb0bf5af720eb52" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:51.994575 24022 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 825) took 2885 ms. Trace:
I20240502 08:16:51.994707 24022 rpcz_store.cc:276] 0502 08:16:49.109289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.661877 (+2552588us) service_pool.cc:225] Handling call
0502 08:16:51.723123 (+ 61246us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.723197 (+ 74us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.991809 (+268612us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.994255 (+ 2446us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:51.994551 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:51.998433 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7ab5fc54834f5783f71e615bfc283a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.999296 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cce64bbdca4946bfb1cdbd52d1d2d59c" } modify_external_catalogs: true
I20240502 08:16:51.999558 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cce64bbdca4946bfb1cdbd52d1d2d59c" } modify_external_catalogs: true
I20240502 08:16:52.000593 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af5d13f956354acca041913011923840. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.001297 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3570787d8242338d1ce01fb0299eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.002388 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 726) took 4905 ms. Trace:
I20240502 08:16:52.002477 24181 rpcz_store.cc:276] 0502 08:16:47.097389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.097551 (+ 162us) service_pool.cc:225] Handling call
0502 08:16:49.893553 (+2796002us) ts_tablet_manager.cc:1120] Deleting tablet c17e1be6ef294769a01981f4d361be21
0502 08:16:49.893591 (+ 38us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:50.767000 (+873409us) spinlock_profiling.cc:243] Waited 9.45 ms on lock 0x6170000b04d0. stack: 00007f9184bfa756 00007f9184bfa510 00007f9183c2d48d 0000000000955804 00000000009557cd 000000000095519a 00007f919036de21 00007f91903fb49c 00007f9190468ae3 00007f9191742853 00007f91941d557c 00007f91942078d4 00007f91941e8bc0 00007f91941e894d 0000000000a60bcc 00007f9184c40b74
0502 08:16:51.066182 (+299182us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.992018 (+925836us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.002367 (+ 10349us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18781,"spinlock_wait_cycles":21742464,"tablet-delete.queue_time_us":2795849}
I20240502 08:16:52.002786 24027 tablet_service.cc:1467] Processing CreateTablet for tablet bb9b75d6be924d0291f94fc4a439dea6 (DEFAULT_TABLE table=default.Table_41d744e17dee458382533bb204fd1197 [id=e99a8a3a2c9343369d31c9852c31de9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.002892 24181 tablet_replica.cc:330] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:52.003895 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4e432e7524d342fa8de7aedc283e9a2f" } modify_external_catalogs: true
I20240502 08:16:52.004118 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4e432e7524d342fa8de7aedc283e9a2f" } modify_external_catalogs: true
I20240502 08:16:52.004674 23919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c072eb45e1b41eeb5aafa6d5a786944" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:52.005545 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d89be39e4ca74509a6b119c0b42413ca" } modify_external_catalogs: true
I20240502 08:16:52.005702 25078 leader_election.cc:288] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:52.005753 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d89be39e4ca74509a6b119c0b42413ca" } modify_external_catalogs: true
I20240502 08:16:52.005476 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ccd43e78574f3288b199fa14bcc0e8" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:52.006606 25076 leader_election.cc:288] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:52.010268 24034 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 838) took 2886 ms. Trace:
I20240502 08:16:52.010362 24034 rpcz_store.cc:276] 0502 08:16:49.123475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.798060 (+2674585us) service_pool.cc:225] Handling call
0502 08:16:51.917037 (+118977us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.917084 (+ 47us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.007525 (+ 90441us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.010087 (+ 2562us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.010241 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.010574 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1e208733844829b13222e00a432439" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:52.010818 23914 raft_consensus.cc:3037] T 5a1e208733844829b13222e00a432439 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.013608 24040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 831) took 2898 ms. Trace:
I20240502 08:16:52.013695 24040 rpcz_store.cc:276] 0502 08:16:49.115555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.689333 (+2573778us) service_pool.cc:225] Handling call
0502 08:16:51.689654 (+ 321us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.689675 (+ 21us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.981709 (+292034us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:51.998998 (+ 17289us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.013585 (+ 14587us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.014513 23897 tablet_service.cc:1467] Processing CreateTablet for tablet e20e126b1c9546e4a194bf9842f444a5 (DEFAULT_TABLE table=default.Table_769de68591d64bdb9b80f7bb337da552 [id=f8b12b3bb4904c18be60b567cb05b739]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.015202 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c377e037894728ac084b6500ad20de" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:52.015502 23920 raft_consensus.cc:2379] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a297a6c7c2504ca68e87b24f6691ffb2 in current term 1: Already voted for candidate 5b89a1f0485a44b7ac1710e2369a809b in this term.
I20240502 08:16:52.015726 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d609a55e47ad420098d279835e9aa090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.016008 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12bd0a3bf7d24d939288c3eca83e1540. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.016185 23919 raft_consensus.cc:2379] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a297a6c7c2504ca68e87b24f6691ffb2 in current term 1: Already voted for candidate 5b89a1f0485a44b7ac1710e2369a809b in this term.
I20240502 08:16:52.016203 23922 raft_consensus.cc:3037] T f5ccd43e78574f3288b199fa14bcc0e8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.016474 23916 raft_consensus.cc:3037] T e549cad1b3be4b28abb0bf5af720eb52 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.019712 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a06fc827d204d02ab81d5eeb44f7421. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:52.021384 24371 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.225.125:36189 timed out after 0.746s (SENT)
I20240502 08:16:52.022897 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dae6a96776c477fb0ada8226d68ea25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.023075 23918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba1baa2279045b3adc58d187a12cbf3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:52.023182 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b5566be23d3744348216cab2e9a0c6bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:52.023849 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5566be23d3744348216cab2e9a0c6bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:52.025658 24022 tablet_service.cc:1467] Processing CreateTablet for tablet d68448a2033e450794440fa84de6a19a (DEFAULT_TABLE table=default.Table_6a6b2f29d9aa4a679e4c41863e3096bd [id=38c7d40fae5b4b7eb187d2c8e43f04d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.026885 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9b75d6be924d0291f94fc4a439dea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.027174 24037 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 829) took 2913 ms. Trace:
I20240502 08:16:52.027274 24037 rpcz_store.cc:276] 0502 08:16:49.113211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.688009 (+2574798us) service_pool.cc:225] Handling call
0502 08:16:51.688381 (+ 372us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.688402 (+ 21us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.980752 (+292350us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.012645 (+ 31893us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.027140 (+ 14495us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.027817 24037 tablet_service.cc:1467] Processing CreateTablet for tablet b7339f1207924812b4ef2ac7ccffc894 (DEFAULT_TABLE table=default.Table_375988d4437044bc890604e6d5ab8af2 [id=6c14f79d3913425e9c0a0d0b524c1c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.028235 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7339f1207924812b4ef2ac7ccffc894. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.030339 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 858b23b8e12b467e86d2c914e63e8fa2 (DEFAULT_TABLE table=default.Table_6bab5c3f6452414eba871948cffb5714 [id=43b8fb37d8704742affd5ed4d1d4a287]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.030061 23918 raft_consensus.cc:2454] T 1ba1baa2279045b3adc58d187a12cbf3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:52.031047 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87a3df4fbb224dd0b4ec6141584d374b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.031989 23903 tablet_service.cc:1467] Processing CreateTablet for tablet e546c1bb78a848ad8125e8e8b08434db (DEFAULT_TABLE table=default.Table_843f461368e94ea9b6b9244506423221 [id=687f28ef29324002821616ba0044d1e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.032639 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b499184f31454c96f66cfdf9143d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.025303 24181 raft_consensus.cc:2229] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:52.033807 24181 raft_consensus.cc:2258] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:52.036767 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1da01e2c2f054ccfb53383e70e16da84" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:52.041213 24181 ts_tablet_manager.cc:1875] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:52.042973 23918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654ce33d1e3d472eb6ae6a726eb803bb" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:52.043033 23919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751808dfc1964b539f6f836d786156aa" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:52.044610 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 336f0eb5085546dba763e0d5c5dd5324 (DEFAULT_TABLE table=default.Table_cda7da2a134f43a8a3c79a6609c3e52e [id=07050bd496e84a97a86fdc4f0b7811ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.044945 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7b37f776df47da9aa6e7fba82a8999" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:52.045104 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 2bb70ee9169046dd98e3f35ffcccdf3a (DEFAULT_TABLE table=default.Table_973ce52632594e5f9c980d6d7ae7371b [id=11d129fb7ef1439fad8010fee542e716]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.045364 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c50800505247cf967607bbfcca6302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.045629 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68448a2033e450794440fa84de6a19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.045718 23914 raft_consensus.cc:2454] T 5a1e208733844829b13222e00a432439 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:52.045991 23904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 783) took 1217 ms. Trace:
I20240502 08:16:52.046109 23904 rpcz_store.cc:276] 0502 08:16:50.821058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:50.821096 (+ 38us) service_pool.cc:225] Handling call
0502 08:16:50.987815 (+166719us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.987848 (+ 33us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.842496 (+854648us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.038555 (+196059us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.038732 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.046378 23918 raft_consensus.cc:2379] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a297a6c7c2504ca68e87b24f6691ffb2 in current term 1: Already voted for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in this term.
I20240502 08:16:52.047984 25012 ts_tablet_manager.cc:1888] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:51.953428 24174 consensus_queue.cc:260] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.053476 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bb70ee9169046dd98e3f35ffcccdf3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.054075 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858b23b8e12b467e86d2c914e63e8fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.054296 23916 raft_consensus.cc:2454] T e549cad1b3be4b28abb0bf5af720eb52 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:52.001449 23754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 904) took 1012 ms. Trace:
I20240502 08:16:52.058075 23754 rpcz_store.cc:276] 0502 08:16:50.989191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.876093 (+886902us) service_pool.cc:225] Handling call
0502 08:16:51.876463 (+ 370us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.876499 (+ 36us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.964706 (+ 88207us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.001060 (+ 36354us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.001393 (+ 333us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.039250 24180 ts_tablet_manager.cc:1888] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:52.059094 24180 log.cc:1196] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/c17e1be6ef294769a01981f4d361be21
I20240502 08:16:52.059393 24180 ts_tablet_manager.cc:1909] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:52.059706 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 782) took 3898 ms. Trace:
I20240502 08:16:52.059772 24180 rpcz_store.cc:276] 0502 08:16:48.161258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.162958 (+ 1700us) service_pool.cc:225] Handling call
0502 08:16:51.815464 (+3652506us) ts_tablet_manager.cc:1120] Deleting tablet c17e1be6ef294769a01981f4d361be21
0502 08:16:51.815473 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:51.939505 (+124032us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.039059 (+ 99554us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.059684 (+ 20625us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3652283}
I20240502 08:16:52.060251 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet c17e1be6ef294769a01981f4d361be21 (table default.Table_498b372c15a04b70b87bd67b543501cf [id=38e955d6bd644d24a7c45172ab60282b]) successfully deleted
I20240502 08:16:52.060380 25012 log.cc:1196] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/bcfacc302ff04178bfffc48111cd7a3f
I20240502 08:16:52.060647 24180 tablet_replica.cc:330] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:52.060828 24180 raft_consensus.cc:2229] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:52.060995 24180 pending_rounds.cc:62] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:16:52.061034 24180 pending_rounds.cc:68] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156462764044288 op_type: NO_OP noop_request { }
I20240502 08:16:52.061134 24180 raft_consensus.cc:2866] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:52.061199 24180 raft_consensus.cc:2258] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:52.061798 24180 ts_tablet_manager.cc:1875] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:52.063426 24023 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 815) took 2965 ms. Trace:
I20240502 08:16:52.063499 24023 rpcz_store.cc:276] 0502 08:16:49.097826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.484201 (+386375us) service_pool.cc:225] Handling call
0502 08:16:50.748129 (+1263928us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.748150 (+ 21us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.996097 (+1247947us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.063277 (+ 67180us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.063406 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:16:52.064781 23993 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 373566us
I20240502 08:16:52.066581 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3078cc1f6247609f8bd017ed076b45" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:52.066938 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f02efa659e457ab57bc32ed3f66bb9" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:52.067279 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b811b81ddf2b4dc9b5e5f80331bff294" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:52.067332 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9402cf8abc4b4d893d42f9a3a99016" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:52.067771 25219 raft_consensus.cc:482] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a297a6c7c2504ca68e87b24f6691ffb2)
I20240502 08:16:52.049949 25158 raft_consensus.cc:482] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:52.067917 25158 raft_consensus.cc:504] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.068657 25158 leader_election.cc:288] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:52.068993 24174 ts_tablet_manager.cc:1398] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.159s user 0.004s sys 0.001s
I20240502 08:16:52.069281 24174 tablet_bootstrap.cc:492] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:52.070642 24174 tablet_bootstrap.cc:654] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.070947 23606 catalog_manager.cc:5484] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:52.071496 23606 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 77c49da1e0f0435ba241a6a66526e790 (table default.Table_8a9cdf1f53b0494694035cd549988bf0 [id=0de18f9c7e194e698b4f37603bfd4780]). Expected version 2 got 0
I20240502 08:16:52.071748 23993 leader_election.cc:302] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:52.072499 25227 raft_consensus.cc:2781] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:52.072615 25227 raft_consensus.cc:686] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:52.073361 23767 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 886) took 2407 ms. Trace:
I20240502 08:16:52.073446 23767 rpcz_store.cc:276] 0502 08:16:49.665866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.815384 (+2149518us) service_pool.cc:225] Handling call
0502 08:16:51.815819 (+ 435us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.815842 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.975552 (+159710us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.073153 (+ 97601us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.073328 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.048815 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 296b1444c5bb4d319ab9c69373fff84e (DEFAULT_TABLE table=default.Table_7227bf354dec4869bd921dd397b17de3 [id=40dbc13e0508467380cc488dfd713ac6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.074463 25012 ts_tablet_manager.cc:1909] T bcfacc302ff04178bfffc48111cd7a3f P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:52.078243 25222 consensus_queue.cc:1035] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:52.080644 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc133adb9a44cc7b56a3746d1eb76af (DEFAULT_TABLE table=default.Table_d20a3b8ecb6449a5a4cc9837354619b7 [id=1c25911cce0845ffb760a96926714b0f]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:52.080233 23993 leader_election.cc:341] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:52.081347 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc133adb9a44cc7b56a3746d1eb76af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.081280 24028 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 841) took 2928 ms. Trace:
W20240502 08:16:52.081832 23993 leader_election.cc:341] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:52.083427 23753 tablet_service.cc:1467] Processing CreateTablet for tablet a4fe3a2f9eac469d9a7da7b9a2fc24a8 (DEFAULT_TABLE table=default.Table_dd5271576d9746048fa96f022073b8fd [id=4640f69b48d44ee3baa1b6d4ab6fd6bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.083678 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 521581fe87e547ce816b599578bddeea (DEFAULT_TABLE table=default.Table_d67c17e0dfd94a088bd9991625f3c31a [id=bd341da28438429895d79410c12dce3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.083839 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4fe3a2f9eac469d9a7da7b9a2fc24a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.081996 24028 rpcz_store.cc:276] 0502 08:16:49.152662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.848734 (+2696072us) service_pool.cc:225] Handling call
0502 08:16:51.917180 (+ 68446us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.917201 (+ 21us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.058756 (+141555us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.079887 (+ 21131us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.081257 (+ 1370us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.084559 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 521581fe87e547ce816b599578bddeea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.085321 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 517354df966c4338a01a425251cccdd2 (DEFAULT_TABLE table=default.Table_1fa44e7d541041d3a86bed4be93b5a03 [id=4e5acee37ab44e979a690a2df5a78d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.085706 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517354df966c4338a01a425251cccdd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.086511 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba1baa2279045b3adc58d187a12cbf3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:52.087399 24026 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 835) took 2967 ms. Trace:
I20240502 08:16:52.087674 24036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 840) took 2961 ms. Trace:
I20240502 08:16:52.088979 24035 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 844) took 2936 ms. Trace:
I20240502 08:16:52.082795 23993 leader_election.cc:302] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:52.094823 23768 tablet_service.cc:1467] Processing CreateTablet for tablet c991463cf8ad4d3b8c0727e45c2317b4 (DEFAULT_TABLE table=default.Table_1014ad7e82a245ea801b1866b929b1c5 [id=39339ca66938431c8de38927672a9f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.083869 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 8075b03643a14a01ab75921fc3d3dace (DEFAULT_TABLE table=default.Table_fabad8a6e72a4724893dfa4c0877fdb2 [id=ea894a4bd467422eb135ad75e7032193]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.084228 25266 raft_consensus.cc:686] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:52.095495 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8075b03643a14a01ab75921fc3d3dace. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.099123 23918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128c88af54d24ffd895e9de19cdb4c8a" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:52.100412 23767 tablet_service.cc:1467] Processing CreateTablet for tablet f5496fa258de4988b9502136f0baf779 (DEFAULT_TABLE table=default.Table_33c693cfb89641a2921c7d39872b5bbc [id=68e36934ceb0416a86dc24c4fc50804b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.103225 23889 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 797) took 1014 ms. Trace:
I20240502 08:16:52.104504 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5496fa258de4988b9502136f0baf779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.105340 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 714) took 4395 ms. Trace:
I20240502 08:16:52.105525 25012 rpcz_store.cc:276] 0502 08:16:47.706730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.706777 (+ 47us) service_pool.cc:225] Handling call
0502 08:16:51.725822 (+4019045us) ts_tablet_manager.cc:1120] Deleting tablet bcfacc302ff04178bfffc48111cd7a3f
0502 08:16:51.725833 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:51.991715 (+265882us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.047640 (+ 55925us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.102085 (+ 54445us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4018915}
I20240502 08:16:52.106242 25012 tablet_replica.cc:330] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:52.106570 25012 raft_consensus.cc:2229] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:52.106911 25012 raft_consensus.cc:2258] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:52.107933 25012 ts_tablet_manager.cc:1875] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:52.108165 23889 rpcz_store.cc:276] 0502 08:16:51.085919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.088648 (+ 2729us) service_pool.cc:225] Handling call
0502 08:16:51.661850 (+573202us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.661875 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.665166 (+ 3291us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.044107 (+378941us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.100538 (+ 56431us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.108959 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 1626382a260044eb9100b8048332abb0 (DEFAULT_TABLE table=default.Table_c9c50670737d40c58a305ff256334951 [id=034e49e9b5cc435cad83782e0f1d9ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.109977 23893 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 802) took 1015 ms. Trace:
I20240502 08:16:52.110371 23893 rpcz_store.cc:276] 0502 08:16:51.086388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.261643 (+175255us) service_pool.cc:225] Handling call
0502 08:16:51.723609 (+461966us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.723629 (+ 20us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.841888 (+118259us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.048104 (+206216us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.101908 (+ 53804us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.110994 23893 tablet_service.cc:1467] Processing CreateTablet for tablet bcb8f7e83e3a456585ba2533e96b76b9 (DEFAULT_TABLE table=default.Table_2c0098a6f7e3460e8a2bed24e0ecfb77 [id=87ecc0bcaedf4181a30dd61d8aef95b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.118525 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3078cc1f6247609f8bd017ed076b45" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:52.122124 23922 raft_consensus.cc:2454] T f5ccd43e78574f3288b199fa14bcc0e8 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:52.123857 24174 tablet_bootstrap.cc:492] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:52.124776 24174 ts_tablet_manager.cc:1367] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.000s
I20240502 08:16:52.132285 23886 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 804) took 1045 ms. Trace:
I20240502 08:16:52.132432 23886 rpcz_store.cc:276] 0502 08:16:51.086557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.650368 (+563811us) service_pool.cc:225] Handling call
0502 08:16:51.650776 (+ 408us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.650804 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.967029 (+316225us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.132031 (+165002us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.132215 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.127487 25266 consensus_queue.cc:237] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.146008 23899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 807) took 1059 ms. Trace:
I20240502 08:16:52.146287 23899 rpcz_store.cc:276] 0502 08:16:51.086743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.669105 (+582362us) service_pool.cc:225] Handling call
0502 08:16:51.669740 (+ 635us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.669787 (+ 47us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.135039 (+465252us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.145621 (+ 10582us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.145979 (+ 358us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.133512 23886 tablet_service.cc:1467] Processing CreateTablet for tablet e0bd774640464d2a92b2e71126c83883 (DEFAULT_TABLE table=default.Table_c456616beb454b6ea33638f85512f97e [id=aa67c143e5cb44078846d98c0cf1345e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.147327 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 548ab3f9d0d84527b1092ae715ec2bdd (DEFAULT_TABLE table=default.Table_59fc9761e46d4ff0bad5029d2b0b25fc [id=54be5ee518c94ed697af4b2a222bde65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.151295 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955e7bd2216646c6aef5acce8d5b0d77" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:52.151794 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c17e1be6ef294769a01981f4d361be21" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:52.152020 23914 raft_consensus.cc:2454] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:52.151336 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c991463cf8ad4d3b8c0727e45c2317b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.142235 24174 raft_consensus.cc:348] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.157527 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbef606ce7ac499d9bd84b49d804621c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:52.159049 23601 catalog_manager.cc:5484] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67). New cstate: current_term: 1 leader_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:52.160118 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 7ed919909289410e867881baab841f53 (table default.Table_2edad641920b44ce9a4d5b7503f0279a [id=ee3912407a934444ad2c62f7b83d93ac]). Expected version 2 got 0
I20240502 08:16:52.162254 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet f797d3c6bdc4493398954fbbf9943971 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:52.156739 24174 raft_consensus.cc:374] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 08:16:52.165402 23606 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.225.65:44121 (request call id 128) took 1025 ms (client timeout 1000 ms). Trace:
W20240502 08:16:52.165717 23606 rpcz_store.cc:269] 0502 08:16:51.139659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.655192 (+515533us) service_pool.cc:225] Handling call
0502 08:16:52.165266 (+510074us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:52.073856 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:52.073966 (+ 110us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:52.073970 (+ 4us) write_op.cc:427] Acquired schema lock
0502 08:16:52.073972 (+ 2us) tablet.cc:660] Decoding operations
0502 08:16:52.074063 (+ 91us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:52.074073 (+ 10us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:52.074078 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0502 08:16:52.074132 (+ 54us) tablet.cc:699] Row locks acquired
0502 08:16:52.074135 (+ 3us) write_op.cc:260] PREPARE: finished
0502 08:16:52.074165 (+ 30us) write_op.cc:270] Start()
0502 08:16:52.074187 (+ 22us) write_op.cc:276] Timestamp: P: 1714637812074129 usec, L: 0
0502 08:16:52.074190 (+ 3us) op_driver.cc:331] REPLICATION: starting
0502 08:16:52.074309 (+ 119us) log.cc:842] Serialized 533 byte log entry
0502 08:16:52.154290 (+ 79981us) op_driver.cc:444] REPLICATION: finished
0502 08:16:52.156059 (+ 1769us) write_op.cc:301] APPLY: starting
0502 08:16:52.156108 (+ 49us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:52.156176 (+ 68us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:52.156204 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:52.156280 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:52.156341 (+ 61us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0502 08:16:52.156349 (+ 8us) write_op.cc:312] APPLY: finished
0502 08:16:52.156491 (+ 142us) log.cc:842] Serialized 23 byte log entry
0502 08:16:52.156570 (+ 79us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:52.156697 (+ 127us) write_op.cc:446] Released schema lock
0502 08:16:52.156706 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:52.162272 (+ 5566us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:52.162306 (+ 34us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":1536,"apply.run_cpu_time_us":598,"apply.run_wall_time_us":6599,"num_ops":1,"prepare.queue_time_us":1575,"prepare.run_cpu_time_us":507,"prepare.run_wall_time_us":506,"replication_time_us":79938}]]}
I20240502 08:16:52.163985 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a97427e819b4826b1ed89e6d021c8d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:52.163375 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet c5c377e037894728ac084b6500ad20de with delete_type TABLET_DATA_DELETED (replaced by tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:52.164927 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7b37f776df47da9aa6e7fba82a8999 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:52.165441 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet f5ccd43e78574f3288b199fa14bcc0e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:52.164970 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet 128c88af54d24ffd895e9de19cdb4c8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:52.165629 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 1da01e2c2f054ccfb53383e70e16da84 with delete_type TABLET_DATA_DELETED (replaced by tablet 3e36be367f9b4c568ce0f3278b6297d9) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:52.164860 23752 tablet_service.cc:1514] Processing DeleteTablet for tablet 654ce33d1e3d472eb6ae6a726eb803bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:52.164952 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ba1baa2279045b3adc58d187a12cbf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:52.165604 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a4104f4cb64889ad66810788ad6b29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:52.165736 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet 751808dfc1964b539f6f836d786156aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:52.163247 24174 raft_consensus.cc:729] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.172252 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e20e126b1c9546e4a194bf9842f444a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.172693 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb8f7e83e3a456585ba2533e96b76b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.176178 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296b1444c5bb4d319ab9c69373fff84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.178225 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b673548cd88d4f51975f7433f1509cc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:52.183140 23904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 811) took 1096 ms. Trace:
I20240502 08:16:52.183257 23904 rpcz_store.cc:276] 0502 08:16:51.086989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.048405 (+961416us) service_pool.cc:225] Handling call
0502 08:16:52.101950 (+ 53545us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.101984 (+ 34us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.179906 (+ 77922us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.182861 (+ 2955us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.183102 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.183840 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 3af208d6231a4cf18823e9fe367ca46d (DEFAULT_TABLE table=default.Table_872fcb31253f47e1976fbed00a641287 [id=30bc0d639d5d46f39aa4da2528842969]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.184281 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af208d6231a4cf18823e9fe367ca46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.185699 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 336f0eb5085546dba763e0d5c5dd5324. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.189541 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1626382a260044eb9100b8048332abb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.198606 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0bd774640464d2a92b2e71126c83883. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:52.196120 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b673548cd88d4f51975f7433f1509cc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:52.203303 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548ab3f9d0d84527b1092ae715ec2bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.094185 24026 rpcz_store.cc:276] 0502 08:16:49.119989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.690858 (+2570869us) service_pool.cc:225] Handling call
0502 08:16:51.691164 (+ 306us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.691181 (+ 17us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.997665 (+306484us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.086829 (+ 89164us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.087365 (+ 536us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.208160 24026 tablet_service.cc:1514] Processing DeleteTablet for tablet ee3d91d55ffd49ceb20d2c33408f5a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:52.208359 24026 tablet_service.cc:1514] Processing DeleteTablet for tablet a0224317e65747e1add533d33472f7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:52.208740 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 26c8325099af4d72ad00afb2316df885 (DEFAULT_TABLE table=default.Table_64b18a844e8d44ba8bca05606e5b34e8 [id=dbd1ab9190f94aebbdc794fd9f1fc074]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.209185 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c8325099af4d72ad00afb2316df885. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.094242 24036 rpcz_store.cc:276] 0502 08:16:49.125867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.847443 (+2721576us) service_pool.cc:225] Handling call
0502 08:16:51.848857 (+ 1414us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.848921 (+ 64us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:51.983433 (+134512us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.057028 (+ 73595us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.087657 (+ 30629us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.210116 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc2270a5461448ca0245bd875f4245a (DEFAULT_TABLE table=default.Table_e29dfece3b88466ba6e6f6db0c1a31fd [id=7c9478b76f0a46c1869bd31c27a8eddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.211067 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet 642744194e22461f8316a78689fd99ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:52.067904 25219 raft_consensus.cc:504] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } }
I20240502 08:16:52.212563 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc2270a5461448ca0245bd875f4245a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.212922 24028 tablet_service.cc:1467] Processing CreateTablet for tablet e34873a3f08043098f2740641b3dc41f (DEFAULT_TABLE table=default.Table_a47e8e08a0ef4797aaf83cfa8adf0019 [id=630b2f4ab433442ab82b9451c8f54f61]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:52.097383 23993 leader_election.cc:341] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: Tablet not RUNNING: STOPPED
W20240502 08:16:52.213639 23993 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 116366us
I20240502 08:16:52.106003 23996 leader_election.cc:302] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
W20240502 08:16:52.217840 23996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 111875us
I20240502 08:16:52.217907 25297 raft_consensus.cc:2781] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:16:52.218940 23894 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 789) took 1133 ms. Trace:
I20240502 08:16:52.082793 23765 tablet_service.cc:1467] Processing CreateTablet for tablet fa3d945d3897423a9c21018b25434fd1 (DEFAULT_TABLE table=default.Table_f14ecd9121174b1db356ecd80bbac528 [id=5550792d0bc942f38dd39b94e642c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.219594 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa3d945d3897423a9c21018b25434fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.217995 25297 raft_consensus.cc:482] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a297a6c7c2504ca68e87b24f6691ffb2)
I20240502 08:16:52.207722 25222 raft_consensus.cc:2726] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:52.208169 24031 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 828) took 3095 ms. Trace:
I20240502 08:16:52.220927 24031 rpcz_store.cc:276] 0502 08:16:49.112355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.687667 (+2575312us) service_pool.cc:225] Handling call
0502 08:16:51.688106 (+ 439us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.688136 (+ 30us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.057439 (+369303us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.064394 (+ 6955us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.208146 (+143752us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.094264 24035 rpcz_store.cc:276] 0502 08:16:49.152919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.920990 (+2768071us) service_pool.cc:225] Handling call
0502 08:16:51.928001 (+ 7011us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.938154 (+ 10153us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.006495 (+ 68341us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.063638 (+ 57143us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.088962 (+ 25324us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.221922 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e34873a3f08043098f2740641b3dc41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.219684 25219 leader_election.cc:288] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:52.098613 24030 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 822) took 2993 ms. Trace:
I20240502 08:16:52.223299 24030 rpcz_store.cc:276] 0502 08:16:49.105560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.500013 (+394453us) service_pool.cc:225] Handling call
0502 08:16:50.748642 (+1248629us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.748656 (+ 14us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.028900 (+1280244us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.098441 (+ 69541us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.098592 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.224058 24031 tablet_service.cc:1467] Processing CreateTablet for tablet b1a2bcc6937c4a95b71103dbf8738702 (DEFAULT_TABLE table=default.Table_fc86e7c02217467c9a9e596e55aeee08 [id=a87a2af04b464ab283571bbc57ead567]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.224493 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a2bcc6937c4a95b71103dbf8738702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.207010 24023 tablet_service.cc:1467] Processing CreateTablet for tablet ba3b3abbb2e94c87ba19cc5695006e07 (DEFAULT_TABLE table=default.Table_39434b47dd9b421796ea5924f95292b9 [id=4c6173704b9e476fa4b836ba7fe5ff4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.223721 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet c48cea94856f43b6861c3ba360414888 with delete_type TABLET_DATA_DELETED (replaced by tablet e34873a3f08043098f2740641b3dc41f) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:52.223332 25154 raft_consensus.cc:504] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.226779 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751808dfc1964b539f6f836d786156aa" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:52.220237 25297 raft_consensus.cc:3037] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:16:52.227679 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b291bdb81a4efbac7f6691e5577bbe" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:52.227958 24024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 837) took 3105 ms. Trace:
I20240502 08:16:52.220798 25222 consensus_queue.cc:1035] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240502 08:16:52.228565 25154 leader_election.cc:288] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:52.226123 24174 consensus_queue.cc:260] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.206507 25227 consensus_queue.cc:237] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.232065 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3b3abbb2e94c87ba19cc5695006e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.231989 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 39cc4ef1174a4358bad777b0471c5f76 (DEFAULT_TABLE table=default.Table_7952191ae36044e9aef62593bbe0bb27 [id=7125223862ae4d04b11a726f81b81073]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.232419 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39cc4ef1174a4358bad777b0471c5f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.228040 24024 rpcz_store.cc:276] 0502 08:16:49.122393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.757434 (+2635041us) service_pool.cc:225] Handling call
0502 08:16:51.838475 (+ 81041us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.838718 (+ 243us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.054416 (+215698us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.225178 (+170762us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.227939 (+ 2761us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.227188 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654ce33d1e3d472eb6ae6a726eb803bb" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:52.243456 23755 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43544 (request call id 898) took 1514 ms. Trace:
I20240502 08:16:52.245232 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2aa20459484e4b98e0ae372418020f (DEFAULT_TABLE table=default.Table_ff75818c766d43569443cacbfcb972c1 [id=22d38df685d2423da2228076975133a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.245692 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b2aa20459484e4b98e0ae372418020f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.245644 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet e2126bf2d4ab44628b52481dc2ec3a99 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:49 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:52.247578 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1da01e2c2f054ccfb53383e70e16da84" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:52.246763 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2d7b6f4e442ae95b2d1002d571b6a" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:52.246821 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642744194e22461f8316a78689fd99ff" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:52.250126 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet f5ccd43e78574f3288b199fa14bcc0e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:52.252740 24180 ts_tablet_manager.cc:1888] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:52.253480 24027 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 849) took 2304 ms. Trace:
I20240502 08:16:52.239763 23901 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 787) took 1155 ms. Trace:
I20240502 08:16:52.240742 23887 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 792) took 1155 ms. Trace:
I20240502 08:16:52.242293 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e546c1bb78a848ad8125e8e8b08434db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.253922 23887 rpcz_store.cc:276] 0502 08:16:51.085447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.088556 (+ 3109us) service_pool.cc:225] Handling call
0502 08:16:51.701499 (+612943us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.701529 (+ 30us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.237712 (+536183us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.240553 (+ 2841us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.240715 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.254144 23901 rpcz_store.cc:276] 0502 08:16:51.084414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.084539 (+ 125us) service_pool.cc:225] Handling call
0502 08:16:51.689784 (+605245us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.689806 (+ 22us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.236753 (+546947us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.239545 (+ 2792us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.239728 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.254377 23894 rpcz_store.cc:276] 0502 08:16:51.085139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.087214 (+ 2075us) service_pool.cc:225] Handling call
0502 08:16:51.251511 (+164297us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.251540 (+ 29us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.127712 (+876172us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.218680 (+ 90968us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.218897 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.255918 24025 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 842) took 3103 ms. Trace:
I20240502 08:16:52.256017 24025 rpcz_store.cc:276] 0502 08:16:49.152781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.919135 (+2766354us) service_pool.cc:225] Handling call
0502 08:16:51.922922 (+ 3787us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.922977 (+ 55us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.053740 (+130763us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.255737 (+201997us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.255896 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.256306 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a4104f4cb64889ad66810788ad6b29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:52.256501 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet f685cbe705b149e69f9536e270a74a42 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:50 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:52.256629 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet fe5e08ad6af2442bb0a8fcc80a99b532 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:50 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:52.256742 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 303a0a3748ee422faab856e1c4bff66d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:49 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:52.256860 24025 tablet_service.cc:1514] Processing DeleteTablet for tablet 29f6c3cea4b14364b6d073a92171119d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:50 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:52.247398 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7b37f776df47da9aa6e7fba82a8999" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:52.253592 24027 rpcz_store.cc:276] 0502 08:16:49.948507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.982724 (+2034217us) service_pool.cc:225] Handling call
0502 08:16:52.003022 (+ 20298us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.003048 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.029819 (+ 26771us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.253332 (+223513us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.253463 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.249676 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a19310948e248d1a85479c41c61bda2" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:52.262547 24030 tablet_service.cc:1514] Processing DeleteTablet for tablet 654ce33d1e3d472eb6ae6a726eb803bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:52.263530 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955e7bd2216646c6aef5acce8d5b0d77" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:52.263824 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbef606ce7ac499d9bd84b49d804621c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:52.267907 24059 raft_consensus.cc:1226] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Rejecting Update request from peer a297a6c7c2504ca68e87b24f6691ffb2 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20240502 08:16:52.272189 24040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 852) took 2323 ms. Trace:
I20240502 08:16:52.272352 24180 log.cc:1196] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/df9402cf8abc4b4d893d42f9a3a99016
I20240502 08:16:52.272313 24040 rpcz_store.cc:276] 0502 08:16:49.948766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.013909 (+2065143us) service_pool.cc:225] Handling call
0502 08:16:52.030531 (+ 16622us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.030554 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.219736 (+189182us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.271971 (+ 52235us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.272161 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.272791 24032 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 847) took 3107 ms. Trace:
I20240502 08:16:52.272881 24032 rpcz_store.cc:276] 0502 08:16:49.165467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.974585 (+2809118us) service_pool.cc:225] Handling call
0502 08:16:51.975158 (+ 573us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.975197 (+ 39us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.233271 (+258074us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.272532 (+ 39261us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.272766 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.257980 24174 ts_tablet_manager.cc:1398] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.133s user 0.005s sys 0.001s
I20240502 08:16:52.243547 23755 rpcz_store.cc:276] 0502 08:16:50.729385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.820264 (+1090879us) service_pool.cc:225] Handling call
0502 08:16:51.871509 (+ 51245us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.871537 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.051643 (+180106us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.057496 (+ 5853us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.243431 (+185935us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.275066 24174 tablet_bootstrap.cc:492] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:52.276532 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_89e8be3491994ad4adc3c32d36c23d27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:52.277333 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89e8be3491994ad4adc3c32d36c23d27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:52.277635 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128c88af54d24ffd895e9de19cdb4c8a" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:52.282331 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1e7d21d372ec4ef9bd840ff21bbea400"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:52.282985 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e7d21d372ec4ef9bd840ff21bbea400 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:52.284453 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 87a3df4fbb224dd0b4ec6141584d374b
I20240502 08:16:52.285502 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b62ee1a1ff144e1d9447b5fb6ee38831"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:52.286084 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b62ee1a1ff144e1d9447b5fb6ee38831 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:52.290436 24034 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 851) took 2341 ms. Trace:
I20240502 08:16:52.290544 24034 rpcz_store.cc:276] 0502 08:16:49.948658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.010617 (+2061959us) service_pool.cc:225] Handling call
0502 08:16:52.045343 (+ 34726us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.045385 (+ 42us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.259753 (+214368us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.290221 (+ 30468us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.290396 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.290947 24021 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 843) took 3138 ms. Trace:
I20240502 08:16:52.291060 24021 rpcz_store.cc:276] 0502 08:16:49.152855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.920005 (+2767150us) service_pool.cc:225] Handling call
0502 08:16:51.920440 (+ 435us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.920465 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.000399 (+ 79934us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.290756 (+290357us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.290919 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.291754 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 9f3c00e5393c47df924bcb5dcf2a0f19
I20240502 08:16:52.292011 24039 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 839) took 3167 ms. Trace:
I20240502 08:16:52.292133 24039 rpcz_store.cc:276] 0502 08:16:49.124562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.844854 (+2720292us) service_pool.cc:225] Handling call
0502 08:16:51.918360 (+ 73506us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.918429 (+ 69us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.219211 (+300782us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.291824 (+ 72613us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.291989 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.292622 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_efdf6a98cfe94b879b0959eccdbf3dac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:52.293334 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_efdf6a98cfe94b879b0959eccdbf3dac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:52.294596 23605 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 8beca1a596be457399295bdc3cdc3a8d
I20240502 08:16:52.295042 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6a3a4cab06a545d2b1af63c5bb0e695a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:52.295657 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a3a4cab06a545d2b1af63c5bb0e695a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:52.297883 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet d2bb4dba23ac4b4eb53b3f2d75d0442e
I20240502 08:16:52.298733 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_03bc1e423ebc4377b041699c570137b1" } new_table_name: "default.Table_e60a5e107a3d4a019d41454b184f9ef9" modify_external_catalogs: true
W20240502 08:16:52.300276 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 54619769ce264106b1ee3765572f13ba consensus state has no leader
I20240502 08:16:52.300670 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a10aba559706443aae94d509cbb5eefa" } new_table_name: "default.Table_762d8c4aee05415b89c25e8b7b3e85b7" modify_external_catalogs: true
I20240502 08:16:52.302059 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 2085ced52fff46ebaa98324cbb2d8c11
I20240502 08:16:52.302619 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fbc10d796e98456ea3d7899ece9dcfc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:52.303156 23610 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36668 (request call id 1039) took 1952 ms. Trace:
W20240502 08:16:52.303212 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fbc10d796e98456ea3d7899ece9dcfc4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:52.303337 23610 rpcz_store.cc:276] 0502 08:16:50.350680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.775536 (+1424856us) service_pool.cc:225] Handling call
0502 08:16:51.776118 (+ 582us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0502 08:16:51.776447 (+ 329us) catalog_manager.cc:2749] Modifying in-memory table state
0502 08:16:51.776538 (+ 91us) catalog_manager.cc:2758] Locking tablets
0502 08:16:52.068689 (+292151us) catalog_manager.cc:2772] Removing table and tablets from system table
0502 08:16:52.151123 (+ 82434us) catalog_manager.cc:2790] Removing table from by-name map
0502 08:16:52.301905 (+150782us) catalog_manager.cc:2806] Committing in-memory state
0502 08:16:52.301970 (+ 65us) catalog_manager.cc:2810] Aborting table tasks
0502 08:16:52.303110 (+ 1140us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:52.071681 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:16:52.071825 (+ 144us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:16:52.071832 (+ 7us) write_op.cc:427] Acquired schema lock
0502 08:16:52.071834 (+ 2us) tablet.cc:660] Decoding operations
0502 08:16:52.071996 (+ 162us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:16:52.072007 (+ 11us) write_op.cc:633] Partition lock acquired for write op
0502 08:16:52.072012 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0502 08:16:52.072165 (+ 153us) tablet.cc:699] Row locks acquired
0502 08:16:52.072168 (+ 3us) write_op.cc:260] PREPARE: finished
0502 08:16:52.072211 (+ 43us) write_op.cc:270] Start()
0502 08:16:52.072245 (+ 34us) write_op.cc:276] Timestamp: P: 1714637812072166 usec, L: 0
0502 08:16:52.072249 (+ 4us) op_driver.cc:331] REPLICATION: starting
0502 08:16:52.072445 (+ 196us) log.cc:842] Serialized 762 byte log entry
0502 08:16:52.150323 (+ 77878us) op_driver.cc:444] REPLICATION: finished
0502 08:16:52.150550 (+ 227us) write_op.cc:301] APPLY: starting
0502 08:16:52.150570 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0502 08:16:52.150624 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0502 08:16:52.150651 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:16:52.150732 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:16:52.150775 (+ 43us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0502 08:16:52.150782 (+ 7us) write_op.cc:312] APPLY: finished
0502 08:16:52.150889 (+ 107us) log.cc:842] Serialized 29 byte log entry
0502 08:16:52.150967 (+ 78us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:52.150989 (+ 22us) write_op.cc:446] Released schema lock
0502 08:16:52.150996 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0502 08:16:52.151066 (+ 70us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:16:52.151072 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":109,"apply.run_cpu_time_us":581,"apply.run_wall_time_us":581,"num_ops":2,"prepare.queue_time_us":2298,"prepare.run_cpu_time_us":841,"prepare.run_wall_time_us":841,"replication_time_us":77993}]]}
I20240502 08:16:52.304410 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0f919156226c440294ddd9a6e379cbff" } new_table_name: "default.Table_82f82cd6e702420283c288c10f905d90" modify_external_catalogs: true
I20240502 08:16:52.320757 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet c17e1be6ef294769a01981f4d361be21 (table default.Table_498b372c15a04b70b87bd67b543501cf [id=38e955d6bd644d24a7c45172ab60282b]) successfully deleted
W20240502 08:16:52.324155 23723 leader_election.cc:341] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:52.325004 24174 tablet_bootstrap.cc:654] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.327628 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_444fa6f47a2d4a21b14a426d5e90b33b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:52.328393 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_444fa6f47a2d4a21b14a426d5e90b33b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:52.329034 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af6efeebaebc495aba017a922e243c39 consensus state has no leader
I20240502 08:16:52.329525 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_43ea406acdfc40b7b657b4f6e1c6db50" } new_table_name: "default.Table_e5ad52e5d61a473c941cc2a712798cf0" modify_external_catalogs: true
W20240502 08:16:52.332408 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89a17261fbe0490fbe73ffe199ebb7be consensus state has no leader
W20240502 08:16:52.334905 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88bf97d0a0a54e828c3b0902976e0500 consensus state has no leader
W20240502 08:16:52.336974 25031 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.005s sys 0.023s
W20240502 08:16:52.337074 25031 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s user 0.005s sys 0.023s
I20240502 08:16:52.337323 24180 ts_tablet_manager.cc:1909] T df9402cf8abc4b4d893d42f9a3a99016 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:52.338531 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet 87a3df4fbb224dd0b4ec6141584d374b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:51 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:52.341945 23887 tablet_service.cc:1467] Processing CreateTablet for tablet e6e633f26b0e4be38e93af3d65e2e636 (DEFAULT_TABLE table=default.Table_db2ecaab8195464388e2ba94695fce03 [id=490733fbe9e84df1b842fa52fd197e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.342425 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e633f26b0e4be38e93af3d65e2e636. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.342792 23901 tablet_service.cc:1467] Processing CreateTablet for tablet e9a35b9486da4aac8f3d3febc93b7816 (DEFAULT_TABLE table=default.Table_47fe484d653742f582516ba50ff30131 [id=10b3982b52054f8cb38f5b9f32739110]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.343111 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 4e32319fa9244ed2a323b2b3f5940748 (DEFAULT_TABLE table=default.Table_41a2401b3f4c4748b1d3eb964d8053d5 [id=959f40776ede4f2e8dfa0b1b849f3f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.343278 24173 tablet_bootstrap.cc:492] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:52.343366 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e32319fa9244ed2a323b2b3f5940748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.343415 24173 ts_tablet_manager.cc:1367] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.671s user 0.012s sys 0.015s
W20240502 08:16:52.344687 23723 leader_election.cc:341] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 9c072eb45e1b41eeb5aafa6d5a786944
W20240502 08:16:52.344875 23723 leader_election.cc:341] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:52.345022 23723 leader_election.cc:341] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:52.345331 23723 leader_election.cc:341] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:52.345549 23723 leader_election.cc:341] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: c17e1be6ef294769a01981f4d361be21
W20240502 08:16:52.345701 23723 leader_election.cc:341] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:52.345804 23723 leader_election.cc:341] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:52.349083 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3c00e5393c47df924bcb5dcf2a0f19 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:51 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:52.349304 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet d2bb4dba23ac4b4eb53b3f2d75d0442e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:52 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:52.349424 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 2085ced52fff46ebaa98324cbb2d8c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:51 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:52.352916 23723 leader_election.cc:341] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:52.356231 24175 tablet_bootstrap.cc:492] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:52.356389 24175 ts_tablet_manager.cc:1367] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.645s user 0.007s sys 0.003s
W20240502 08:16:52.356611 23996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 138500us
I20240502 08:16:52.310700 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c8131846dc41678827b86e87be18d3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:52.357417 23918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c8131846dc41678827b86e87be18d3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:52.360101 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a35b9486da4aac8f3d3febc93b7816. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.362987 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c377e037894728ac084b6500ad20de" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:52.363282 23784 raft_consensus.cc:2379] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in this term.
I20240502 08:16:52.363565 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797d3c6bdc4493398954fbbf9943971" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:16:52.367460 23723 leader_election.cc:341] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:52.358899 24173 raft_consensus.cc:348] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.367951 24173 raft_consensus.cc:374] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.368041 24173 raft_consensus.cc:729] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
W20240502 08:16:52.360877 23996 leader_election.cc:341] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:52.371949 23883 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:33614 (request call id 154) took 2289 ms. Trace:
I20240502 08:16:52.372062 23883 rpcz_store.cc:276] 0502 08:16:50.083015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:50.083088 (+ 73us) service_pool.cc:225] Handling call
0502 08:16:50.730990 (+647902us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x616000134e54. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:50.745404 (+ 14414us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x6160000a5754. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:51.157684 (+412280us) spinlock_profiling.cc:243] Waited 411 ms on lock 0x6160001f2154. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:51.696491 (+538807us) spinlock_profiling.cc:243] Waited 53.4 ms on lock 0x616000136c54. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:52.076073 (+379582us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x6160001f2754. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:52.122418 (+ 46345us) spinlock_profiling.cc:243] Waited 46 ms on lock 0x6160000a6f54. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:52.371914 (+249496us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2083907840}
I20240502 08:16:52.372632 25184 log.cc:824] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:52.372853 23900 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 806) took 1154 ms. Trace:
I20240502 08:16:52.368290 24173 consensus_queue.cc:260] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:52.379223 23597 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: DeleteTablet request on kudu.tserver.TabletServerAdminService from 127.0.0.1:33620 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 17 similar messages]
I20240502 08:16:52.386077 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 785) took 4222 ms. Trace:
I20240502 08:16:52.386222 24180 rpcz_store.cc:276] 0502 08:16:48.163577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.164714 (+ 1137us) service_pool.cc:225] Handling call
0502 08:16:52.060569 (+3895855us) ts_tablet_manager.cc:1120] Deleting tablet df9402cf8abc4b4d893d42f9a3a99016
0502 08:16:52.060580 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.248904 (+188324us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.252640 (+ 3736us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.386053 (+133413us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3895707}
I20240502 08:16:52.387041 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet df9402cf8abc4b4d893d42f9a3a99016 (table default.Table_4b967ef2b2fc4e2caa4219afcae841a6 [id=c32fc96889554c32bdf1d4550633e91f]) successfully deleted
I20240502 08:16:52.372965 23900 rpcz_store.cc:276] 0502 08:16:51.086673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.665620 (+578947us) service_pool.cc:225] Handling call
0502 08:16:51.666337 (+ 717us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.666387 (+ 50us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.232219 (+565832us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.241265 (+ 9046us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.241402 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.388664 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 8d44b485936845b992315645891a5004 (DEFAULT_TABLE table=default.Table_ba6d528fd464438eb4c79eeade694e24 [id=65a77f5506414a849458777eab928d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.367594 23723 leader_election.cc:302] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:52.389082 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d44b485936845b992315645891a5004. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.390048 25296 raft_consensus.cc:2726] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:52.393381 23893 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 813) took 1306 ms. Trace:
I20240502 08:16:52.395015 24173 ts_tablet_manager.cc:1398] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
I20240502 08:16:52.397136 23895 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 799) took 1311 ms. Trace:
I20240502 08:16:52.369904 25317 consensus_queue.cc:1035] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.028s
I20240502 08:16:52.370041 23996 leader_election.cc:302] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:52.398871 25227 raft_consensus.cc:2726] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:52.380373 23886 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 814) took 1293 ms. Trace:
I20240502 08:16:52.400539 23886 rpcz_store.cc:276] 0502 08:16:51.087319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.132999 (+1045680us) service_pool.cc:225] Handling call
0502 08:16:52.147853 (+ 14854us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.147913 (+ 60us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.244470 (+ 96557us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.380137 (+135667us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.380342 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.393497 23893 rpcz_store.cc:276] 0502 08:16:51.087248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.110560 (+1023312us) service_pool.cc:225] Handling call
0502 08:16:52.111172 (+ 612us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.111196 (+ 24us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.202329 (+ 91133us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.393161 (+190832us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.393350 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.401695 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 98f4f2475d4848018414c3025a54488e (DEFAULT_TABLE table=default.Table_282e8b7fa027438c9e38f997527d7406 [id=760e39ce1ac54ca0b347324eb9127630]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.402135 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98f4f2475d4848018414c3025a54488e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.404474 24180 tablet_replica.cc:330] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:52.404722 24180 raft_consensus.cc:2229] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:52.404799 23723 leader_election.cc:302] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:52.404886 24180 raft_consensus.cc:2258] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:52.405014 23723 raft_consensus.cc:2670] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:16:52.405442 23723 leader_election.cc:302] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:52.405527 24180 ts_tablet_manager.cc:1875] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:52.405903 23723 leader_election.cc:302] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
W20240502 08:16:52.406088 23723 raft_consensus.cc:2670] T 9c072eb45e1b41eeb5aafa6d5a786944 P a297a6c7c2504ca68e87b24f6691ffb2: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:16:52.406462 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b291bdb81a4efbac7f6691e5577bbe" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:52.410820 23785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b811b81ddf2b4dc9b5e5f80331bff294" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:52.411259 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "050c16739efb4037a4adc78a2ffd3cba" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:52.412716 23889 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 812) took 1325 ms. Trace:
I20240502 08:16:52.412842 23889 rpcz_store.cc:276] 0502 08:16:51.087101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.108533 (+1021432us) service_pool.cc:225] Handling call
0502 08:16:52.109283 (+ 750us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.109331 (+ 48us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.192658 (+ 83327us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.412528 (+219870us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.412689 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.413650 23902 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 796) took 1278 ms. Trace:
I20240502 08:16:52.413751 23902 rpcz_store.cc:276] 0502 08:16:51.085822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.088628 (+ 2806us) service_pool.cc:225] Handling call
0502 08:16:51.671816 (+583188us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.671841 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.121273 (+449432us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.354621 (+233348us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.364575 (+ 9954us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.414280 25304 raft_consensus.cc:2726] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:52.416270 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7cbf1c0a37454beb966aa61b4f8caf72 consensus state has no leader
I20240502 08:16:52.416457 23890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 810) took 1329 ms. Trace:
I20240502 08:16:52.416536 23890 rpcz_store.cc:276] 0502 08:16:51.086917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.994514 (+907597us) service_pool.cc:225] Handling call
0502 08:16:52.100625 (+106111us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.102078 (+ 1453us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.205802 (+103724us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.416291 (+210489us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.416426 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.401068 23886 tablet_service.cc:1467] Processing CreateTablet for tablet d5c82820278b4a0c8f7d586f87104bf0 (DEFAULT_TABLE table=default.Table_722c9cabaa9a4f43a7c8aeab8a17dc05 [id=98df2952710847e9af30703324a08d66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.423302 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f2d7b6f4e442ae95b2d1002d571b6a" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:52.423489 23723 leader_election.cc:302] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:52.423791 25304 raft_consensus.cc:2781] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:52.423903 25304 raft_consensus.cc:482] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:52.423988 25304 raft_consensus.cc:3037] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.428972 23918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9402cf8abc4b4d893d42f9a3a99016" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:52.429196 23918 raft_consensus.cc:3037] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:16:52.397228 23895 rpcz_store.cc:276] 0502 08:16:51.086067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.088702 (+ 2635us) service_pool.cc:225] Handling call
0502 08:16:51.705706 (+617004us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.705729 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.236705 (+530976us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.396955 (+160250us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.397102 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.430828 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "050c16739efb4037a4adc78a2ffd3cba" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:52.431658 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 1cebfc5ffc184de9a2e703aa4e333c3a (DEFAULT_TABLE table=default.Table_18f465f6e9d048ac8bb97321715d2a6e [id=6be0266503de408a97643c9a8b255096]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.432104 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cebfc5ffc184de9a2e703aa4e333c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.432538 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 88bf97d0a0a54e828c3b0902976e0500 (DEFAULT_TABLE table=default.Table_a10aba559706443aae94d509cbb5eefa [id=bec84f8866b44454a38303786e974759]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.433185 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88bf97d0a0a54e828c3b0902976e0500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.433507 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c82820278b4a0c8f7d586f87104bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.441326 24029 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 846) took 3276 ms. Trace:
I20240502 08:16:52.442154 23884 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:33614 (request call id 149) took 3751 ms. Trace:
I20240502 08:16:52.442267 23884 rpcz_store.cc:276] 0502 08:16:48.690871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.691033 (+ 162us) service_pool.cc:225] Handling call
0502 08:16:48.939233 (+248200us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x6160000e5654. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:49.368488 (+429255us) spinlock_profiling.cc:243] Waited 426 ms on lock 0x616000135454. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:49.875539 (+507051us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x6160001d1a54. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:50.749717 (+874178us) spinlock_profiling.cc:243] Waited 873 ms on lock 0x6160000a5754. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:51.154613 (+404896us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x6160001f2154. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:51.895942 (+741329us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x616000136c54. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:52.442097 (+546155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5917709312}
I20240502 08:16:52.452667 23895 tablet_service.cc:1467] Processing CreateTablet for tablet dfd897ba97204b6788fd2d99cf2a0303 (DEFAULT_TABLE table=default.Table_97534a2075bc4b7297b36916b7a5f77c [id=5823995f324846088c24d9ffd5d742c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.422542 23904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 816) took 1335 ms. Trace:
I20240502 08:16:52.453136 23904 rpcz_store.cc:276] 0502 08:16:51.087457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.183544 (+1096087us) service_pool.cc:225] Handling call
0502 08:16:52.184077 (+ 533us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.184111 (+ 34us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.374998 (+190887us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.422378 (+ 47380us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.422520 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.454144 23904 tablet_service.cc:1467] Processing CreateTablet for tablet d87820f2490f4631b455e8dc942f7147 (DEFAULT_TABLE table=default.Table_fc2d835b22c1417bad53bb6f66bde481 [id=d361511883cd45d28d4965e0a718e0f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.457924 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d87820f2490f4631b455e8dc942f7147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.441459 24029 rpcz_store.cc:276] 0502 08:16:49.164357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.929114 (+2764757us) service_pool.cc:225] Handling call
0502 08:16:51.947429 (+ 18315us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.947480 (+ 51us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.088468 (+140988us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.441087 (+352619us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.441291 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.465520 24173 tablet_bootstrap.cc:492] T 1ba1baa2279045b3adc58d187a12cbf3 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:52.467056 24033 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 808) took 3376 ms. Trace:
I20240502 08:16:52.467193 24033 rpcz_store.cc:276] 0502 08:16:49.090564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.454491 (+363927us) service_pool.cc:225] Handling call
0502 08:16:50.754858 (+1300367us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:50.754878 (+ 20us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.056498 (+1301620us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.466895 (+410397us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.467036 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:16:52.471505 23597 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 92261.3us
W20240502 08:16:52.471789 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 48324.9us
W20240502 08:16:52.472160 23723 leader_election.cc:341] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:52.472160 25296 consensus_queue.cc:1035] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:52.472978 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet a3043c5ed98c401b82c6e967ca3f3613 (table default.Table_3e0123590a92473dabaacc9c158859e7 [id=d1a317371a5b43f185eb6453861e112a]) successfully deleted
I20240502 08:16:52.472282 23723 leader_election.cc:302] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
W20240502 08:16:52.473749 23723 leader_election.cc:341] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:52.473873 23723 leader_election.cc:302] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:52.474035 25158 raft_consensus.cc:2726] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:52.474306 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet bcfacc302ff04178bfffc48111cd7a3f (table default.Table_d416b40fe3d24dafa8db5ee61d13acc0 [id=fa5100e581a7430ab8f0c19b2daae994]) successfully deleted
I20240502 08:16:52.476138 23993 leader_election.cc:302] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:52.476677 23993 leader_election.cc:341] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: df9402cf8abc4b4d893d42f9a3a99016
I20240502 08:16:52.477001 25222 raft_consensus.cc:2726] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:52.477105 25317 consensus_queue.cc:1035] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:52.479244 24037 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 853) took 2530 ms. Trace:
I20240502 08:16:52.479331 24037 rpcz_store.cc:276] 0502 08:16:49.948870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.027538 (+2078668us) service_pool.cc:225] Handling call
0502 08:16:52.028036 (+ 498us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.028068 (+ 32us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.214413 (+186345us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.468421 (+254008us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.479225 (+ 10804us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.479739 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 506644a1ead54a1faeaa8583ef96b449 (DEFAULT_TABLE table=default.Table_2e5b16fa5b3b4be3af4d6843606d91f6 [id=362af6ee46a740b28b0fa3e9ae3391fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.480218 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 506644a1ead54a1faeaa8583ef96b449. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.481359 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd897ba97204b6788fd2d99cf2a0303. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.486018 24038 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 845) took 3322 ms. Trace:
I20240502 08:16:52.486214 25012 ts_tablet_manager.cc:1888] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:52.486311 25012 log.cc:1196] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/643ce929e74c499da835fbe9b63b8880
W20240502 08:16:52.486380 23858 leader_election.cc:341] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 4a57ed188315493198e95cefb6907ab6
I20240502 08:16:52.486498 23858 leader_election.cc:302] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:52.486738 25308 raft_consensus.cc:2726] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:52.487035 23858 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 86830.3us
W20240502 08:16:52.487910 23858 leader_election.cc:341] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 319fabf91458438ba5e40ab68b323055
I20240502 08:16:52.489619 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3a66619663c642f5be35fd8249184e1a" } modify_external_catalogs: true
I20240502 08:16:52.489727 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a6c2b028992644d987b8a3ca62b1d325"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:52.489885 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3a66619663c642f5be35fd8249184e1a" } modify_external_catalogs: true
W20240502 08:16:52.490290 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6c2b028992644d987b8a3ca62b1d325 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:52.492404 25297 raft_consensus.cc:504] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.492743 24052 raft_consensus.cc:1226] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Rejecting Update request from peer a297a6c7c2504ca68e87b24f6691ffb2 for earlier term 1. Current term is 2. Ops: []
I20240502 08:16:52.493463 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8a9cdf1f53b0494694035cd549988bf0" } modify_external_catalogs: true
I20240502 08:16:52.493701 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8a9cdf1f53b0494694035cd549988bf0" } modify_external_catalogs: true
I20240502 08:16:52.493908 23886 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 821) took 1406 ms. Trace:
I20240502 08:16:52.493981 23886 rpcz_store.cc:276] 0502 08:16:51.087878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.400773 (+1312895us) service_pool.cc:225] Handling call
0502 08:16:52.422818 (+ 22045us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.422846 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.490880 (+ 68034us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.493738 (+ 2858us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.493885 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.494429 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2c637e208044d8aaf1774f448e0527 (DEFAULT_TABLE table=default.Table_0f6ad9ecb09e4a03b995f7350e0a3af0 [id=0b630609c7294606bff513bbcd28c4a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.496775 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_482a1b0b7a064e02805ac3b86fc62f99" } modify_external_catalogs: true
I20240502 08:16:52.496973 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_482a1b0b7a064e02805ac3b86fc62f99" } modify_external_catalogs: true
I20240502 08:16:52.499600 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4c4a02d4a6eb4c87971a03d670f5fdaf" } modify_external_catalogs: true
I20240502 08:16:52.499752 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4c4a02d4a6eb4c87971a03d670f5fdaf" } modify_external_catalogs: true
I20240502 08:16:52.500540 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet 18d284f71937459db0c0676ce4568175 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:52.504180 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_80ca7b4c29584de0b0fddf31e64d9f0b" } new_table_name: "default.Table_787b2d1011974a8d95e16bf54ec92ffd" modify_external_catalogs: true
I20240502 08:16:52.504398 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e67c0ee30255468aa0674964089b483e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:52.504976 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e67c0ee30255468aa0674964089b483e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:52.506842 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fabad8a6e72a4724893dfa4c0877fdb2" } modify_external_catalogs: true
I20240502 08:16:52.507027 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fabad8a6e72a4724893dfa4c0877fdb2" } modify_external_catalogs: true
I20240502 08:16:52.510445 25012 ts_tablet_manager.cc:1909] T 643ce929e74c499da835fbe9b63b8880 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:52.511909 24173 tablet_bootstrap.cc:654] T 1ba1baa2279045b3adc58d187a12cbf3 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.486124 24038 rpcz_store.cc:276] 0502 08:16:49.163528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.923937 (+2760409us) service_pool.cc:225] Handling call
0502 08:16:51.947885 (+ 23948us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.947916 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.257162 (+309246us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.485761 (+228599us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.485990 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.517982 25296 raft_consensus.cc:2726] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:52.518085 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 44338.5us
I20240502 08:16:52.518317 25297 leader_election.cc:288] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:52.518363 23723 leader_election.cc:341] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:52.518465 23723 leader_election.cc:302] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:52.518857 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b811b81ddf2b4dc9b5e5f80331bff294" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:52.518977 23723 leader_election.cc:302] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:52.519280 25296 raft_consensus.cc:2726] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:52.519455 25296 raft_consensus.cc:2726] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:52.519480 23723 leader_election.cc:341] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:52.519585 23723 leader_election.cc:302] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:52.519563 25296 consensus_queue.cc:1035] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:52.519901 25296 raft_consensus.cc:2726] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:52.526690 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 518cb4b86ea34e95bef51c5c2a64d89b has no consensus state
I20240502 08:16:52.527593 24026 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 858) took 2578 ms. Trace:
I20240502 08:16:52.527700 24026 rpcz_store.cc:276] 0502 08:16:49.949331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.208485 (+2259154us) service_pool.cc:225] Handling call
0502 08:16:52.208966 (+ 481us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.208997 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.374557 (+165560us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.527392 (+152835us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.527565 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.527820 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 050c16739efb4037a4adc78a2ffd3cba
W20240502 08:16:52.528160 23723 leader_election.cc:341] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:52.529839 25297 consensus_queue.cc:1035] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:52.531477 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet 050c16739efb4037a4adc78a2ffd3cba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:52 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:52.532423 24173 tablet_bootstrap.cc:492] T 1ba1baa2279045b3adc58d187a12cbf3 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:52.532547 24173 ts_tablet_manager.cc:1367] T 1ba1baa2279045b3adc58d187a12cbf3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.000s
I20240502 08:16:52.534296 24173 raft_consensus.cc:348] T 1ba1baa2279045b3adc58d187a12cbf3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.534480 24173 raft_consensus.cc:374] T 1ba1baa2279045b3adc58d187a12cbf3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.534538 24173 raft_consensus.cc:729] T 1ba1baa2279045b3adc58d187a12cbf3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.534838 24173 consensus_queue.cc:260] T 1ba1baa2279045b3adc58d187a12cbf3 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.535264 24173 ts_tablet_manager.cc:1398] T 1ba1baa2279045b3adc58d187a12cbf3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:52.537158 24173 tablet_bootstrap.cc:492] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:52.538652 23723 leader_election.cc:302] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:52.539062 24036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 859) took 2589 ms. Trace:
I20240502 08:16:52.539134 24036 rpcz_store.cc:276] 0502 08:16:49.949404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.209890 (+2260486us) service_pool.cc:225] Handling call
0502 08:16:52.210311 (+ 421us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.210342 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.281042 (+ 70700us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.519838 (+238796us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.539045 (+ 19207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.541327 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 713) took 4834 ms. Trace:
I20240502 08:16:52.543119 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0b5edfb1bf6e470f97bbbb62d09e353f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:52.543388 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7e325050ee86487aa23e9b0edfbd2e1d" } new_table_name: "default.Table_ff158afbc2b3466d9f6027b544a797dd" modify_external_catalogs: true
W20240502 08:16:52.543834 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b5edfb1bf6e470f97bbbb62d09e353f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:52.543952 24173 tablet_bootstrap.cc:654] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.545089 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5e9c513ec4ac4635bf68b84207bfce51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:52.545660 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e9c513ec4ac4635bf68b84207bfce51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:52.488343 23858 leader_election.cc:341] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 1e5de793e74247dd9e73b7b1d58027d9
I20240502 08:16:52.546480 24181 ts_tablet_manager.cc:1888] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:52.546605 24181 log.cc:1196] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/319fabf91458438ba5e40ab68b323055
I20240502 08:16:52.546938 24181 ts_tablet_manager.cc:1909] T 319fabf91458438ba5e40ab68b323055 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:52.547260 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 724) took 5450 ms. Trace:
I20240502 08:16:52.547330 24181 rpcz_store.cc:276] 0502 08:16:47.097149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.099881 (+ 2732us) service_pool.cc:225] Handling call
0502 08:16:52.002839 (+4902958us) ts_tablet_manager.cc:1120] Deleting tablet 319fabf91458438ba5e40ab68b323055
0502 08:16:52.002848 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.350611 (+347763us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.546370 (+195759us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.547239 (+ 869us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4902697}
I20240502 08:16:52.547631 24181 tablet_replica.cc:330] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:52.547781 24181 raft_consensus.cc:2229] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:52.547924 24181 raft_consensus.cc:2258] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:52.548417 24181 ts_tablet_manager.cc:1875] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:52.534538 24022 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 850) took 2585 ms. Trace:
I20240502 08:16:52.548894 24022 rpcz_store.cc:276] 0502 08:16:49.948593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.995008 (+2046415us) service_pool.cc:225] Handling call
0502 08:16:52.026022 (+ 31014us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.026076 (+ 54us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.371009 (+344933us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.534328 (+163319us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.534509 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.550482 24031 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 861) took 2600 ms. Trace:
I20240502 08:16:52.550554 24031 rpcz_store.cc:276] 0502 08:16:49.949579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.221173 (+2271594us) service_pool.cc:225] Handling call
0502 08:16:52.224284 (+ 3111us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.224321 (+ 37us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.513216 (+288895us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.550343 (+ 37127us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.550464 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.551553 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a77d44d9077d44fb9db187597d42756b" } new_table_name: "default.Table_e91fa6b91a704e739fbfacbe1ce7ef29" modify_external_catalogs: true
I20240502 08:16:52.555722 24023 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 854) took 2606 ms. Trace:
I20240502 08:16:52.555802 24023 rpcz_store.cc:276] 0502 08:16:49.948994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.063679 (+2114685us) service_pool.cc:225] Handling call
0502 08:16:52.224891 (+161212us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.224930 (+ 39us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.517199 (+292269us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.555572 (+ 38373us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.555704 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.556000 25308 raft_consensus.cc:482] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:52.556149 25308 raft_consensus.cc:504] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.558213 24028 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 860) took 2608 ms. Trace:
I20240502 08:16:52.558300 24028 rpcz_store.cc:276] 0502 08:16:49.949474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.211239 (+2261765us) service_pool.cc:225] Handling call
0502 08:16:52.213130 (+ 1891us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.213172 (+ 42us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.509530 (+296358us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.558077 (+ 48547us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.558196 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.546223 23858 leader_election.cc:302] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:52.560637 25308 leader_election.cc:288] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:52.560777 25308 raft_consensus.cc:2726] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:52.561547 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcc7169796e4ee0910f5e7084849012" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:52.561811 24052 raft_consensus.cc:2454] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:52.562705 23861 leader_election.cc:302] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:52.562997 25308 raft_consensus.cc:2781] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:16:52.565770 23901 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 817) took 1478 ms. Trace:
I20240502 08:16:52.565945 23901 rpcz_store.cc:276] 0502 08:16:51.087536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.254340 (+1166804us) service_pool.cc:225] Handling call
0502 08:16:52.342977 (+ 88637us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.343007 (+ 30us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.457262 (+114255us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.565549 (+108287us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.565745 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.571388 23889 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 823) took 1483 ms. Trace:
I20240502 08:16:52.571573 23889 rpcz_store.cc:276] 0502 08:16:51.088041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.413086 (+1325045us) service_pool.cc:225] Handling call
0502 08:16:52.431890 (+ 18804us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.431923 (+ 33us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.553767 (+121844us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.571193 (+ 17426us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.571361 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.578642 24028 tablet_service.cc:1467] Processing CreateTablet for tablet f5496fa258de4988b9502136f0baf779 (DEFAULT_TABLE table=default.Table_33c693cfb89641a2921c7d39872b5bbc [id=68e36934ceb0416a86dc24c4fc50804b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.579377 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc133adb9a44cc7b56a3746d1eb76af (DEFAULT_TABLE table=default.Table_d20a3b8ecb6449a5a4cc9837354619b7 [id=1c25911cce0845ffb760a96926714b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.580124 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 521581fe87e547ce816b599578bddeea (DEFAULT_TABLE table=default.Table_d67c17e0dfd94a088bd9991625f3c31a [id=bd341da28438429895d79410c12dce3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.581852 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 8075b03643a14a01ab75921fc3d3dace (DEFAULT_TABLE table=default.Table_fabad8a6e72a4724893dfa4c0877fdb2 [id=ea894a4bd467422eb135ad75e7032193]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.582298 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 517354df966c4338a01a425251cccdd2 (DEFAULT_TABLE table=default.Table_1fa44e7d541041d3a86bed4be93b5a03 [id=4e5acee37ab44e979a690a2df5a78d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.582733 24022 tablet_service.cc:1467] Processing CreateTablet for tablet fa3d945d3897423a9c21018b25434fd1 (DEFAULT_TABLE table=default.Table_f14ecd9121174b1db356ecd80bbac528 [id=5550792d0bc942f38dd39b94e642c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.583271 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3c00e5393c47df924bcb5dcf2a0f19 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:51 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:52.583434 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet d2bb4dba23ac4b4eb53b3f2d75d0442e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:52 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:52.583508 24024 tablet_service.cc:1467] Processing CreateTablet for tablet a4fe3a2f9eac469d9a7da7b9a2fc24a8 (DEFAULT_TABLE table=default.Table_dd5271576d9746048fa96f022073b8fd [id=4640f69b48d44ee3baa1b6d4ab6fd6bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.583812 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet 87a3df4fbb224dd0b4ec6141584d374b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:51 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:52.584129 24026 tablet_service.cc:1467] Processing CreateTablet for tablet c991463cf8ad4d3b8c0727e45c2317b4 (DEFAULT_TABLE table=default.Table_1014ad7e82a245ea801b1866b929b1c5 [id=39339ca66938431c8de38927672a9f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.573648 23892 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 790) took 1488 ms. Trace:
I20240502 08:16:52.578716 23901 tablet_service.cc:1467] Processing CreateTablet for tablet d1218fd56b8a4e88827f275f952ad782 (DEFAULT_TABLE table=default.Table_f17a7a3f2d4547799bdbea376e3771b2 [id=8a6b12ac10854ce986dc1ad7c67b3153]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.584906 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1218fd56b8a4e88827f275f952ad782. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.585451 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 2085ced52fff46ebaa98324cbb2d8c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:51 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:52.580438 23890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 825) took 1492 ms. Trace:
I20240502 08:16:52.586138 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 050c16739efb4037a4adc78a2ffd3cba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:52 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:52.563060 25308 raft_consensus.cc:482] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:52.586259 25308 raft_consensus.cc:3037] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:16:52.582160 23903 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 809) took 1495 ms. Trace:
I20240502 08:16:52.586508 23903 rpcz_store.cc:276] 0502 08:16:51.086852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.987432 (+900580us) service_pool.cc:225] Handling call
0502 08:16:52.032193 (+ 44761us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.032232 (+ 39us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.507810 (+475578us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.582018 (+ 74208us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.582127 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.582230 25012 rpcz_store.cc:276] 0502 08:16:47.706636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.706834 (+ 198us) service_pool.cc:225] Handling call
0502 08:16:52.106092 (+4399258us) ts_tablet_manager.cc:1120] Deleting tablet 643ce929e74c499da835fbe9b63b8880
0502 08:16:52.106102 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.361191 (+255089us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.486132 (+124941us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.541299 (+ 55167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4399128}
I20240502 08:16:52.586959 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 715) took 4879 ms. Trace:
I20240502 08:16:52.587246 23890 rpcz_store.cc:276] 0502 08:16:51.088184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.416720 (+1328536us) service_pool.cc:225] Handling call
0502 08:16:52.432717 (+ 15997us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.432743 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.496600 (+ 63857us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.580275 (+ 83675us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.580419 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.589233 24174 tablet_bootstrap.cc:492] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:52.589726 24180 ts_tablet_manager.cc:1888] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:52.572331 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b811b81ddf2b4dc9b5e5f80331bff294" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:52.584326 23892 rpcz_store.cc:276] 0502 08:16:51.085244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.088530 (+ 3286us) service_pool.cc:225] Handling call
0502 08:16:51.697646 (+609116us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.697676 (+ 30us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.241494 (+543818us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.573481 (+331987us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.573629 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.593753 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5496fa258de4988b9502136f0baf779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.593909 23888 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 805) took 1462 ms. Trace:
I20240502 08:16:52.593982 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc133adb9a44cc7b56a3746d1eb76af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.593997 23888 rpcz_store.cc:276] 0502 08:16:51.086623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.656324 (+569701us) service_pool.cc:225] Handling call
0502 08:16:51.656777 (+ 453us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.656803 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.153529 (+496726us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.548893 (+395364us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.549047 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.594431 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517354df966c4338a01a425251cccdd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.587034 25012 rpcz_store.cc:276] 0502 08:16:47.707102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.707179 (+ 77us) service_pool.cc:225] Handling call
0502 08:16:52.586829 (+4879650us) ts_tablet_manager.cc:1120] Deleting tablet 48d96e42813f4169969dd5931e169923
0502 08:16:52.586840 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.586946 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4879503}
I20240502 08:16:52.594462 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 36dece738afd42b99f3f38d7fd11a0af (DEFAULT_TABLE table=default.Table_b1d809964c99404891db2d83cb16887f [id=6851bfc588bd45d097ed07ee8ad1b81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.594853 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36dece738afd42b99f3f38d7fd11a0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.594899 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 521581fe87e547ce816b599578bddeea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.595026 25012 tablet_replica.cc:330] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:52.595090 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa3d945d3897423a9c21018b25434fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.595260 25012 raft_consensus.cc:2229] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:52.595439 25012 raft_consensus.cc:2258] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:52.596041 23903 tablet_service.cc:1467] Processing CreateTablet for tablet ed545e81a6934cf8972ff84181048156 (DEFAULT_TABLE table=default.Table_33297782b5e64503b4186fe4a347f8e9 [id=d68c0a6c1e6441d18a6c08e4b1373ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.596378 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed545e81a6934cf8972ff84181048156. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.597290 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 1f42627a877c4f83a75320f1b5ab539f (DEFAULT_TABLE table=default.Table_5fe628d3da624233941f440e3743da88 [id=ca161ce6780b446fa5743d3974cfa27b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.597599 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f42627a877c4f83a75320f1b5ab539f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.597993 23893 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 822) took 1510 ms. Trace:
I20240502 08:16:52.598055 23893 rpcz_store.cc:276] 0502 08:16:51.087961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.401471 (+1313510us) service_pool.cc:225] Handling call
0502 08:16:52.401912 (+ 441us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.401947 (+ 35us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.513839 (+111892us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.580195 (+ 66356us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.597977 (+ 17782us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.598371 23893 tablet_service.cc:1467] Processing CreateTablet for tablet af5d13f956354acca041913011923840 (DEFAULT_TABLE table=default.Table_69664f97f411484e8277f8a0a02b6f90 [id=562481e7fd0c4211a090d3e382bfd64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.598651 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af5d13f956354acca041913011923840. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.600335 23723 leader_election.cc:302] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:52.601284 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c991463cf8ad4d3b8c0727e45c2317b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.607203 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4fe3a2f9eac469d9a7da7b9a2fc24a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.608984 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8075b03643a14a01ab75921fc3d3dace. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.616943 24173 tablet_bootstrap.cc:492] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:52.617385 24173 ts_tablet_manager.cc:1367] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.081s user 0.005s sys 0.005s
I20240502 08:16:52.621513 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 319fabf91458438ba5e40ab68b323055 (table default.Table_becb097785284ffba69cdbe40e6480ab [id=4454421c70094eed8a8f3e42dcc8a524]) successfully deleted
I20240502 08:16:52.623298 24173 raft_consensus.cc:348] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.626678 24173 raft_consensus.cc:374] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.626942 24173 raft_consensus.cc:729] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.628017 25326 raft_consensus.cc:482] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:52.628149 25326 raft_consensus.cc:504] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.628170 24173 consensus_queue.cc:260] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.628906 25326 leader_election.cc:288] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:52.629781 24173 ts_tablet_manager.cc:1398] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240502 08:16:52.630060 24173 tablet_bootstrap.cc:492] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:52.631373 24173 tablet_bootstrap.cc:654] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.632973 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_4c203e674ba54d1ba7d1e45ba8145077"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:52.633683 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c203e674ba54d1ba7d1e45ba8145077 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:52.643689 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 40fab016ed4f489aa3ccdeafcfbe1477 consensus state has no leader
I20240502 08:16:52.644241 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_15baf520893a4ad28df2c15c35649e15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:52.644970 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15baf520893a4ad28df2c15c35649e15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:52.646567 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_cea1e21aa3c3460aadbb92dd84a0acd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:52.647086 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cea1e21aa3c3460aadbb92dd84a0acd7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:52.648375 23605 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 77c49da1e0f0435ba241a6a66526e790 (table default.Table_8a9cdf1f53b0494694035cd549988bf0 [id=0de18f9c7e194e698b4f37603bfd4780]). Expected version 2 got 0
I20240502 08:16:52.648658 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_be653bb574ef4b5fba3f88ab616996be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:52.649082 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be653bb574ef4b5fba3f88ab616996be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:52.650233 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2d16819a129448d582a23d49efdaabd2 consensus state has no leader
I20240502 08:16:52.650549 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c9c50670737d40c58a305ff256334951" } new_table_name: "default.Table_5aa2066b846c4cc2b13f2cfa5c2dec4f" modify_external_catalogs: true
I20240502 08:16:52.653679 25266 raft_consensus.cc:2726] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:52.655268 23898 consensus_meta.cc:287] T 12bd0a3bf7d24d939288c3eca83e1540 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent flushing consensus metadata: real 0.578s user 0.003s sys 0.000s
I20240502 08:16:52.655642 23898 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 786) took 1571 ms. Trace:
I20240502 08:16:52.655716 23898 rpcz_store.cc:276] 0502 08:16:51.084288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.084819 (+ 531us) service_pool.cc:225] Handling call
0502 08:16:51.653952 (+569133us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.653983 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.077347 (+423364us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.655458 (+578111us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.655619 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.656127 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3570787d8242338d1ce01fb0299eb2 (DEFAULT_TABLE table=default.Table_d03c41e82d684920af0c093caa7cd1e6 [id=1a0ddcb26d794ee99068fa11c3b345e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.656519 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3570787d8242338d1ce01fb0299eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.656886 23996 leader_election.cc:302] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:52.657286 25326 raft_consensus.cc:2726] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:16:52.657812 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 7ecde716a0c643cdb0698a37cde1c906 (DEFAULT_TABLE table=default.Table_e00f744a28c6496e98f875d47bdccf95 [id=bdc6f08aa8d144b88f0612bcd8bd8ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.657943 24175 raft_consensus.cc:348] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.658107 24175 raft_consensus.cc:374] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.658154 24175 raft_consensus.cc:729] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.658334 24175 consensus_queue.cc:260] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.648658 23605 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 7ed919909289410e867881baab841f53 (table default.Table_2edad641920b44ce9a4d5b7503f0279a [id=ee3912407a934444ad2c62f7b83d93ac]). Expected version 2 got 0
I20240502 08:16:52.634824 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59c2f93d6514c9cbf358ab96f1a06b5" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:52.662874 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59c2f93d6514c9cbf358ab96f1a06b5" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:52.663233 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 9935d3cfff50430dbe8adff249d2c68e (DEFAULT_TABLE table=default.Table_9ff1ae77ef32484abb840fd10fcb88e0 [id=538a33bb4a7f4f1aa22866bf3c8698bb]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:52.663954 23993 leader_election.cc:341] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:52.664526 24174 ts_tablet_manager.cc:1367] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.314s user 0.009s sys 0.005s
I20240502 08:16:52.665184 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2c637e208044d8aaf1774f448e0527. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.669633 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 876d7dc6291f47e4b0e671b50f2fa8c7 (DEFAULT_TABLE table=default.Table_201d8955331641b4afc23044b3706aa7 [id=ad42fcf5cadb4f87988142f02096d703]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:52.671140 23723 raft_consensus.cc:2670] T c17e1be6ef294769a01981f4d361be21 P a297a6c7c2504ca68e87b24f6691ffb2: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:16:52.671407 23905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 794) took 1492 ms. Trace:
I20240502 08:16:52.671499 23905 rpcz_store.cc:276] 0502 08:16:51.085640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.088587 (+ 2947us) service_pool.cc:225] Handling call
0502 08:16:51.670449 (+581862us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.670471 (+ 22us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.354447 (+683976us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.551076 (+196629us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.577761 (+ 26685us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.671533 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 54f751261c414f8fae185c16c9636951 (DEFAULT_TABLE table=default.Table_efbec5d24d154d4ab7354736f7ef8fe5 [id=e88aff379cc341758c4b941e47261c33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.672017 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 43307fc65b3d437b85eab085a8ebe499 (DEFAULT_TABLE table=default.Table_682d60898e9545ceb4723b24bb852374 [id=bdcdb48a59d04fce89f635539505da93]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:52.672557 23996 leader_election.cc:341] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:52.672670 23996 leader_election.cc:302] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:52.673022 25326 raft_consensus.cc:2726] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:52.673530 23890 tablet_service.cc:1467] Processing CreateTablet for tablet d86d534538de4613bd148228a78e5e9a (DEFAULT_TABLE table=default.Table_b4905971ad3f4250a6b2e8f12c6ecd96 [id=5722267c6c5047fd99430504a3a5e06c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.677145 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 58f538766e934282932fd4ab41b49831 (DEFAULT_TABLE table=default.Table_4610f864dbdd41378b26279a24da77f9 [id=0357f7b063e64b88a93bf4bbaee37b99]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:52.677376 25322 log.cc:924] Time spent T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20240502 08:16:52.677603 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 518cb4b86ea34e95bef51c5c2a64d89b (DEFAULT_TABLE table=default.Table_787b2d1011974a8d95e16bf54ec92ffd [id=7ba6cc241eb4425f930294ee985c9ac6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.677034 24180 log.cc:1196] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/bd4a31e5aa834adeb9f59267247c6980
I20240502 08:16:52.677881 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9935d3cfff50430dbe8adff249d2c68e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.678126 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 106c39807ac14e7fba27fcec9f129642 (DEFAULT_TABLE table=default.Table_447f4a9a45334a60a8f4fbb914b1e546 [id=7740c6a602d64e17996e4cb4bc1554b0]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:52.678162 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 139007us
I20240502 08:16:52.677865 24174 raft_consensus.cc:348] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.682811 24174 raft_consensus.cc:374] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.682885 24174 raft_consensus.cc:729] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.683117 24174 consensus_queue.cc:260] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.683674 24174 ts_tablet_manager.cc:1398] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240502 08:16:52.683966 24174 tablet_bootstrap.cc:492] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:52.684103 25304 raft_consensus.cc:504] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.684775 23882 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:33614 (request call id 151) took 3893 ms. Trace:
I20240502 08:16:52.684870 23882 rpcz_store.cc:276] 0502 08:16:48.791356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.791437 (+ 81us) service_pool.cc:225] Handling call
0502 08:16:48.939804 (+148367us) spinlock_profiling.cc:243] Waited 71.9 ms on lock 0x6160000e5654. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:49.360912 (+421108us) spinlock_profiling.cc:243] Waited 420 ms on lock 0x616000135454. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:49.885306 (+524394us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x6160001d1a54. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:50.745342 (+860036us) spinlock_profiling.cc:243] Waited 859 ms on lock 0x6160000a5754. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:51.154277 (+408935us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x6160001f2154. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:51.774098 (+619821us) spinlock_profiling.cc:243] Waited 69.7 ms on lock 0x616000136c54. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:52.074845 (+300747us) spinlock_profiling.cc:243] Waited 35.4 ms on lock 0x6160001f2754. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:52.123044 (+ 48199us) spinlock_profiling.cc:243] Waited 47.9 ms on lock 0x6160000a6f54. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:52.684743 (+561699us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5562382080}
I20240502 08:16:52.685925 23885 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:33614 (request call id 152) took 3392 ms. Trace:
I20240502 08:16:52.686002 23885 rpcz_store.cc:276] 0502 08:16:49.293212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.294220 (+ 1008us) service_pool.cc:225] Handling call
0502 08:16:49.674777 (+380557us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x6160001f2d54. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:49.887248 (+212471us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x6160001d1a54. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:50.754427 (+867179us) spinlock_profiling.cc:243] Waited 866 ms on lock 0x6160000a5754. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:51.155025 (+400598us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x6160001f2154. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:51.751919 (+596894us) spinlock_profiling.cc:243] Waited 73 ms on lock 0x616000136c54. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:52.074175 (+322256us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x6160001f2754. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:52.122766 (+ 48591us) spinlock_profiling.cc:243] Waited 48.2 ms on lock 0x6160000a6f54. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:52.685889 (+563123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4129862912}
I20240502 08:16:52.690887 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 43307fc65b3d437b85eab085a8ebe499 (DEFAULT_TABLE table=default.Table_682d60898e9545ceb4723b24bb852374 [id=bdcdb48a59d04fce89f635539505da93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.691287 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43307fc65b3d437b85eab085a8ebe499. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.691399 23881 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:33614 (request call id 153) took 3398 ms. Trace:
I20240502 08:16:52.691511 23881 rpcz_store.cc:276] 0502 08:16:49.293294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:49.297628 (+ 4334us) service_pool.cc:225] Handling call
0502 08:16:49.683570 (+385942us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x6160001f2d54. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:49.884374 (+200804us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x6160001d1a54. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:50.750357 (+865983us) spinlock_profiling.cc:243] Waited 865 ms on lock 0x6160000a5754. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:51.154820 (+404463us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x6160001f2154. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:51.771784 (+616964us) spinlock_profiling.cc:243] Waited 58.4 ms on lock 0x616000136c54. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:52.076749 (+304965us) spinlock_profiling.cc:243] Waited 36.6 ms on lock 0x6160001f2754. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:52.129982 (+ 53233us) spinlock_profiling.cc:243] Waited 52.1 ms on lock 0x6160000a6f54. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:52.691372 (+561390us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4175181056}
I20240502 08:16:52.693915 25012 ts_tablet_manager.cc:1875] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:52.696554 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876d7dc6291f47e4b0e671b50f2fa8c7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:52.697021 23723 leader_election.cc:341] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:52.697144 23723 leader_election.cc:302] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:52.697130 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 518cb4b86ea34e95bef51c5c2a64d89b (DEFAULT_TABLE table=default.Table_787b2d1011974a8d95e16bf54ec92ffd [id=7ba6cc241eb4425f930294ee985c9ac6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.697460 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 518cb4b86ea34e95bef51c5c2a64d89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.697677 25301 raft_consensus.cc:2726] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:52.697746 23723 leader_election.cc:341] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:52.700050 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a19310948e248d1a85479c41c61bda2" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:52.700085 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad53633ce57478194437f95135d81de (DEFAULT_TABLE table=default.Table_9522d368c4c84f25bce3eb822a9bcad8 [id=e07e5990e2244361bcda7aa9108994b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.701027 25308 raft_consensus.cc:504] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.701465 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 876d7dc6291f47e4b0e671b50f2fa8c7 (DEFAULT_TABLE table=default.Table_201d8955331641b4afc23044b3706aa7 [id=ad42fcf5cadb4f87988142f02096d703]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.701871 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876d7dc6291f47e4b0e671b50f2fa8c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.705718 24180 ts_tablet_manager.cc:1909] T bd4a31e5aa834adeb9f59267247c6980 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:52.710741 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 54f751261c414f8fae185c16c9636951 (DEFAULT_TABLE table=default.Table_efbec5d24d154d4ab7354736f7ef8fe5 [id=e88aff379cc341758c4b941e47261c33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.711164 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f751261c414f8fae185c16c9636951. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.713017 23918 raft_consensus.cc:2454] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 2.
I20240502 08:16:52.713536 23996 leader_election.cc:302] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:52.713734 23996 raft_consensus.cc:2670] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:16:52.713903 24035 tablet_service.cc:1467] Processing CreateTablet for tablet fc87f91093b94b209caa6f89691bcde3 (DEFAULT_TABLE table=default.Table_fd05e15f7383485b9966bdf3ab0ab045 [id=9e1920400453498d8ff2245a70455c68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.714320 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc87f91093b94b209caa6f89691bcde3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.714610 25308 leader_election.cc:288] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:52.716178 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcc7169796e4ee0910f5e7084849012" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:52.716379 24053 raft_consensus.cc:3037] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:16:52.718680 24053 raft_consensus.cc:2454] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 2.
I20240502 08:16:52.719504 23861 leader_election.cc:302] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:52.719774 25308 raft_consensus.cc:2781] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:16:52.719894 25308 raft_consensus.cc:686] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [term 2 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:16:52.720139 25308 consensus_queue.cc:237] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:52.720678 23861 consensus_peers.cc:485] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 319fabf91458438ba5e40ab68b323055. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:52.721875 24181 ts_tablet_manager.cc:1888] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:52.721985 24181 log.cc:1196] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/df9402cf8abc4b4d893d42f9a3a99016
I20240502 08:16:52.722288 24181 ts_tablet_manager.cc:1909] T df9402cf8abc4b4d893d42f9a3a99016 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:52.722728 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 727) took 5619 ms. Trace:
I20240502 08:16:52.722823 24181 rpcz_store.cc:276] 0502 08:16:47.102955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.104753 (+ 1798us) service_pool.cc:225] Handling call
0502 08:16:52.547587 (+5442834us) ts_tablet_manager.cc:1120] Deleting tablet df9402cf8abc4b4d893d42f9a3a99016
0502 08:16:52.547596 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.561372 (+ 13776us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.721798 (+160426us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.722706 (+ 908us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5442648}
I20240502 08:16:52.723248 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 731) took 5524 ms. Trace:
I20240502 08:16:52.723307 24181 rpcz_store.cc:276] 0502 08:16:47.198793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.198845 (+ 52us) service_pool.cc:225] Handling call
0502 08:16:52.723156 (+5524311us) ts_tablet_manager.cc:1120] Deleting tablet ba3aa2621fdb43fd8299f81af486ef1e
0502 08:16:52.723163 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.723234 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5524118}
I20240502 08:16:52.723505 24181 tablet_replica.cc:330] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:52.723645 24181 raft_consensus.cc:2229] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:52.723801 24181 pending_rounds.cc:62] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5: Trying to abort 1 pending ops.
I20240502 08:16:52.723834 24181 pending_rounds.cc:68] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156454317977600 op_type: NO_OP noop_request { }
I20240502 08:16:52.723910 24181 raft_consensus.cc:2866] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:52.723994 24181 raft_consensus.cc:2258] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:52.724475 24181 ts_tablet_manager.cc:1875] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:52.722236 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 787) took 4538 ms. Trace:
I20240502 08:16:52.727596 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad53633ce57478194437f95135d81de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.729770 23887 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 819) took 1642 ms. Trace:
I20240502 08:16:52.729876 23887 rpcz_store.cc:276] 0502 08:16:51.087687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.341674 (+1253987us) service_pool.cc:225] Handling call
0502 08:16:52.342203 (+ 529us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.342243 (+ 40us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.538003 (+195760us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.660636 (+122633us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.729742 (+ 69106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.730383 23887 tablet_service.cc:1467] Processing CreateTablet for tablet d6e589e879f245378cff543fe430be5d (DEFAULT_TABLE table=default.Table_5fb8fa42fdb945729f5026cab15051b0 [id=323ecf6fa9a1474f869f2452d7b63da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.730777 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e589e879f245378cff543fe430be5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.731300 23900 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 820) took 1643 ms. Trace:
I20240502 08:16:52.731371 23900 rpcz_store.cc:276] 0502 08:16:51.087781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.388351 (+1300570us) service_pool.cc:225] Handling call
0502 08:16:52.388873 (+ 522us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.388905 (+ 32us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.574471 (+185566us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.661152 (+ 86681us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.731284 (+ 70132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.731814 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 37b499184f31454c96f66cfdf9143d14 (DEFAULT_TABLE table=default.Table_7a0267df98fe45a583a32684e35f9416 [id=953199c936cc4d859bc7ff49dc0c2ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.732147 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b499184f31454c96f66cfdf9143d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.732666 23904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 827) took 1634 ms. Trace:
I20240502 08:16:52.732731 23904 rpcz_store.cc:276] 0502 08:16:51.098672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.453847 (+1355175us) service_pool.cc:225] Handling call
0502 08:16:52.457708 (+ 3861us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.457744 (+ 36us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.582799 (+125055us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.606886 (+ 24087us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.732651 (+125765us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.733054 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 6a7ab5fc54834f5783f71e615bfc283a (DEFAULT_TABLE table=default.Table_e97c69e9998149d98b2954d150b8d220 [id=025b29b37a4f40539680de19b892cd15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.725219 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_47d67e47e5a24496aaa6876575d954f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:52.733295 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7ab5fc54834f5783f71e615bfc283a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.725157 24180 rpcz_store.cc:276] 0502 08:16:48.183683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.183820 (+ 137us) service_pool.cc:225] Handling call
0502 08:16:52.386585 (+4202765us) ts_tablet_manager.cc:1120] Deleting tablet bd4a31e5aa834adeb9f59267247c6980
0502 08:16:52.386596 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.426137 (+ 39541us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.589625 (+163488us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.721777 (+132152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4202505}
I20240502 08:16:52.735057 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet bd4a31e5aa834adeb9f59267247c6980 (table default.Table_4e40fa1cf0ab4dedb2c6aadb2418901b [id=fd781805eb4c45f09fe6460144472816]) successfully deleted
I20240502 08:16:52.735563 23891 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 800) took 1646 ms. Trace:
I20240502 08:16:52.735646 23891 rpcz_store.cc:276] 0502 08:16:51.086198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.088729 (+ 2531us) service_pool.cc:225] Handling call
0502 08:16:51.689363 (+600634us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.689400 (+ 37us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.655190 (+965790us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.678202 (+ 23012us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.732704 (+ 54502us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.737167 23888 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 834) took 1637 ms. Trace:
I20240502 08:16:52.737246 23888 rpcz_store.cc:276] 0502 08:16:51.099537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.594158 (+1494621us) service_pool.cc:225] Handling call
0502 08:16:52.594697 (+ 539us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.594717 (+ 20us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.728820 (+134103us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.737017 (+ 8197us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.737149 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.737608 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 39cc4ef1174a4358bad777b0471c5f76 (DEFAULT_TABLE table=default.Table_7952191ae36044e9aef62593bbe0bb27 [id=7125223862ae4d04b11a726f81b81073]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.737689 23896 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 803) took 1651 ms. Trace:
I20240502 08:16:52.737771 23896 rpcz_store.cc:276] 0502 08:16:51.086479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.649712 (+563233us) service_pool.cc:225] Handling call
0502 08:16:51.727938 (+ 78226us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:51.728076 (+ 138us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.684676 (+956600us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.687012 (+ 2336us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.737672 (+ 50660us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.737944 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39cc4ef1174a4358bad777b0471c5f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.738095 23896 tablet_service.cc:1467] Processing CreateTablet for tablet bb9b75d6be924d0291f94fc4a439dea6 (DEFAULT_TABLE table=default.Table_41d744e17dee458382533bb204fd1197 [id=e99a8a3a2c9343369d31c9852c31de9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.738380 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9b75d6be924d0291f94fc4a439dea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.738646 23891 tablet_service.cc:1467] Processing CreateTablet for tablet ce8dc203c36149cc91cc56176e929da6 (DEFAULT_TABLE table=default.Table_effefe18d3cd42f7855fed709d87de8c [id=596b181baa6140c0b44a9f900e9cbb5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.739042 23894 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 818) took 1651 ms. Trace:
I20240502 08:16:52.739125 23894 rpcz_store.cc:276] 0502 08:16:51.087624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.254591 (+1166967us) service_pool.cc:225] Handling call
0502 08:16:52.343236 (+ 88645us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.343254 (+ 18us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.483657 (+140403us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.696526 (+212869us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.739025 (+ 42499us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.739485 23897 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 808) took 1652 ms. Trace:
I20240502 08:16:52.739547 23897 rpcz_store.cc:276] 0502 08:16:51.086799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.928375 (+841576us) service_pool.cc:225] Handling call
0502 08:16:52.014903 (+ 86528us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.014962 (+ 59us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.484898 (+469936us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.594330 (+109432us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.739472 (+145142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.739966 23901 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 829) took 1640 ms. Trace:
I20240502 08:16:52.732775 23886 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 828) took 1634 ms. Trace:
I20240502 08:16:52.740339 23886 rpcz_store.cc:276] 0502 08:16:51.099045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.494166 (+1395121us) service_pool.cc:225] Handling call
0502 08:16:52.494623 (+ 457us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.494650 (+ 27us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.668050 (+173400us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.685815 (+ 17765us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.732767 (+ 46952us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.740821 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 2bb70ee9169046dd98e3f35ffcccdf3a (DEFAULT_TABLE table=default.Table_973ce52632594e5f9c980d6d7ae7371b [id=11d129fb7ef1439fad8010fee542e716]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.740900 23895 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 826) took 1652 ms. Trace:
I20240502 08:16:52.741218 23895 rpcz_store.cc:276] 0502 08:16:51.098169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.430665 (+1332496us) service_pool.cc:225] Handling call
0502 08:16:52.462924 (+ 32259us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.462978 (+ 54us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.691249 (+228271us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.700347 (+ 9098us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.740885 (+ 40538us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.742012 25012 ts_tablet_manager.cc:1888] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:52.742174 25012 log.cc:1196] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/4f1736b496354e2dadd844242d3867f8
I20240502 08:16:52.742370 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce8dc203c36149cc91cc56176e929da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.742676 23897 tablet_service.cc:1467] Processing CreateTablet for tablet b7339f1207924812b4ef2ac7ccffc894 (DEFAULT_TABLE table=default.Table_375988d4437044bc890604e6d5ab8af2 [id=6c14f79d3913425e9c0a0d0b524c1c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.742915 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d86d534538de4613bd148228a78e5e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.743842 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ecde716a0c643cdb0698a37cde1c906. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.741168 24180 tablet_replica.cc:330] T aacdf506221044ffada0fec0c7d42d0b P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:52.745803 23899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 815) took 1658 ms. Trace:
I20240502 08:16:52.745880 23899 rpcz_store.cc:276] 0502 08:16:51.087389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.146761 (+1059372us) service_pool.cc:225] Handling call
0502 08:16:52.150457 (+ 3696us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.150514 (+ 57us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.688550 (+538036us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.745644 (+ 57094us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.745786 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.746209 23899 tablet_service.cc:1467] Processing CreateTablet for tablet ba3b3abbb2e94c87ba19cc5695006e07 (DEFAULT_TABLE table=default.Table_39434b47dd9b421796ea5924f95292b9 [id=4c6173704b9e476fa4b836ba7fe5ff4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.746495 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3b3abbb2e94c87ba19cc5695006e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.746855 23903 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 831) took 1647 ms. Trace:
I20240502 08:16:52.746924 23903 rpcz_store.cc:276] 0502 08:16:51.099285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.586711 (+1487426us) service_pool.cc:225] Handling call
0502 08:16:52.596221 (+ 9510us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.596249 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.716778 (+120529us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.737535 (+ 20757us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.746839 (+ 9304us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.747457 23902 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 824) took 1659 ms. Trace:
I20240502 08:16:52.748035 23893 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 835) took 1648 ms. Trace:
W20240502 08:16:52.750006 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47d67e47e5a24496aaa6876575d954f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:52.752146 24180 raft_consensus.cc:2229] T aacdf506221044ffada0fec0c7d42d0b P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:52.756120 24180 raft_consensus.cc:2258] T aacdf506221044ffada0fec0c7d42d0b P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:52.764093 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet df9402cf8abc4b4d893d42f9a3a99016 (table default.Table_4b967ef2b2fc4e2caa4219afcae841a6 [id=c32fc96889554c32bdf1d4550633e91f]) successfully deleted
W20240502 08:16:52.765133 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet ba3aa2621fdb43fd8299f81af486ef1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba3aa2621fdb43fd8299f81af486ef1e already in progress: creating tablet
I20240502 08:16:52.772897 25266 raft_consensus.cc:482] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:52.773119 25266 raft_consensus.cc:504] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.773836 25266 leader_election.cc:288] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:52.775223 25326 raft_consensus.cc:482] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:52.775380 25326 raft_consensus.cc:504] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.748170 23904 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 840) took 1647 ms. Trace:
I20240502 08:16:52.748441 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7339f1207924812b4ef2ac7ccffc894. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.781149 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bb70ee9169046dd98e3f35ffcccdf3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.748589 23894 tablet_service.cc:1467] Processing CreateTablet for tablet d68448a2033e450794440fa84de6a19a (DEFAULT_TABLE table=default.Table_6a6b2f29d9aa4a679e4c41863e3096bd [id=38c7d40fae5b4b7eb187d2c8e43f04d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.740089 23901 rpcz_store.cc:276] 0502 08:16:51.099144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.566331 (+1467187us) service_pool.cc:225] Handling call
0502 08:16:52.584657 (+ 18326us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.584684 (+ 27us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.611211 (+ 26527us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.687012 (+ 75801us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.739952 (+ 52940us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18}
I20240502 08:16:52.781831 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet f5ccd43e78574f3288b199fa14bcc0e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:52.781884 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68448a2033e450794440fa84de6a19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.782006 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 642744194e22461f8316a78689fd99ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:52.782121 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet a0224317e65747e1add533d33472f7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:52.782234 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 654ce33d1e3d472eb6ae6a726eb803bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:52.782334 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet f797d3c6bdc4493398954fbbf9943971 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:52.782620 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 26c8325099af4d72ad00afb2316df885 (DEFAULT_TABLE table=default.Table_64b18a844e8d44ba8bca05606e5b34e8 [id=dbd1ab9190f94aebbdc794fd9f1fc074]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.782963 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c8325099af4d72ad00afb2316df885. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.783653 24180 ts_tablet_manager.cc:1875] T aacdf506221044ffada0fec0c7d42d0b P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:52.748867 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 858b23b8e12b467e86d2c914e63e8fa2 (DEFAULT_TABLE table=default.Table_6bab5c3f6452414eba871948cffb5714 [id=43b8fb37d8704742affd5ed4d1d4a287]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:52.750633 23858 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 262251us
I20240502 08:16:52.786470 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858b23b8e12b467e86d2c914e63e8fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.786497 23858 leader_election.cc:302] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:52.786758 25337 raft_consensus.cc:2726] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:52.787137 23858 leader_election.cc:302] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:52.747560 23902 rpcz_store.cc:276] 0502 08:16:51.088104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.413959 (+1325855us) service_pool.cc:225] Handling call
0502 08:16:52.479958 (+ 65999us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.479995 (+ 37us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.688272 (+208277us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.747327 (+ 59055us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.747431 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.787660 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc2270a5461448ca0245bd875f4245a (DEFAULT_TABLE table=default.Table_e29dfece3b88466ba6e6f6db0c1a31fd [id=7c9478b76f0a46c1869bd31c27a8eddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.788026 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc2270a5461448ca0245bd875f4245a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.788551 23894 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 844) took 1687 ms. Trace:
W20240502 08:16:52.788623 23594 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 2d16819a129448d582a23d49efdaabd2 (table default.Table_e91fa6b91a704e739fbfacbe1ce7ef29 [id=6261d1c2ceb745a5bd2fc6feda69d6f7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2d16819a129448d582a23d49efdaabd2 consensus state has no leader
I20240502 08:16:52.788638 23894 rpcz_store.cc:276] 0502 08:16:51.101347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.739293 (+1637946us) service_pool.cc:225] Handling call
0502 08:16:52.781735 (+ 42442us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.781765 (+ 30us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.784395 (+ 2630us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.786751 (+ 2356us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.788533 (+ 1782us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.791611 23897 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 847) took 1689 ms. Trace:
I20240502 08:16:52.791721 23897 rpcz_store.cc:276] 0502 08:16:51.101674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.742530 (+1640856us) service_pool.cc:225] Handling call
0502 08:16:52.742815 (+ 285us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.742834 (+ 19us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.781017 (+ 38183us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.791490 (+ 10473us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.791595 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.791949 23904 rpcz_store.cc:276] 0502 08:16:51.101063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.732907 (+1631844us) service_pool.cc:225] Handling call
0502 08:16:52.733179 (+ 272us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.733198 (+ 19us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.736231 (+ 3033us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.748050 (+ 11819us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.748158 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.792098 23895 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 846) took 1690 ms. Trace:
I20240502 08:16:52.752374 25012 ts_tablet_manager.cc:1909] T 4f1736b496354e2dadd844242d3867f8 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:52.753365 23890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 832) took 1654 ms. Trace:
I20240502 08:16:52.792307 23890 rpcz_store.cc:276] 0502 08:16:51.099359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.587459 (+1488100us) service_pool.cc:225] Handling call
0502 08:16:52.739704 (+152245us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.739734 (+ 30us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.745321 (+ 5587us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.753243 (+ 7922us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.753348 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.793062 25337 raft_consensus.cc:2726] T 96fcb895c8f449deb0190de75af18c3c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20240502 08:16:52.793587 23858 consensus_peers.cc:485] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8b1fa17dad143f0a7f1751e651eb81a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:52.793865 23858 leader_election.cc:302] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:52.794171 25337 raft_consensus.cc:2726] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:52.794668 23858 leader_election.cc:341] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 9c072eb45e1b41eeb5aafa6d5a786944
I20240502 08:16:52.794803 23858 leader_election.cc:302] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:52.795110 25337 raft_consensus.cc:2726] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20240502 08:16:52.795732 23858 leader_election.cc:341] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 319fabf91458438ba5e40ab68b323055
I20240502 08:16:52.796797 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 716) took 5089 ms. Trace:
I20240502 08:16:52.796953 25012 rpcz_store.cc:276] 0502 08:16:47.707232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.707276 (+ 44us) service_pool.cc:225] Handling call
0502 08:16:52.594605 (+4887329us) ts_tablet_manager.cc:1120] Deleting tablet 4f1736b496354e2dadd844242d3867f8
0502 08:16:52.594616 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.724208 (+129592us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.741893 (+ 17685us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.796777 (+ 54884us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3389,"spinlock_wait_cycles":5376,"tablet-delete.queue_time_us":4887209}
I20240502 08:16:52.797493 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 717) took 5089 ms. Trace:
I20240502 08:16:52.789517 23894 tablet_service.cc:1467] Processing CreateTablet for tablet e34873a3f08043098f2740641b3dc41f (DEFAULT_TABLE table=default.Table_a47e8e08a0ef4797aaf83cfa8adf0019 [id=630b2f4ab433442ab82b9451c8f54f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.797638 25012 rpcz_store.cc:276] 0502 08:16:47.707584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.707631 (+ 47us) service_pool.cc:225] Handling call
0502 08:16:52.797369 (+5089738us) ts_tablet_manager.cc:1120] Deleting tablet ba3aa2621fdb43fd8299f81af486ef1e
0502 08:16:52.797380 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.797477 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5089373}
I20240502 08:16:52.760890 23887 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 838) took 1661 ms. Trace:
I20240502 08:16:52.761130 23903 tablet_service.cc:1514] Processing DeleteTablet for tablet 642744194e22461f8316a78689fd99ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:52.798203 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 718) took 5090 ms. Trace:
I20240502 08:16:52.766904 23898 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 836) took 1667 ms. Trace:
I20240502 08:16:52.798409 23898 rpcz_store.cc:276] 0502 08:16:51.099690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.655901 (+1556211us) service_pool.cc:225] Handling call
0502 08:16:52.656303 (+ 402us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.656328 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.715443 (+ 59115us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.766787 (+ 51344us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.766888 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.798364 25012 rpcz_store.cc:276] 0502 08:16:47.708206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.708251 (+ 45us) service_pool.cc:225] Handling call
0502 08:16:52.798051 (+5089800us) ts_tablet_manager.cc:1120] Deleting tablet 202fa612a085450ca9c1225434cb7fe4
0502 08:16:52.798059 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.798187 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5089641}
I20240502 08:16:52.767220 23905 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 837) took 1667 ms. Trace:
I20240502 08:16:52.776772 23889 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 830) took 1648 ms. Trace:
I20240502 08:16:52.798875 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 719) took 5090 ms. Trace:
I20240502 08:16:52.792193 23895 rpcz_store.cc:276] 0502 08:16:51.101517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.741458 (+1639941us) service_pool.cc:225] Handling call
0502 08:16:52.786321 (+ 44863us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.786344 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.788978 (+ 2634us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.791960 (+ 2982us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.792086 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.756515 23892 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 833) took 1657 ms. Trace:
I20240502 08:16:52.799189 23892 rpcz_store.cc:276] 0502 08:16:51.099432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.593067 (+1493635us) service_pool.cc:225] Handling call
0502 08:16:52.739188 (+146121us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.739236 (+ 48us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.753868 (+ 14632us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.756333 (+ 2465us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.756478 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.761685 23900 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 839) took 1660 ms. Trace:
I20240502 08:16:52.799389 23858 leader_election.cc:302] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:52.799492 23900 rpcz_store.cc:276] 0502 08:16:51.100947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.731602 (+1630655us) service_pool.cc:225] Handling call
0502 08:16:52.731996 (+ 394us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.732020 (+ 24us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.743189 (+ 11169us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.761547 (+ 18358us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.761669 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.798938 23889 rpcz_store.cc:276] 0502 08:16:51.099232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.584548 (+1485316us) service_pool.cc:225] Handling call
0502 08:16:52.597454 (+ 12906us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.597473 (+ 19us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.705736 (+108263us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.721790 (+ 16054us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.748196 (+ 26406us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.797919 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e34873a3f08043098f2740641b3dc41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.799748 25337 raft_consensus.cc:2726] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:16:52.800334 23888 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 842) took 1699 ms. Trace:
I20240502 08:16:52.802814 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 106c39807ac14e7fba27fcec9f129642 (DEFAULT_TABLE table=default.Table_447f4a9a45334a60a8f4fbb914b1e546 [id=7740c6a602d64e17996e4cb4bc1554b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.804152 25326 leader_election.cc:288] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:52.805869 24371 leader_election.cc:334] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 84f3c07a31414c4f8fb61c44dc9f16e1 (127.22.225.125:36189): Timed out: RequestConsensusVote RPC to 127.22.225.125:36189 timed out after 0.746s (SENT)
I20240502 08:16:52.805866 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 58f538766e934282932fd4ab41b49831 (DEFAULT_TABLE table=default.Table_4610f864dbdd41378b26279a24da77f9 [id=0357f7b063e64b88a93bf4bbaee37b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.806041 24371 leader_election.cc:302] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcc8a323e6b9429eaaf4e0c1698ad077; no voters: 84f3c07a31414c4f8fb61c44dc9f16e1, f87df6a45ad94428a9aa6b2796a0fd25
I20240502 08:16:52.800819 23886 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 845) took 1699 ms. Trace:
I20240502 08:16:52.806790 23886 rpcz_store.cc:276] 0502 08:16:51.101433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.740614 (+1639181us) service_pool.cc:225] Handling call
0502 08:16:52.741092 (+ 478us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.741110 (+ 18us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.794668 (+ 53558us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.800660 (+ 5992us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.800800 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.807217 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106c39807ac14e7fba27fcec9f129642. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.798122 23887 rpcz_store.cc:276] 0502 08:16:51.099833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.730113 (+1630280us) service_pool.cc:225] Handling call
0502 08:16:52.730598 (+ 485us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.730628 (+ 30us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.758488 (+ 27860us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.760750 (+ 2262us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.760872 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.809532 23891 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 841) took 1708 ms. Trace:
I20240502 08:16:52.809641 23891 rpcz_store.cc:276] 0502 08:16:51.101132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.735824 (+1634692us) service_pool.cc:225] Handling call
0502 08:16:52.738779 (+ 2955us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.738798 (+ 19us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.781016 (+ 42218us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.809382 (+ 28366us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.809516 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.811812 23901 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 856) took 1104 ms. Trace:
I20240502 08:16:52.811913 23901 rpcz_store.cc:276] 0502 08:16:51.707358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.782403 (+1075045us) service_pool.cc:225] Handling call
0502 08:16:52.782796 (+ 393us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.782819 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.785257 (+ 2438us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.811641 (+ 26384us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.811790 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.812719 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58f538766e934282932fd4ab41b49831. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.817426 25317 consensus_queue.cc:1035] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:52.819507 23996 consensus_peers.cc:485] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:52.821339 23896 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 843) took 1720 ms. Trace:
I20240502 08:16:52.828158 23758 tablet_service.cc:1467] Processing CreateTablet for tablet fc87f91093b94b209caa6f89691bcde3 (DEFAULT_TABLE table=default.Table_fd05e15f7383485b9966bdf3ab0ab045 [id=9e1920400453498d8ff2245a70455c68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.837857 25304 leader_election.cc:288] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:52.841730 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba1baa2279045b3adc58d187a12cbf3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:52.841899 24053 raft_consensus.cc:3037] T 1ba1baa2279045b3adc58d187a12cbf3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:52.842474 23723 leader_election.cc:341] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:52.843111 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577b149f52e44d5d9a9f6b9d85a6864d" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:52.843143 23723 leader_election.cc:302] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:52.843439 25296 consensus_queue.cc:1035] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.145s
I20240502 08:16:52.843734 25158 raft_consensus.cc:2726] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:52.843889 25301 consensus_queue.cc:1035] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:52.841114 23785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7b37f776df47da9aa6e7fba82a8999" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:52.844194 23785 raft_consensus.cc:2454] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:52.844563 23993 leader_election.cc:302] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:52.844916 25326 raft_consensus.cc:2781] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:52.845002 25326 raft_consensus.cc:482] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:52.845094 25326 raft_consensus.cc:3037] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.798825 23905 rpcz_store.cc:276] 0502 08:16:51.099767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.671735 (+1571968us) service_pool.cc:225] Handling call
0502 08:16:52.700337 (+ 28602us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.727430 (+ 27093us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.737224 (+ 9794us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.767079 (+ 29855us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.767205 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.801680 23893 rpcz_store.cc:276] 0502 08:16:51.099612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.598220 (+1498608us) service_pool.cc:225] Handling call
0502 08:16:52.598518 (+ 298us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.598540 (+ 22us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.680523 (+ 81983us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.712250 (+ 31727us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.748018 (+ 35768us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.799044 25012 rpcz_store.cc:276] 0502 08:16:47.708603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.708645 (+ 42us) service_pool.cc:225] Handling call
0502 08:16:52.798744 (+5090099us) ts_tablet_manager.cc:1120] Deleting tablet 4baa2bc2752c450fa212471c150cf098
0502 08:16:52.798755 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.798859 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5089918}
I20240502 08:16:52.847887 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 720) took 5138 ms. Trace:
I20240502 08:16:52.848016 25012 rpcz_store.cc:276] 0502 08:16:47.709076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.709132 (+ 56us) service_pool.cc:225] Handling call
0502 08:16:52.847744 (+5138612us) ts_tablet_manager.cc:1120] Deleting tablet d36e7cd33f9a4edabe690e75602590b9
0502 08:16:52.847757 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.847866 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5138440}
I20240502 08:16:52.848441 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 705) took 5144 ms. Trace:
I20240502 08:16:52.848554 25012 rpcz_store.cc:276] 0502 08:16:47.704048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.709522 (+ 5474us) service_pool.cc:225] Handling call
0502 08:16:52.848288 (+5138766us) ts_tablet_manager.cc:1120] Deleting tablet 25c8a6aaabe2413d8ce88f54ce45a4c4
0502 08:16:52.848316 (+ 28us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.848425 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5138523}
I20240502 08:16:52.846046 25241 consensus_queue.cc:1046] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20240502 08:16:52.848982 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642744194e22461f8316a78689fd99ff" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:52.849340 25012 tablet_replica.cc:330] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:52.849073 25241 raft_consensus.cc:3032] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:16:52.858970 25241 raft_consensus.cc:729] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:52.823045 23902 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 857) took 1115 ms. Trace:
I20240502 08:16:52.859892 23902 rpcz_store.cc:276] 0502 08:16:51.707443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.787426 (+1079983us) service_pool.cc:225] Handling call
0502 08:16:52.787847 (+ 421us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.787874 (+ 27us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.791170 (+ 3296us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.822909 (+ 31739us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.823025 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.849720 25012 raft_consensus.cc:2229] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:52.805326 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7b37f776df47da9aa6e7fba82a8999" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:52.801239 23888 rpcz_store.cc:276] 0502 08:16:51.101187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.737407 (+1636220us) service_pool.cc:225] Handling call
0502 08:16:52.737785 (+ 378us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.737808 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.742196 (+ 4388us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.784086 (+ 41890us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.786249 (+ 2163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.803342 23858 leader_election.cc:302] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:52.862329 23996 leader_election.cc:341] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:52.863068 24371 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.125:36189, user_credentials={real_user=slave}} blocked reactor thread for 57259.7us
I20240502 08:16:52.867076 23894 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 858) took 1159 ms. Trace:
I20240502 08:16:52.867264 23894 rpcz_store.cc:276] 0502 08:16:51.707535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.788895 (+1081360us) service_pool.cc:225] Handling call
0502 08:16:52.797768 (+ 8873us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.797791 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.814347 (+ 16556us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.862060 (+ 47713us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.867040 (+ 4980us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.821417 23896 rpcz_store.cc:276] 0502 08:16:51.101259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.737924 (+1636665us) service_pool.cc:225] Handling call
0502 08:16:52.738254 (+ 330us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.738274 (+ 20us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.794676 (+ 56402us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.821215 (+ 26539us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.821321 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.871934 25301 consensus_queue.cc:1035] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:52.872757 23723 leader_election.cc:341] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:52.873716 25345 tablet.cc:1722] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:52.878219 25301 consensus_queue.cc:1035] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:52.879834 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 9935d3cfff50430dbe8adff249d2c68e (DEFAULT_TABLE table=default.Table_9ff1ae77ef32484abb840fd10fcb88e0 [id=538a33bb4a7f4f1aa22866bf3c8698bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.880098 25301 consensus_queue.cc:1035] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:52.880265 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9935d3cfff50430dbe8adff249d2c68e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.882994 25012 raft_consensus.cc:2258] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:52.882915 25326 raft_consensus.cc:504] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.883682 25012 ts_tablet_manager.cc:1875] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:52.884136 25337 raft_consensus.cc:2726] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:52.884832 23858 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 81951.2us
I20240502 08:16:52.885244 25337 consensus_queue.cc:1035] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:52.886066 23858 leader_election.cc:302] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:52.886379 25337 raft_consensus.cc:2726] T f4b291bdb81a4efbac7f6691e5577bbe P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:52.886878 23858 leader_election.cc:302] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:52.891265 24053 raft_consensus.cc:2454] T 1ba1baa2279045b3adc58d187a12cbf3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:52.895140 23858 leader_election.cc:302] T 36f2d7b6f4e442ae95b2d1002d571b6a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:52.895606 23858 leader_election.cc:302] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:52.895769 25337 raft_consensus.cc:2726] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:52.900818 25241 consensus_queue.cc:260] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.901278 25241 raft_consensus.cc:3037] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:16:52.912269 25326 leader_election.cc:288] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:52.914994 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7b37f776df47da9aa6e7fba82a8999" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:52.915845 23996 leader_election.cc:341] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:52.923153 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7b37f776df47da9aa6e7fba82a8999" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:52.925138 23788 raft_consensus.cc:3037] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.937419 25337 raft_consensus.cc:2726] T 36f2d7b6f4e442ae95b2d1002d571b6a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:52.937626 25337 raft_consensus.cc:2726] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:16:52.937898 23993 consensus_peers.cc:485] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:52.938138 25346 tablet.cc:1722] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:52.939635 23899 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33620 (request call id 848) took 1837 ms. Trace:
I20240502 08:16:52.939790 23899 rpcz_store.cc:276] 0502 08:16:51.101767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.746044 (+1644277us) service_pool.cc:225] Handling call
0502 08:16:52.746355 (+ 311us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:52.746377 (+ 22us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:52.797218 (+ 50841us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.939346 (+142128us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:52.939610 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:52.944412 23917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.22.225.65:36563 (request call id 236) took 1030 ms. Trace:
I20240502 08:16:52.953963 23723 leader_election.cc:302] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:52.955039 24181 ts_tablet_manager.cc:1888] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:52.955183 24181 log.cc:1196] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/27346a7eae6a488fa3415646262a4b88
I20240502 08:16:52.955560 24181 ts_tablet_manager.cc:1909] T 27346a7eae6a488fa3415646262a4b88 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:52.955888 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 729) took 5784 ms. Trace:
I20240502 08:16:52.955973 24181 rpcz_store.cc:276] 0502 08:16:47.171088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.217774 (+ 46686us) service_pool.cc:225] Handling call
0502 08:16:52.723471 (+5505697us) ts_tablet_manager.cc:1120] Deleting tablet 27346a7eae6a488fa3415646262a4b88
0502 08:16:52.723478 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.819120 (+ 95642us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.954934 (+135814us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.955866 (+ 932us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5505319}
I20240502 08:16:52.956404 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 732) took 5701 ms. Trace:
I20240502 08:16:52.956499 24181 rpcz_store.cc:276] 0502 08:16:47.254912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.254996 (+ 84us) service_pool.cc:225] Handling call
0502 08:16:52.956270 (+5701274us) ts_tablet_manager.cc:1120] Deleting tablet 202fa612a085450ca9c1225434cb7fe4
0502 08:16:52.956280 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.956384 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5701020}
I20240502 08:16:52.956791 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 733) took 5674 ms. Trace:
I20240502 08:16:52.956848 24181 rpcz_store.cc:276] 0502 08:16:47.282422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.284246 (+ 1824us) service_pool.cc:225] Handling call
0502 08:16:52.956702 (+5672456us) ts_tablet_manager.cc:1120] Deleting tablet 4baa2bc2752c450fa212471c150cf098
0502 08:16:52.956710 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.956779 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5672163}
I20240502 08:16:52.957101 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 728) took 5806 ms. Trace:
I20240502 08:16:52.957157 24181 rpcz_store.cc:276] 0502 08:16:47.151149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.286019 (+134870us) service_pool.cc:225] Handling call
0502 08:16:52.957017 (+5670998us) ts_tablet_manager.cc:1120] Deleting tablet bb4b4a4d6a364de8af3b54d99a0ae138
0502 08:16:52.957026 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.957088 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5670741}
I20240502 08:16:52.957904 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 730) took 5777 ms. Trace:
I20240502 08:16:52.958035 24181 rpcz_store.cc:276] 0502 08:16:47.180372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.292495 (+112123us) service_pool.cc:225] Handling call
0502 08:16:52.957809 (+5665314us) ts_tablet_manager.cc:1120] Deleting tablet 48d96e42813f4169969dd5931e169923
0502 08:16:52.957817 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.957890 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5665029}
I20240502 08:16:52.958388 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 734) took 5638 ms. Trace:
I20240502 08:16:52.958516 24181 rpcz_store.cc:276] 0502 08:16:47.320289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.320395 (+ 106us) service_pool.cc:225] Handling call
0502 08:16:52.958299 (+5637904us) ts_tablet_manager.cc:1120] Deleting tablet d36e7cd33f9a4edabe690e75602590b9
0502 08:16:52.958307 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.958375 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5631826}
I20240502 08:16:52.958868 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 735) took 5138 ms. Trace:
I20240502 08:16:52.958977 24181 rpcz_store.cc:276] 0502 08:16:47.820325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.823255 (+ 2930us) service_pool.cc:225] Handling call
0502 08:16:52.958776 (+5135521us) ts_tablet_manager.cc:1120] Deleting tablet 13cbab4352bb44f389d70d4379e92457
0502 08:16:52.958784 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.958854 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5135169}
I20240502 08:16:52.959317 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 736) took 5138 ms. Trace:
I20240502 08:16:52.959445 24181 rpcz_store.cc:276] 0502 08:16:47.821372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.824386 (+ 3014us) service_pool.cc:225] Handling call
0502 08:16:52.959224 (+5134838us) ts_tablet_manager.cc:1120] Deleting tablet 21d2db2158d4447f92eedc5780b46056
0502 08:16:52.959234 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.959304 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5134539}
I20240502 08:16:52.959769 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 737) took 5120 ms. Trace:
I20240502 08:16:52.959877 24181 rpcz_store.cc:276] 0502 08:16:47.839431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.839525 (+ 94us) service_pool.cc:225] Handling call
0502 08:16:52.959677 (+5120152us) ts_tablet_manager.cc:1120] Deleting tablet 0ae92276246c4d39b194d7e9b868fc02
0502 08:16:52.959686 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.959753 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5119901}
I20240502 08:16:52.960213 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 738) took 5058 ms. Trace:
I20240502 08:16:52.960343 24181 rpcz_store.cc:276] 0502 08:16:47.902089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.902175 (+ 86us) service_pool.cc:225] Handling call
0502 08:16:52.960125 (+5057950us) ts_tablet_manager.cc:1120] Deleting tablet 21d2db2158d4447f92eedc5780b46056
0502 08:16:52.960132 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.960200 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5057715}
I20240502 08:16:52.960691 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 740) took 5036 ms. Trace:
I20240502 08:16:52.960808 24181 rpcz_store.cc:276] 0502 08:16:47.924085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.924996 (+ 911us) service_pool.cc:225] Handling call
0502 08:16:52.960605 (+5035609us) ts_tablet_manager.cc:1120] Deleting tablet 40964817cf2a4712be42f40e37639bc9
0502 08:16:52.960613 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.960679 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5035397}
I20240502 08:16:52.961128 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 741) took 5036 ms. Trace:
I20240502 08:16:52.961232 24181 rpcz_store.cc:276] 0502 08:16:47.924201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.925235 (+ 1034us) service_pool.cc:225] Handling call
0502 08:16:52.961043 (+5035808us) ts_tablet_manager.cc:1120] Deleting tablet 4a0684db8dd947d79343afa5a2090614
0502 08:16:52.961051 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.961116 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5035705}
I20240502 08:16:52.961601 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 742) took 5037 ms. Trace:
I20240502 08:16:52.961716 24181 rpcz_store.cc:276] 0502 08:16:47.924284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.925372 (+ 1088us) service_pool.cc:225] Handling call
0502 08:16:52.961495 (+5036123us) ts_tablet_manager.cc:1120] Deleting tablet a4d151770ee44130adfd822518b9b017
0502 08:16:52.961505 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.961584 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5035989}
I20240502 08:16:52.962049 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 744) took 4949 ms. Trace:
I20240502 08:16:52.962168 24181 rpcz_store.cc:276] 0502 08:16:48.012871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.012933 (+ 62us) service_pool.cc:225] Handling call
0502 08:16:52.961961 (+4949028us) ts_tablet_manager.cc:1120] Deleting tablet 28a947b7e7484120b3f4069f6eef5520
0502 08:16:52.961968 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.962035 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4948819}
I20240502 08:16:52.962479 24181 tablet_replica.cc:330] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:52.967805 25296 raft_consensus.cc:2781] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:52.968103 25296 raft_consensus.cc:686] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:52.968396 25296 consensus_queue.cc:237] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:52.970818 25012 ts_tablet_manager.cc:1888] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:52.970963 25012 log.cc:1196] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/e686db13bb7541c69ae88f0fb7e27061
I20240502 08:16:52.971293 25227 consensus_queue.cc:1035] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:52.972034 24173 tablet_bootstrap.cc:492] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:52.972568 24173 ts_tablet_manager.cc:1367] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.343s user 0.006s sys 0.003s
I20240502 08:16:52.978197 25012 ts_tablet_manager.cc:1909] T e686db13bb7541c69ae88f0fb7e27061 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:52.978477 24175 ts_tablet_manager.cc:1398] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.622s user 0.007s sys 0.019s
W20240502 08:16:52.978616 24175 ts_tablet_manager.cc:1430] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b: Tablet startup took 622ms
I20240502 08:16:52.980176 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 27346a7eae6a488fa3415646262a4b88 (table default.Table_5d5022ea23c34e4f960c366d5ac96c39 [id=999c7344afec4d3c858dc29a986a8426]) successfully deleted
I20240502 08:16:52.980386 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 721) took 5270 ms. Trace:
I20240502 08:16:52.980556 25012 rpcz_store.cc:276] 0502 08:16:47.709480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.732536 (+ 23056us) service_pool.cc:225] Handling call
0502 08:16:52.849286 (+5116750us) ts_tablet_manager.cc:1120] Deleting tablet e686db13bb7541c69ae88f0fb7e27061
0502 08:16:52.849297 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:52.950435 (+101138us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.970721 (+ 20286us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:52.980365 (+ 9644us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5116314}
W20240502 08:16:52.980634 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 202fa612a085450ca9c1225434cb7fe4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 202fa612a085450ca9c1225434cb7fe4 already in progress: creating tablet
I20240502 08:16:52.980895 25012 tablet_replica.cc:330] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
W20240502 08:16:52.980995 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 4baa2bc2752c450fa212471c150cf098 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4baa2bc2752c450fa212471c150cf098 already in progress: creating tablet
I20240502 08:16:52.981108 25012 raft_consensus.cc:2229] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:52.981221 25012 raft_consensus.cc:2258] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:52.981555 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet bb4b4a4d6a364de8af3b54d99a0ae138 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb4b4a4d6a364de8af3b54d99a0ae138 already in progress: creating tablet
I20240502 08:16:52.981704 25012 ts_tablet_manager.cc:1875] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:52.944506 23917 rpcz_store.cc:276] 0502 08:16:51.914245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.914309 (+ 64us) service_pool.cc:225] Handling call
0502 08:16:51.933396 (+ 19087us) raft_consensus.cc:1457] Updating replica for 2 ops
0502 08:16:51.933592 (+ 196us) raft_consensus.cc:1520] Early marking committed up to index 0
0502 08:16:51.933597 (+ 5us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0502 08:16:51.934101 (+ 504us) alter_schema_op.cc:119] START. Timestamp: 1714637808186112
0502 08:16:52.240729 (+306628us) log.cc:842] Serialized 199 byte log entry
0502 08:16:52.721060 (+480331us) raft_consensus.cc:1631] Marking committed up to 2
0502 08:16:52.721208 (+ 148us) log.cc:842] Serialized 14 byte log entry
0502 08:16:52.892911 (+171703us) raft_consensus.cc:1683] Filling consensus response to leader.
0502 08:16:52.896014 (+ 3103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0502 08:16:52.896026 (+ 12us) raft_consensus.cc:1671] finished
0502 08:16:52.896028 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0502 08:16:52.944339 (+ 48311us) spinlock_profiling.cc:243] Waited 18.9 ms on lock 0x616000164b50. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e616f9 00007fb204b4010d 00007fb1ff03bb56 00007fb1ff03b885 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:52.944388 (+ 49us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:52.225863 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0502 08:16:52.225990 (+ 127us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0502 08:16:52.226000 (+ 10us) alter_schema_op.cc:59] Acquired schema lock
0502 08:16:52.226007 (+ 7us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0502 08:16:52.721512 (+495505us) op_driver.cc:444] REPLICATION: finished
0502 08:16:52.873675 (+152163us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0502 08:16:52.874205 (+ 530us) log.cc:842] Serialized 5 byte log entry
0502 08:16:52.921332 (+ 47127us) tablet_metadata.cc:641] Metadata flushed
Metrics: {"prepare.queue_time_us":36,"prepare.run_cpu_time_us":10,"prepare.run_wall_time_us":10,"raft.queue_time_us":40965,"raft.run_cpu_time_us":66,"raft.run_wall_time_us":67,"spinlock_wait_cycles":43417984,"thread_start_us":480107,"threads_started":1,"wal-append.queue_time_us":476959,"child_traces":[["op",{"apply.queue_time_us":152099,"apply.run_cpu_time_us":4529,"apply.run_wall_time_us":77872,"prepare.queue_time_us":291658,"prepare.run_cpu_time_us":168,"prepare.run_wall_time_us":170,"replication_time_us":787404,"thread_start_us":474067,"threads_started":2}]]}
W20240502 08:16:52.978725 24175 ts_tablet_manager.cc:1432] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b: Trace:
0502 08:16:38.709297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:38.709346 (+ 49us) service_pool.cc:225] Handling call
0502 08:16:38.709835 (+ 489us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:38.709879 (+ 44us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:38.713202 (+ 3323us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:38.745361 (+ 32159us) tablet_replica.cc:205] Creating consensus instance
0502 08:16:38.745582 (+ 221us) inbound_call.cc:173] Queueing success response
0502 08:16:51.711656 (+12966074us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0502 08:16:52.356474 (+644818us) ts_tablet_manager.cc:1399] Starting tablet replica
0502 08:16:52.657376 (+300902us) tablet_replica.cc:259] Starting instrumentation
0502 08:16:52.657546 (+ 170us) tablet_replica.cc:278] Starting consensus
Metrics: {"raft.queue_time_us":58347,"raft.run_cpu_time_us":40,"raft.run_wall_time_us":39,"tablet-open.queue_time_us":12966105,"thread_start_us":319626,"threads_started":1}
I20240502 08:16:52.984762 24175 tablet_bootstrap.cc:492] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:52.988148 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 48d96e42813f4169969dd5931e169923 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48d96e42813f4169969dd5931e169923 already in progress: creating tablet
W20240502 08:16:52.988658 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet d36e7cd33f9a4edabe690e75602590b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d36e7cd33f9a4edabe690e75602590b9 already in progress: creating tablet
W20240502 08:16:52.988834 23858 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 93241.6us
W20240502 08:16:52.989120 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 13cbab4352bb44f389d70d4379e92457 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13cbab4352bb44f389d70d4379e92457 already in progress: creating tablet
I20240502 08:16:52.989086 23858 leader_election.cc:302] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:52.989706 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet f685cbe705b149e69f9536e270a74a42 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:50 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:52.989987 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet fe5e08ad6af2442bb0a8fcc80a99b532 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:50 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:52.990054 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 303a0a3748ee422faab856e1c4bff66d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:49 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:52.990446 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet 29f6c3cea4b14364b6d073a92171119d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:50 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:52.990780 23899 tablet_service.cc:1467] Processing CreateTablet for tablet f5496fa258de4988b9502136f0baf779 (DEFAULT_TABLE table=default.Table_33c693cfb89641a2921c7d39872b5bbc [id=68e36934ceb0416a86dc24c4fc50804b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.991194 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5496fa258de4988b9502136f0baf779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.991338 23788 raft_consensus.cc:2454] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:52.991796 23993 leader_election.cc:302] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:52.992100 25297 raft_consensus.cc:2781] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:52.992342 25297 raft_consensus.cc:686] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
W20240502 08:16:52.992744 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 21d2db2158d4447f92eedc5780b46056 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21d2db2158d4447f92eedc5780b46056 already in progress: creating tablet
I20240502 08:16:52.992785 25297 consensus_queue.cc:237] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:52.993976 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 0ae92276246c4d39b194d7e9b868fc02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ae92276246c4d39b194d7e9b868fc02 already in progress: creating tablet
W20240502 08:16:52.995054 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 21d2db2158d4447f92eedc5780b46056 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21d2db2158d4447f92eedc5780b46056 already in progress: creating tablet
W20240502 08:16:52.996254 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 40964817cf2a4712be42f40e37639bc9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40964817cf2a4712be42f40e37639bc9 already in progress: creating tablet
W20240502 08:16:52.997355 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 4a0684db8dd947d79343afa5a2090614 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a0684db8dd947d79343afa5a2090614 already in progress: creating tablet
I20240502 08:16:52.998262 23894 tablet_service.cc:1467] Processing CreateTablet for tablet fa3d945d3897423a9c21018b25434fd1 (DEFAULT_TABLE table=default.Table_f14ecd9121174b1db356ecd80bbac528 [id=5550792d0bc942f38dd39b94e642c6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.998374 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc133adb9a44cc7b56a3746d1eb76af (DEFAULT_TABLE table=default.Table_d20a3b8ecb6449a5a4cc9837354619b7 [id=1c25911cce0845ffb760a96926714b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.998662 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa3d945d3897423a9c21018b25434fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.998850 23888 tablet_service.cc:1467] Processing CreateTablet for tablet a4fe3a2f9eac469d9a7da7b9a2fc24a8 (DEFAULT_TABLE table=default.Table_dd5271576d9746048fa96f022073b8fd [id=4640f69b48d44ee3baa1b6d4ab6fd6bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.999164 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4fe3a2f9eac469d9a7da7b9a2fc24a8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:52.999429 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet a4d151770ee44130adfd822518b9b017 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4d151770ee44130adfd822518b9b017 already in progress: creating tablet
I20240502 08:16:52.999804 25354 tablet.cc:1722] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
W20240502 08:16:53.001039 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 28a947b7e7484120b3f4069f6eef5520 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28a947b7e7484120b3f4069f6eef5520 already in progress: creating tablet
I20240502 08:16:53.002959 25347 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bcc8a323e6b9429eaaf4e0c1698ad077 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:53.008824 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 521581fe87e547ce816b599578bddeea (DEFAULT_TABLE table=default.Table_d67c17e0dfd94a088bd9991625f3c31a [id=bd341da28438429895d79410c12dce3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.010716 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 521581fe87e547ce816b599578bddeea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.012502 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc133adb9a44cc7b56a3746d1eb76af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.019295 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 8075b03643a14a01ab75921fc3d3dace
I20240502 08:16:53.019909 23606 master_service.cc:946] ReplaceTablet: received request to replace tablet 2ebf55c8024c483f9ef0afe6945a02ff from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:53.020112 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet 8075b03643a14a01ab75921fc3d3dace with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:52 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.020421 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 77c49da1e0f0435ba241a6a66526e790
I20240502 08:16:53.021080 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet 77c49da1e0f0435ba241a6a66526e790 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:52 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.021057 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9aa2ee2fdc9b4b91996a406f42e5c440" } new_table_name: "default.Table_b960fd6e0faf418f94cb45fd181cf7ef" modify_external_catalogs: true
I20240502 08:16:53.023393 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b23dfc41a96043ed98c03260ce4d0d32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.023944 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b23dfc41a96043ed98c03260ce4d0d32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.025240 24022 tablet_service.cc:1514] Processing DeleteTablet for tablet 77c49da1e0f0435ba241a6a66526e790 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:52 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:53.025673 24022 tablet_service.cc:1514] Processing DeleteTablet for tablet dd3078cc1f6247609f8bd017ed076b45 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:53.025848 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 8075b03643a14a01ab75921fc3d3dace with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:52 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:53.026952 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1626382a260044eb9100b8048332abb0 consensus state has no leader
I20240502 08:16:53.027434 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c2b1aa8d133542ffb45da2eacecf427b" } new_table_name: "default.Table_224b3429692a4ba1a1ac71d94089886b" modify_external_catalogs: true
I20240502 08:16:53.028657 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 07344b78894c42809391d0fe2a9cd8f7
I20240502 08:16:53.029281 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b2b68fb4c3c14505b960a7ec8623b0e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.029745 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2b68fb4c3c14505b960a7ec8623b0e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.030982 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet 07344b78894c42809391d0fe2a9cd8f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:52 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.031047 24022 tablet_service.cc:1514] Processing DeleteTablet for tablet 07344b78894c42809391d0fe2a9cd8f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:52 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:53.031219 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f02efa659e457ab57bc32ed3f66bb9 with delete_type TABLET_DATA_DELETED (replaced by tablet df06e0edb57f45e5ac71b568c14769cd) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:53.032096 23601 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet b811b81ddf2b4dc9b5e5f80331bff294 (table default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418]). Expected version 2 got 0
I20240502 08:16:53.032135 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_08a0e708a9ad4c619295ddc6c62e1a83" } modify_external_catalogs: true
I20240502 08:16:53.032306 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_08a0e708a9ad4c619295ddc6c62e1a83" } modify_external_catalogs: true
I20240502 08:16:53.033322 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 1cd8338500f54604abcd6d42702c2eca
I20240502 08:16:53.034020 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8cbe6e485a474bdba481239cfa55ee8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.034456 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cbe6e485a474bdba481239cfa55ee8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.035148 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d67c17e0dfd94a088bd9991625f3c31a" } new_table_name: "default.Table_8e4b36ac2e7b41559cb0941df14e55a1" modify_external_catalogs: true
I20240502 08:16:53.036747 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cd8338500f54604abcd6d42702c2eca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:52 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.037501 25359 tablet.cc:1722] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240502 08:16:53.046103 23603 catalog_manager.cc:5484] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
W20240502 08:16:53.050086 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90d4e2f23c9649a4b1dfb6a179ca7be5 consensus state has no leader
I20240502 08:16:53.052138 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 2ebf55c8024c483f9ef0afe6945a02ff
I20240502 08:16:53.052533 23606 catalog_manager.cc:6554] ReplaceTablet: tablet 2ebf55c8024c483f9ef0afe6945a02ff deleted and replaced by tablet ed5916a2229545b4ba13515219d459cb
I20240502 08:16:53.052894 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8fd21899dd67434986e4e8fb3b799ce1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.053359 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fd21899dd67434986e4e8fb3b799ce1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.053992 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ebf55c8024c483f9ef0afe6945a02ff with delete_type TABLET_DATA_DELETED (replaced by tablet ed5916a2229545b4ba13515219d459cb) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.056569 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_efe189f09a734489854ea6184b7ce522"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.056993 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_efe189f09a734489854ea6184b7ce522 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.061707 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b62ee1a1ff144e1d9447b5fb6ee38831" } new_table_name: "default.Table_5f3b7adae72c466fa28ec5465a266bcc" modify_external_catalogs: true
I20240502 08:16:53.063686 24181 raft_consensus.cc:2229] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:53.064102 24181 pending_rounds.cc:62] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5: Trying to abort 1 pending ops.
I20240502 08:16:53.064205 24181 pending_rounds.cc:68] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156476883591168 op_type: NO_OP noop_request { }
I20240502 08:16:53.064376 24181 raft_consensus.cc:2866] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:53.064538 24181 raft_consensus.cc:2258] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:53.065191 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 8075b03643a14a01ab75921fc3d3dace (DEFAULT_TABLE table=default.Table_fabad8a6e72a4724893dfa4c0877fdb2 [id=ea894a4bd467422eb135ad75e7032193]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.065770 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8075b03643a14a01ab75921fc3d3dace. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:53.069026 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 397003e47c78418182e0167c3b876d4e consensus state has no leader
I20240502 08:16:53.069152 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cd8338500f54604abcd6d42702c2eca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:52 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:53.070216 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet 2b98cc8728c042f4b7f6496fcc5ef913 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:53.070272 24022 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ebf55c8024c483f9ef0afe6945a02ff with delete_type TABLET_DATA_DELETED (replaced by tablet ed5916a2229545b4ba13515219d459cb) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:53.071974 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 33bbebbca7464ea6ad59b81d7b7c560d
I20240502 08:16:53.073019 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet 33bbebbca7464ea6ad59b81d7b7c560d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.069460 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_077eda1e02b342df838221680e40fc2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.074540 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_077eda1e02b342df838221680e40fc2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:53.076236 23723 consensus_peers.cc:485] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:53.076704 23607 master_service.cc:946] ReplaceTablet: received request to replace tablet 45f02efa659e457ab57bc32ed3f66bb9 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:53.078876 24022 tablet_service.cc:1514] Processing DeleteTablet for tablet 33bbebbca7464ea6ad59b81d7b7c560d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:53.079046 25357 raft_consensus.cc:2726] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:53.079957 23888 tablet_service.cc:1514] Processing DeleteTablet for tablet 87a3df4fbb224dd0b4ec6141584d374b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:51 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.080112 23894 tablet_service.cc:1467] Processing CreateTablet for tablet c991463cf8ad4d3b8c0727e45c2317b4 (DEFAULT_TABLE table=default.Table_1014ad7e82a245ea801b1866b929b1c5 [id=39339ca66938431c8de38927672a9f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.080585 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c991463cf8ad4d3b8c0727e45c2317b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.083479 23888 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3c00e5393c47df924bcb5dcf2a0f19 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:51 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.083817 23888 tablet_service.cc:1514] Processing DeleteTablet for tablet d2bb4dba23ac4b4eb53b3f2d75d0442e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:52 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.084592 23888 tablet_service.cc:1514] Processing DeleteTablet for tablet 2085ced52fff46ebaa98324cbb2d8c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:51 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.086864 23888 tablet_service.cc:1514] Processing DeleteTablet for tablet ee3d91d55ffd49ceb20d2c33408f5a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.090387 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 517354df966c4338a01a425251cccdd2 (DEFAULT_TABLE table=default.Table_1fa44e7d541041d3a86bed4be93b5a03 [id=4e5acee37ab44e979a690a2df5a78d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.092188 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517354df966c4338a01a425251cccdd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.093736 24181 ts_tablet_manager.cc:1875] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:53.095999 25297 consensus_queue.cc:1035] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:53.107254 24173 raft_consensus.cc:348] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.107473 24173 raft_consensus.cc:374] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.107518 24173 raft_consensus.cc:729] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.107704 24173 consensus_queue.cc:260] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.108191 24173 ts_tablet_manager.cc:1398] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.135s user 0.001s sys 0.001s
W20240502 08:16:53.112921 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 521581fe87e547ce816b599578bddeea consensus state has no leader
W20240502 08:16:53.114796 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f82e7b6c450c456b8547dd7c2ffc5c9a has no consensus state
I20240502 08:16:53.124640 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet 955e7bd2216646c6aef5acce8d5b0d77 with delete_type TABLET_DATA_DELETED (replaced by tablet 4645143a00d440a7b6d56c5e30d249f1) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.127620 25296 raft_consensus.cc:482] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:53.127943 25296 raft_consensus.cc:504] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.129812 25296 leader_election.cc:288] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:53.136229 25297 consensus_queue.cc:1035] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:53.137364 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet 418ab502915c4e4ea9998dd2d0180c48 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:53.138338 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c8131846dc41678827b86e87be18d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.138846 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc87f91093b94b209caa6f89691bcde3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.139037 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet fbef606ce7ac499d9bd84b49d804621c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.141228 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106c39807ac14e7fba27fcec9f129642. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.145813 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58f538766e934282932fd4ab41b49831. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.147331 24174 tablet_bootstrap.cc:654] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.149861 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 518cb4b86ea34e95bef51c5c2a64d89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.154296 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f751261c414f8fae185c16c9636951. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.157503 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43307fc65b3d437b85eab085a8ebe499. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.162838 25354 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:43544 (request call id 918) took 1228 ms. Trace:
I20240502 08:16:53.165710 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet c5c377e037894728ac084b6500ad20de with delete_type TABLET_DATA_DELETED (replaced by tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.165849 23898 tablet_service.cc:1467] Processing CreateTablet for tablet b1a2bcc6937c4a95b71103dbf8738702 (DEFAULT_TABLE table=default.Table_fc86e7c02217467c9a9e596e55aeee08 [id=a87a2af04b464ab283571bbc57ead567]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.166191 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet c48cea94856f43b6861c3ba360414888 with delete_type TABLET_DATA_DELETED (replaced by tablet e34873a3f08043098f2740641b3dc41f) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.166345 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a2bcc6937c4a95b71103dbf8738702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.166666 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet e2126bf2d4ab44628b52481dc2ec3a99 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:49 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.167454 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2aa20459484e4b98e0ae372418020f (DEFAULT_TABLE table=default.Table_ff75818c766d43569443cacbfcb972c1 [id=22d38df685d2423da2228076975133a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.167845 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b2aa20459484e4b98e0ae372418020f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.170081 25297 raft_consensus.cc:482] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:53.170228 25297 raft_consensus.cc:504] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.170841 25297 leader_election.cc:288] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:53.171468 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751808dfc1964b539f6f836d786156aa" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:53.171978 23996 leader_election.cc:341] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:53.172463 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751808dfc1964b539f6f836d786156aa" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:53.172636 23787 raft_consensus.cc:2454] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:53.172978 23993 leader_election.cc:302] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:53.173333 25297 raft_consensus.cc:2781] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:53.173405 25297 raft_consensus.cc:482] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:53.173463 25297 raft_consensus.cc:3037] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:53.174944 24175 tablet_bootstrap.cc:654] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.175956 25297 raft_consensus.cc:504] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.176548 25297 leader_election.cc:288] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:53.177188 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751808dfc1964b539f6f836d786156aa" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:53.177734 23996 leader_election.cc:341] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:53.178572 24174 tablet_bootstrap.cc:492] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:53.178689 24174 ts_tablet_manager.cc:1367] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.495s user 0.007s sys 0.029s
I20240502 08:16:53.177075 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "751808dfc1964b539f6f836d786156aa" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:53.179541 23787 raft_consensus.cc:3037] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:53.163003 25354 rpcz_store.cc:276] 0502 08:16:51.933956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.934953 (+ 997us) service_pool.cc:225] Handling call
0502 08:16:53.162813 (+1227860us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:16:52.025337 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0502 08:16:52.025454 (+ 117us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0502 08:16:52.025467 (+ 13us) alter_schema_op.cc:59] Acquired schema lock
0502 08:16:52.025477 (+ 10us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0502 08:16:52.025528 (+ 51us) alter_schema_op.cc:119] START. Timestamp: 1714637812025495
0502 08:16:52.025531 (+ 3us) op_driver.cc:331] REPLICATION: starting
0502 08:16:52.025813 (+ 282us) log.cc:842] Serialized 173 byte log entry
0502 08:16:52.870070 (+844257us) op_driver.cc:444] REPLICATION: finished
0502 08:16:52.999770 (+129700us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0502 08:16:53.145894 (+146124us) log.cc:842] Serialized 5 byte log entry
0502 08:16:53.162396 (+ 16502us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:53.162720 (+ 324us) log.cc:842] Serialized 14 byte log entry
0502 08:16:53.162773 (+ 53us) alter_schema_op.cc:65] Released schema lock
0502 08:16:53.162778 (+ 5us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":49617,"prepare.queue_time_us":89901,"prepare.run_cpu_time_us":2506,"prepare.run_wall_time_us":69815,"replication_time_us":844515,"thread_start_us":215977,"threads_started":4,"wal-append.queue_time_us":106242,"wal-append.run_cpu_time_us":559,"wal-append.run_wall_time_us":1001428}]]}
I20240502 08:16:53.181636 24174 raft_consensus.cc:348] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.181807 24174 raft_consensus.cc:374] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.181859 24174 raft_consensus.cc:729] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.182073 24174 consensus_queue.cc:260] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.182555 24174 ts_tablet_manager.cc:1398] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:53.198387 24022 tablet_service.cc:1467] Processing CreateTablet for tablet b55f0c7949dc4a83bbca9d80765e3fab (DEFAULT_TABLE table=default.Table_4c203e674ba54d1ba7d1e45ba8145077 [id=6df290af1822451692fc7272bc6cae58]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.199306 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 61f247661946448e9a69eb8dfd3b29a9 (DEFAULT_TABLE table=default.Table_5e9c513ec4ac4635bf68b84207bfce51 [id=dfbcd63f45df4a45bfb4d5380a193836]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.199586 24026 tablet_service.cc:1467] Processing CreateTablet for tablet e6815b2b3f79412fb04debc5d71f214c (DEFAULT_TABLE table=default.Table_e67c0ee30255468aa0674964089b483e [id=8d71ccbbc5ab41dcb45188399a7ce396]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.200287 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ba1baa2279045b3adc58d187a12cbf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.201092 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 58f538766e934282932fd4ab41b49831 (DEFAULT_TABLE table=default.Table_4610f864dbdd41378b26279a24da77f9 [id=0357f7b063e64b88a93bf4bbaee37b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.201520 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58f538766e934282932fd4ab41b49831. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.203022 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 050c16739efb4037a4adc78a2ffd3cba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:52 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.206238 24031 tablet_service.cc:1467] Processing CreateTablet for tablet f3312ac30e73492f88c0edb602343952 (DEFAULT_TABLE table=default.Table_a6c2b028992644d987b8a3ca62b1d325 [id=8d63c8f5e8c849629d90d5081e64bac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.207782 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 418ab502915c4e4ea9998dd2d0180c48
I20240502 08:16:53.208338 23602 catalog_manager.cc:6554] ReplaceTablet: tablet 418ab502915c4e4ea9998dd2d0180c48 deleted and replaced by tablet 89fefde55e514e5fabae3f4e21f44ca3
I20240502 08:16:53.209017 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet 418ab502915c4e4ea9998dd2d0180c48 with delete_type TABLET_DATA_DELETED (replaced by tablet 89fefde55e514e5fabae3f4e21f44ca3) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:53.209318 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6746c25111d04f20a6f30ce0838d71f5" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:53.209371 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 1daa7305e0054fbb971864ad8449c33f (DEFAULT_TABLE table=default.Table_1e7d21d372ec4ef9bd840ff21bbea400 [id=ea12c5554a364ebb9f7cde1f7d29bf11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.209924 24173 tablet_bootstrap.cc:492] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:53.209962 23723 leader_election.cc:341] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:53.212430 24173 tablet_bootstrap.cc:654] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.212833 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 7209bd5b466e4f3a8ab31704b1e83f0f (DEFAULT_TABLE table=default.Table_efdf6a98cfe94b879b0959eccdbf3dac [id=6ae69bfd2fb84c5ea034b58138bac5d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.213129 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1daa7305e0054fbb971864ad8449c33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.213349 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7209bd5b466e4f3a8ab31704b1e83f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.213575 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6815b2b3f79412fb04debc5d71f214c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.217911 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55f0c7949dc4a83bbca9d80765e3fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.229063 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f247661946448e9a69eb8dfd3b29a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.229276 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 876d7dc6291f47e4b0e671b50f2fa8c7 (DEFAULT_TABLE table=default.Table_201d8955331641b4afc23044b3706aa7 [id=ad42fcf5cadb4f87988142f02096d703]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.229552 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 518cb4b86ea34e95bef51c5c2a64d89b (DEFAULT_TABLE table=default.Table_787b2d1011974a8d95e16bf54ec92ffd [id=7ba6cc241eb4425f930294ee985c9ac6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.229990 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 54f751261c414f8fae185c16c9636951 (DEFAULT_TABLE table=default.Table_efbec5d24d154d4ab7354736f7ef8fe5 [id=e88aff379cc341758c4b941e47261c33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.234153 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 106c39807ac14e7fba27fcec9f129642 (DEFAULT_TABLE table=default.Table_447f4a9a45334a60a8f4fbb914b1e546 [id=7740c6a602d64e17996e4cb4bc1554b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.235069 23893 tablet_service.cc:1467] Processing CreateTablet for tablet fc87f91093b94b209caa6f89691bcde3 (DEFAULT_TABLE table=default.Table_fd05e15f7383485b9966bdf3ab0ab045 [id=9e1920400453498d8ff2245a70455c68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.235916 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 8075b03643a14a01ab75921fc3d3dace with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:52 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.236541 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3312ac30e73492f88c0edb602343952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.237061 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 77c49da1e0f0435ba241a6a66526e790 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:52 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.238173 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 07344b78894c42809391d0fe2a9cd8f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:52 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.239497 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 711a38d7c47749f8a7f59e52b81b51ee (DEFAULT_TABLE table=default.Table_6a3a4cab06a545d2b1af63c5bb0e695a [id=de7696830453467990b30118453a1d78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.240324 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711a38d7c47749f8a7f59e52b81b51ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.243019 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2df607ff8993422187123671e2dcede3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.243602 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2df607ff8993422187123671e2dcede3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.244439 23758 tablet_service.cc:1467] Processing CreateTablet for tablet b55f0c7949dc4a83bbca9d80765e3fab (DEFAULT_TABLE table=default.Table_4c203e674ba54d1ba7d1e45ba8145077 [id=6df290af1822451692fc7272bc6cae58]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.244889 24174 tablet_bootstrap.cc:492] T dd3078cc1f6247609f8bd017ed076b45 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:53.245700 23787 raft_consensus.cc:2454] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:53.246134 23763 tablet_service.cc:1467] Processing CreateTablet for tablet f3312ac30e73492f88c0edb602343952 (DEFAULT_TABLE table=default.Table_a6c2b028992644d987b8a3ca62b1d325 [id=8d63c8f5e8c849629d90d5081e64bac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.246351 23755 tablet_service.cc:1467] Processing CreateTablet for tablet e6815b2b3f79412fb04debc5d71f214c (DEFAULT_TABLE table=default.Table_e67c0ee30255468aa0674964089b483e [id=8d71ccbbc5ab41dcb45188399a7ce396]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.246611 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet 418ab502915c4e4ea9998dd2d0180c48 with delete_type TABLET_DATA_DELETED (replaced by tablet 89fefde55e514e5fabae3f4e21f44ca3) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.248382 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3312ac30e73492f88c0edb602343952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.248914 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 61f247661946448e9a69eb8dfd3b29a9 (DEFAULT_TABLE table=default.Table_5e9c513ec4ac4635bf68b84207bfce51 [id=dfbcd63f45df4a45bfb4d5380a193836]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.248971 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_346dec12764b4564b56f28b373f24b0b" } new_table_name: "default.Table_0f71ff280bad45758c41f3056333ed38" modify_external_catalogs: true
I20240502 08:16:53.249260 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f247661946448e9a69eb8dfd3b29a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.249605 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ff158afbc2b3466d9f6027b544a797dd" } modify_external_catalogs: true
I20240502 08:16:53.249794 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 1daa7305e0054fbb971864ad8449c33f (DEFAULT_TABLE table=default.Table_1e7d21d372ec4ef9bd840ff21bbea400 [id=ea12c5554a364ebb9f7cde1f7d29bf11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.250074 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1daa7305e0054fbb971864ad8449c33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.250211 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b13bfe8c518a4f27b9c2006d6f35b5c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.250711 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b13bfe8c518a4f27b9c2006d6f35b5c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.251271 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_efe189f09a734489854ea6184b7ce522" } modify_external_catalogs: true
I20240502 08:16:53.251806 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_213f4bea9fad44c4a1aa726aed361575" } modify_external_catalogs: true
I20240502 08:16:53.252133 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_077eda1e02b342df838221680e40fc2b" } modify_external_catalogs: true
I20240502 08:16:53.253405 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 7209bd5b466e4f3a8ab31704b1e83f0f (DEFAULT_TABLE table=default.Table_efdf6a98cfe94b879b0959eccdbf3dac [id=6ae69bfd2fb84c5ea034b58138bac5d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.253099 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c19c35e94c9748758a471eeb0896e0a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:53.253710 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7209bd5b466e4f3a8ab31704b1e83f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.253790 23993 leader_election.cc:302] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:53.254268 25297 raft_consensus.cc:2781] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:53.254400 25297 raft_consensus.cc:686] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
W20240502 08:16:53.254411 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c19c35e94c9748758a471eeb0896e0a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.254707 24174 tablet_bootstrap.cc:654] T dd3078cc1f6247609f8bd017ed076b45 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.254644 25297 consensus_queue.cc:237] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.256590 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55f0c7949dc4a83bbca9d80765e3fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.256829 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6815b2b3f79412fb04debc5d71f214c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.257992 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 43307fc65b3d437b85eab085a8ebe499 (DEFAULT_TABLE table=default.Table_682d60898e9545ceb4723b24bb852374 [id=bdcdb48a59d04fce89f635539505da93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.259039 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cd8338500f54604abcd6d42702c2eca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:52 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.260219 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ebf55c8024c483f9ef0afe6945a02ff with delete_type TABLET_DATA_DELETED (replaced by tablet ed5916a2229545b4ba13515219d459cb) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.261322 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 33bbebbca7464ea6ad59b81d7b7c560d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.262385 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e7432f1df94431a83cf68356e3e78a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.264909 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ff158afbc2b3466d9f6027b544a797dd" } modify_external_catalogs: true
I20240502 08:16:53.265833 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_efe189f09a734489854ea6184b7ce522" } modify_external_catalogs: true
I20240502 08:16:53.267668 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_213f4bea9fad44c4a1aa726aed361575" } modify_external_catalogs: true
I20240502 08:16:53.268904 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_077eda1e02b342df838221680e40fc2b" } modify_external_catalogs: true
I20240502 08:16:53.269992 23905 tablet_service.cc:1467] Processing CreateTablet for tablet b55f0c7949dc4a83bbca9d80765e3fab (DEFAULT_TABLE table=default.Table_4c203e674ba54d1ba7d1e45ba8145077 [id=6df290af1822451692fc7272bc6cae58]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:53.283771 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d609a55e47ad420098d279835e9aa090 consensus state has no leader
I20240502 08:16:53.284288 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b960fd6e0faf418f94cb45fd181cf7ef" } new_table_name: "default.Table_b2e1c08bfc404c7ebb2720631ad10495" modify_external_catalogs: true
I20240502 08:16:53.293550 24024 tablet_service.cc:1467] Processing CreateTablet for tablet f82e7b6c450c456b8547dd7c2ffc5c9a (DEFAULT_TABLE table=default.Table_5f3b7adae72c466fa28ec5465a266bcc [id=646be2f6f121446a838aa9a400095781]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.294533 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f82e7b6c450c456b8547dd7c2ffc5c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.296716 23609 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 216bf14d11a1470399a1577e9970163e
I20240502 08:16:53.297122 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fc9f3ca03af3472d9628fdfe64a89450"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.297694 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc9f3ca03af3472d9628fdfe64a89450 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.297742 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9 (DEFAULT_TABLE table=default.Table_0b5edfb1bf6e470f97bbbb62d09e353f [id=c79cdf3f081f4e42a01d425cc46ba015]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.298360 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.298501 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9 (DEFAULT_TABLE table=default.Table_0b5edfb1bf6e470f97bbbb62d09e353f [id=c79cdf3f081f4e42a01d425cc46ba015]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.299144 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 40fab016ed4f489aa3ccdeafcfbe1477
I20240502 08:16:53.299602 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.299674 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 552e46f69230474189278911cfeec6c7 (DEFAULT_TABLE table=default.Table_562886123f0548bfab83210c1c45b552 [id=91d11e967c41466daa2971eace82b289]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.299978 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 552e46f69230474189278911cfeec6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.299888 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7d7936b041894d9090bff1b6341088fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.300463 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d7936b041894d9090bff1b6341088fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.302130 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f96549ae28214964b5530d879c8f04b6" } new_table_name: "default.Table_55a033a602c24586bc4d6d660749a0fe" modify_external_catalogs: true
I20240502 08:16:53.302233 24022 tablet_service.cc:1467] Processing CreateTablet for tablet eaea443ae9044b76b48b3b5f92a9ae7e (DEFAULT_TABLE table=default.Table_b5566be23d3744348216cab2e9a0c6bf [id=6fd6e744b27b4a6fbfc1fab435a68c7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.302877 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaea443ae9044b76b48b3b5f92a9ae7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.303267 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 711a38d7c47749f8a7f59e52b81b51ee (DEFAULT_TABLE table=default.Table_6a3a4cab06a545d2b1af63c5bb0e695a [id=de7696830453467990b30118453a1d78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.303591 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711a38d7c47749f8a7f59e52b81b51ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.303814 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet df87b8b1ff504031bf007880792192dd
I20240502 08:16:53.304611 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_4cc1a3bc9a5c4574958aaa494a0dad5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.305073 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4cc1a3bc9a5c4574958aaa494a0dad5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.306455 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0cd4a9137f44f2be21056164a35afc (DEFAULT_TABLE table=default.Table_89e8be3491994ad4adc3c32d36c23d27 [id=9fa5384988334b57b1be8ddbb6f0c1c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.306670 23753 tablet_service.cc:1467] Processing CreateTablet for tablet f82e7b6c450c456b8547dd7c2ffc5c9a (DEFAULT_TABLE table=default.Table_5f3b7adae72c466fa28ec5465a266bcc [id=646be2f6f121446a838aa9a400095781]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.305548 23765 tablet_service.cc:1467] Processing CreateTablet for tablet eaea443ae9044b76b48b3b5f92a9ae7e (DEFAULT_TABLE table=default.Table_b5566be23d3744348216cab2e9a0c6bf [id=6fd6e744b27b4a6fbfc1fab435a68c7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.307235 23610 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 19ac958fbbaa4dc0b994aa83c215c1b9
I20240502 08:16:53.307268 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0cd4a9137f44f2be21056164a35afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.307586 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6a3a4cab06a545d2b1af63c5bb0e695a" } modify_external_catalogs: true
I20240502 08:16:53.307760 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6a3a4cab06a545d2b1af63c5bb0e695a" } modify_external_catalogs: true
I20240502 08:16:53.310737 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_cd64f661d15b4fddbd06f615b876bfc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.311249 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd64f661d15b4fddbd06f615b876bfc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.263119 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 9935d3cfff50430dbe8adff249d2c68e (DEFAULT_TABLE table=default.Table_9ff1ae77ef32484abb840fd10fcb88e0 [id=538a33bb4a7f4f1aa22866bf3c8698bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.312815 23603 master_service.cc:946] ReplaceTablet: received request to replace tablet 23ba091d8efb44858bf90a104c0f442f from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:53.316298 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7ee63691fc8240ad9fd8db92d778eb3c" } new_table_name: "default.Table_b638d47286da4c038bf21e1fda876cff" modify_external_catalogs: true
I20240502 08:16:53.318316 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 552e46f69230474189278911cfeec6c7 (DEFAULT_TABLE table=default.Table_562886123f0548bfab83210c1c45b552 [id=91d11e967c41466daa2971eace82b289]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.319136 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 552e46f69230474189278911cfeec6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.321067 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet 40fab016ed4f489aa3ccdeafcfbe1477 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:53.321271 24173 tablet_bootstrap.cc:492] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:53.321421 24173 ts_tablet_manager.cc:1367] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.112s user 0.002s sys 0.002s
I20240502 08:16:53.322036 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet df87b8b1ff504031bf007880792192dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:53.322551 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_655c02f28fab4cfaacd923e876e91078"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.326435 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_655c02f28fab4cfaacd923e876e91078 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.327677 24173 raft_consensus.cc:348] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.327909 24173 raft_consensus.cc:374] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.327980 24173 raft_consensus.cc:729] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.328101 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 968a301fd817452a8b4552e959d2e371 (DEFAULT_TABLE table=default.Table_b673548cd88d4f51975f7433f1509cc9 [id=909d7a034f194299ac30c21aa666d2a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.328194 24173 consensus_queue.cc:260] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.328483 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968a301fd817452a8b4552e959d2e371. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.328743 24173 ts_tablet_manager.cc:1398] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240502 08:16:53.329026 24173 tablet_bootstrap.cc:492] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:53.334890 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 61f247661946448e9a69eb8dfd3b29a9 (DEFAULT_TABLE table=default.Table_5e9c513ec4ac4635bf68b84207bfce51 [id=dfbcd63f45df4a45bfb4d5380a193836]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.336616 24173 tablet_bootstrap.cc:654] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.336568 24036 tablet_service.cc:1467] Processing CreateTablet for tablet f9f29ed2608a4966ba878ad2dae45975 (DEFAULT_TABLE table=default.Table_fbc10d796e98456ea3d7899ece9dcfc4 [id=e640a7b706b64dc5a6de4ba14584cee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.338815 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f29ed2608a4966ba878ad2dae45975. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.342253 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1364eb9eb640bdb315f6e2aa24be48 (DEFAULT_TABLE table=default.Table_444fa6f47a2d4a21b14a426d5e90b33b [id=4650b53969234e2e8267990ec8228fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.342610 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1364eb9eb640bdb315f6e2aa24be48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.346946 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876d7dc6291f47e4b0e671b50f2fa8c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.347198 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 518cb4b86ea34e95bef51c5c2a64d89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.348906 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f247661946448e9a69eb8dfd3b29a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.357719 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106c39807ac14e7fba27fcec9f129642. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.358588 24173 tablet_bootstrap.cc:492] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:53.358721 24173 ts_tablet_manager.cc:1367] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20240502 08:16:53.359598 24173 raft_consensus.cc:348] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.359782 24173 raft_consensus.cc:374] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.359824 24173 raft_consensus.cc:729] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.360023 24173 consensus_queue.cc:260] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.360805 24173 ts_tablet_manager.cc:1398] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:53.361069 24173 tablet_bootstrap.cc:492] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:53.361855 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc87f91093b94b209caa6f89691bcde3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.362356 24173 tablet_bootstrap.cc:654] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.366055 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55f0c7949dc4a83bbca9d80765e3fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.366861 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f751261c414f8fae185c16c9636951. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.376889 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet 40fab016ed4f489aa3ccdeafcfbe1477 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.377350 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0cd4a9137f44f2be21056164a35afc (DEFAULT_TABLE table=default.Table_89e8be3491994ad4adc3c32d36c23d27 [id=9fa5384988334b57b1be8ddbb6f0c1c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.377925 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 968a301fd817452a8b4552e959d2e371 (DEFAULT_TABLE table=default.Table_b673548cd88d4f51975f7433f1509cc9 [id=909d7a034f194299ac30c21aa666d2a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.378645 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 577b149f52e44d5d9a9f6b9d85a6864d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.379559 23758 tablet_service.cc:1467] Processing CreateTablet for tablet f9f29ed2608a4966ba878ad2dae45975 (DEFAULT_TABLE table=default.Table_fbc10d796e98456ea3d7899ece9dcfc4 [id=e640a7b706b64dc5a6de4ba14584cee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.384032 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f82e7b6c450c456b8547dd7c2ffc5c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.384907 25241 consensus_queue.cc:1035] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:53.388103 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet df87b8b1ff504031bf007880792192dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.390401 24173 tablet_bootstrap.cc:492] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:53.390630 24173 ts_tablet_manager.cc:1367] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.003s
I20240502 08:16:53.391727 24180 ts_tablet_manager.cc:1888] T aacdf506221044ffada0fec0c7d42d0b P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:53.391816 24180 log.cc:1196] T aacdf506221044ffada0fec0c7d42d0b P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/aacdf506221044ffada0fec0c7d42d0b
I20240502 08:16:53.393878 24180 ts_tablet_manager.cc:1909] T aacdf506221044ffada0fec0c7d42d0b P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:53.396059 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1364eb9eb640bdb315f6e2aa24be48 (DEFAULT_TABLE table=default.Table_444fa6f47a2d4a21b14a426d5e90b33b [id=4650b53969234e2e8267990ec8228fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.397099 24173 raft_consensus.cc:348] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.397357 24173 raft_consensus.cc:374] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.399608 24173 raft_consensus.cc:729] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.399921 24173 consensus_queue.cc:260] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.404683 24173 ts_tablet_manager.cc:1398] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.014s user 0.000s sys 0.001s
I20240502 08:16:53.405009 24173 tablet_bootstrap.cc:492] T 1da01e2c2f054ccfb53383e70e16da84 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:53.410789 24175 tablet_bootstrap.cc:492] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:53.411001 24175 ts_tablet_manager.cc:1367] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.426s user 0.005s sys 0.001s
I20240502 08:16:53.411746 24175 raft_consensus.cc:348] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.412027 24175 raft_consensus.cc:374] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.412118 24175 raft_consensus.cc:729] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.412487 24175 consensus_queue.cc:260] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.413301 24175 ts_tablet_manager.cc:1398] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:53.414345 24174 tablet_bootstrap.cc:492] T dd3078cc1f6247609f8bd017ed076b45 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:53.414546 24174 ts_tablet_manager.cc:1367] T dd3078cc1f6247609f8bd017ed076b45 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.170s user 0.005s sys 0.004s
I20240502 08:16:53.415221 24174 raft_consensus.cc:348] T dd3078cc1f6247609f8bd017ed076b45 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.415521 24174 raft_consensus.cc:374] T dd3078cc1f6247609f8bd017ed076b45 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.415627 24174 raft_consensus.cc:729] T dd3078cc1f6247609f8bd017ed076b45 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
W20240502 08:16:53.415956 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90d4e2f23c9649a4b1dfb6a179ca7be5 consensus state has no leader
I20240502 08:16:53.416008 24174 consensus_queue.cc:260] T dd3078cc1f6247609f8bd017ed076b45 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.416667 24174 ts_tablet_manager.cc:1398] T dd3078cc1f6247609f8bd017ed076b45 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:53.416507 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d5b106e54e3b4396ac55b919e56ba322"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:53.416677 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_444fa6f47a2d4a21b14a426d5e90b33b" } new_table_name: "default.Table_c627152fc8054febb0250aeabeb08124" modify_external_catalogs: true
W20240502 08:16:53.417112 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5b106e54e3b4396ac55b919e56ba322 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.417742 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43307fc65b3d437b85eab085a8ebe499. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.418311 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_606b0e91907c4c5384fab7ba4980f048"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.418810 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_606b0e91907c4c5384fab7ba4980f048 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:53.421579 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74b4c563d6bd4a44bb3bab58a3fcdc07 consensus state has no leader
I20240502 08:16:53.422252 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 711a38d7c47749f8a7f59e52b81b51ee
I20240502 08:16:53.423141 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 711a38d7c47749f8a7f59e52b81b51ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.425694 24173 tablet_bootstrap.cc:654] T 1da01e2c2f054ccfb53383e70e16da84 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:53.427222 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 96013fa1fa014b1891eb621cc3a81cdc consensus state has no leader
I20240502 08:16:53.442531 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ba1baa2279045b3adc58d187a12cbf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:53.442847 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7b37f776df47da9aa6e7fba82a8999 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:53.447328 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968a301fd817452a8b4552e959d2e371. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.447782 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1364eb9eb640bdb315f6e2aa24be48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.451534 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 711a38d7c47749f8a7f59e52b81b51ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:53.456424 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d1364eb9eb640bdb315f6e2aa24be48 consensus state has no leader
I20240502 08:16:53.461122 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f29ed2608a4966ba878ad2dae45975. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.476248 24174 tablet_bootstrap.cc:492] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:53.477216 23835 heartbeater.cc:347] Connected to a master server at 127.22.225.125:36189
I20240502 08:16:53.477793 24174 tablet_bootstrap.cc:654] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.480928 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0cd4a9137f44f2be21056164a35afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.481434 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaea443ae9044b76b48b3b5f92a9ae7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.480935 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 788) took 5297 ms. Trace:
I20240502 08:16:53.484791 24180 rpcz_store.cc:276] 0502 08:16:48.183795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.184243 (+ 448us) service_pool.cc:225] Handling call
0502 08:16:52.739362 (+4555119us) ts_tablet_manager.cc:1120] Deleting tablet aacdf506221044ffada0fec0c7d42d0b
0502 08:16:52.739594 (+ 232us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:53.163236 (+423642us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:53.391649 (+228413us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:53.441070 (+ 49421us) spinlock_profiling.cc:243] Waited 40.4 ms on lock 0x61800000fd18. stack: 00007f1068831756 00007f1068831510 00007f106786448d 0000000000955804 00000000009557cd 000000000095519a 00007f106882719d 00007f106882820e 00007f107bd2d4ed 00007f107bca96b3 00007f1077e0c9dd 00007f1077e3e8d4 00007f1077e1fbc0 00007f1077e1f94d 0000000000a60bcc 00007f1068877b74
0502 08:16:53.480924 (+ 39854us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":92988288,"tablet-delete.queue_time_us":4553799}
I20240502 08:16:53.485436 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 789) took 5301 ms. Trace:
I20240502 08:16:53.485606 24180 rpcz_store.cc:276] 0502 08:16:48.183883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.184432 (+ 549us) service_pool.cc:225] Handling call
0502 08:16:53.485290 (+5300858us) ts_tablet_manager.cc:1120] Deleting tablet e608dd4f99004bb795de5e05ea700e08
0502 08:16:53.485306 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:53.485420 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5300735}
I20240502 08:16:53.486091 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 794) took 5211 ms. Trace:
I20240502 08:16:53.486222 24180 rpcz_store.cc:276] 0502 08:16:48.274360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.274414 (+ 54us) service_pool.cc:225] Handling call
0502 08:16:53.485975 (+5211561us) ts_tablet_manager.cc:1120] Deleting tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58
0502 08:16:53.485985 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:53.486074 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5211367}
I20240502 08:16:53.486617 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 795) took 5176 ms. Trace:
I20240502 08:16:53.486791 24180 rpcz_store.cc:276] 0502 08:16:48.310174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.310230 (+ 56us) service_pool.cc:225] Handling call
0502 08:16:53.486512 (+5176282us) ts_tablet_manager.cc:1120] Deleting tablet 7d07efb303b64ce19d993c4c76a47dfc
0502 08:16:53.486522 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:53.486604 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5176111}
I20240502 08:16:53.487190 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 797) took 5058 ms. Trace:
I20240502 08:16:53.487305 24180 rpcz_store.cc:276] 0502 08:16:48.429261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.429327 (+ 66us) service_pool.cc:225] Handling call
0502 08:16:53.487060 (+5057733us) ts_tablet_manager.cc:1120] Deleting tablet 18625a1d971c43549b9f80e6046efc9c
0502 08:16:53.487067 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:53.487174 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5057495}
I20240502 08:16:53.487605 24180 tablet_replica.cc:330] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:53.488073 24180 raft_consensus.cc:2229] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:53.488310 24180 raft_consensus.cc:2258] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:53.488495 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet aacdf506221044ffada0fec0c7d42d0b (table default.Table_b2e1c08bfc404c7ebb2720631ad10495 [id=9480d59a76fc401e959a1e41caafc845]) successfully deleted
W20240502 08:16:53.488752 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e608dd4f99004bb795de5e05ea700e08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e608dd4f99004bb795de5e05ea700e08 already in progress: creating tablet
I20240502 08:16:53.488708 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_53ed03e87b794949a6d4d86f2a617612" } new_table_name: "default.Table_308171ca5d9c4bfdb1d31a3b6e92b62f" modify_external_catalogs: true
W20240502 08:16:53.488996 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58 already in progress: creating tablet
W20240502 08:16:53.489187 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 7d07efb303b64ce19d993c4c76a47dfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d07efb303b64ce19d993c4c76a47dfc already in progress: creating tablet
I20240502 08:16:53.489303 24180 ts_tablet_manager.cc:1875] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:53.489449 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 18625a1d971c43549b9f80e6046efc9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18625a1d971c43549b9f80e6046efc9c already in progress: creating tablet
I20240502 08:16:53.495054 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6e5de9033944979c30e3bd4d1e24fd (DEFAULT_TABLE table=default.Table_b13bfe8c518a4f27b9c2006d6f35b5c7 [id=f36f139558ba426b9c3a71ad07ed7fef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.495811 23750 tablet_service.cc:1467] Processing CreateTablet for tablet ac7a7cec2e154d6fa095d68d1ba3308d (DEFAULT_TABLE table=default.Table_8cbe6e485a474bdba481239cfa55ee8a [id=ed0cc9cd5fbb448e8c68f94f395e8fcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.496730 23768 tablet_service.cc:1467] Processing CreateTablet for tablet e80acef39ff34f4c95f704df55d379a8 (DEFAULT_TABLE table=default.Table_c19c35e94c9748758a471eeb0896e0a9 [id=af1dae0c2e3a41ae92d48b6fdd72465d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.497154 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 577b149f52e44d5d9a9f6b9d85a6864d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:53.497251 24023 tablet_service.cc:1467] Processing CreateTablet for tablet ac7a7cec2e154d6fa095d68d1ba3308d (DEFAULT_TABLE table=default.Table_8cbe6e485a474bdba481239cfa55ee8a [id=ed0cc9cd5fbb448e8c68f94f395e8fcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.497642 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7a7cec2e154d6fa095d68d1ba3308d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.497772 24022 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6e5de9033944979c30e3bd4d1e24fd (DEFAULT_TABLE table=default.Table_b13bfe8c518a4f27b9c2006d6f35b5c7 [id=f36f139558ba426b9c3a71ad07ed7fef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.498055 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6e5de9033944979c30e3bd4d1e24fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.499344 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 9672f5e61436482abcfc4e6bc033c49a (DEFAULT_TABLE table=default.Table_be653bb574ef4b5fba3f88ab616996be [id=7959f5e1f8dc43e59c4565810bc13bc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.499511 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 17c4456172d24ce885e6a8503f2fad85 (DEFAULT_TABLE table=default.Table_cea1e21aa3c3460aadbb92dd84a0acd7 [id=af616a92fcc045eb92c27fa250591f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.499884 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 2459cb52e4f74d67b6d3b5bd37f96a5d (DEFAULT_TABLE table=default.Table_b23dfc41a96043ed98c03260ce4d0d32 [id=93352a34dca1412c8f88e647fed14260]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.500270 24181 ts_tablet_manager.cc:1888] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:53.500371 24181 log.cc:1196] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/5a1e208733844829b13222e00a432439
I20240502 08:16:53.500674 24181 ts_tablet_manager.cc:1909] T 5a1e208733844829b13222e00a432439 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:53.501816 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 2459cb52e4f74d67b6d3b5bd37f96a5d (DEFAULT_TABLE table=default.Table_b23dfc41a96043ed98c03260ce4d0d32 [id=93352a34dca1412c8f88e647fed14260]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.502116 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2459cb52e4f74d67b6d3b5bd37f96a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.502249 23905 tablet_service.cc:1467] Processing CreateTablet for tablet f3312ac30e73492f88c0edb602343952 (DEFAULT_TABLE table=default.Table_a6c2b028992644d987b8a3ca62b1d325 [id=8d63c8f5e8c849629d90d5081e64bac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.502637 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3312ac30e73492f88c0edb602343952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.502804 23888 tablet_service.cc:1467] Processing CreateTablet for tablet e6815b2b3f79412fb04debc5d71f214c (DEFAULT_TABLE table=default.Table_e67c0ee30255468aa0674964089b483e [id=8d71ccbbc5ab41dcb45188399a7ce396]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.503091 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6815b2b3f79412fb04debc5d71f214c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.497471 24036 tablet_service.cc:1467] Processing CreateTablet for tablet e80acef39ff34f4c95f704df55d379a8 (DEFAULT_TABLE table=default.Table_c19c35e94c9748758a471eeb0896e0a9 [id=af1dae0c2e3a41ae92d48b6fdd72465d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.506596 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e80acef39ff34f4c95f704df55d379a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.509397 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9935d3cfff50430dbe8adff249d2c68e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.510006 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 5604212dd32c430ab35f8c50fb9d7023 (DEFAULT_TABLE table=default.Table_b2b68fb4c3c14505b960a7ec8623b0e2 [id=285bec7abdeb418fb4bd6cab90dbea85]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:53.511476 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ae2b20ae3a014c85af33d5ee51b704e1 consensus state has no leader
I20240502 08:16:53.512991 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_20b409a694114049bf0d966535b376a8" } modify_external_catalogs: true
I20240502 08:16:53.513192 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_20b409a694114049bf0d966535b376a8" } modify_external_catalogs: true
I20240502 08:16:53.513160 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_cc09d7ab929c4f4ba017a922d1b1cad9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.513778 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc09d7ab929c4f4ba017a922d1b1cad9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.514518 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2439a8f2e30f4bdf8b18d7ac811ab586" } modify_external_catalogs: true
I20240502 08:16:53.514691 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2439a8f2e30f4bdf8b18d7ac811ab586" } modify_external_catalogs: true
I20240502 08:16:53.515856 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_562886123f0548bfab83210c1c45b552" } modify_external_catalogs: true
I20240502 08:16:53.516013 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_562886123f0548bfab83210c1c45b552" } modify_external_catalogs: true
I20240502 08:16:53.518545 24175 tablet_bootstrap.cc:492] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:53.520258 24175 tablet_bootstrap.cc:654] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.522303 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 1daa7305e0054fbb971864ad8449c33f (DEFAULT_TABLE table=default.Table_1e7d21d372ec4ef9bd840ff21bbea400 [id=ea12c5554a364ebb9f7cde1f7d29bf11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.523114 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 711a38d7c47749f8a7f59e52b81b51ee (DEFAULT_TABLE table=default.Table_6a3a4cab06a545d2b1af63c5bb0e695a [id=de7696830453467990b30118453a1d78]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.523571 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 7209bd5b466e4f3a8ab31704b1e83f0f (DEFAULT_TABLE table=default.Table_efdf6a98cfe94b879b0959eccdbf3dac [id=6ae69bfd2fb84c5ea034b58138bac5d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.523905 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 418ab502915c4e4ea9998dd2d0180c48 with delete_type TABLET_DATA_DELETED (replaced by tablet 89fefde55e514e5fabae3f4e21f44ca3) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.528542 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_212251b2938346e19add6081bc09d254" } modify_external_catalogs: true
I20240502 08:16:53.528734 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_212251b2938346e19add6081bc09d254" } modify_external_catalogs: true
I20240502 08:16:53.530858 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 552e46f69230474189278911cfeec6c7 (DEFAULT_TABLE table=default.Table_562886123f0548bfab83210c1c45b552 [id=91d11e967c41466daa2971eace82b289]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.531069 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9 (DEFAULT_TABLE table=default.Table_0b5edfb1bf6e470f97bbbb62d09e353f [id=c79cdf3f081f4e42a01d425cc46ba015]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.531569 23886 tablet_service.cc:1514] Processing DeleteTablet for tablet df87b8b1ff504031bf007880792192dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.531694 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 9672f5e61436482abcfc4e6bc033c49a (DEFAULT_TABLE table=default.Table_be653bb574ef4b5fba3f88ab616996be [id=7959f5e1f8dc43e59c4565810bc13bc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.531888 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 968a301fd817452a8b4552e959d2e371 (DEFAULT_TABLE table=default.Table_b673548cd88d4f51975f7433f1509cc9 [id=909d7a034f194299ac30c21aa666d2a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.532125 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9672f5e61436482abcfc4e6bc033c49a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.532176 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet 40fab016ed4f489aa3ccdeafcfbe1477 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.532151 24038 tablet_service.cc:1467] Processing CreateTablet for tablet ed5916a2229545b4ba13515219d459cb (DEFAULT_TABLE table=default.Table_6cde1d82c6b1472a843c94cccec9ee6c [id=00551fb59aa04e23a8c7cb96f413164d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.532553 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed5916a2229545b4ba13515219d459cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.532493 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 86990bef52294256a3c4604415551fb8 (DEFAULT_TABLE table=default.Table_8fd21899dd67434986e4e8fb3b799ce1 [id=d3ad9c99da954d3dac95473eb64d00dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.532810 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86990bef52294256a3c4604415551fb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.534257 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 17c4456172d24ce885e6a8503f2fad85 (DEFAULT_TABLE table=default.Table_cea1e21aa3c3460aadbb92dd84a0acd7 [id=af616a92fcc045eb92c27fa250591f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.534562 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c4456172d24ce885e6a8503f2fad85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.537792 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 5604212dd32c430ab35f8c50fb9d7023 (DEFAULT_TABLE table=default.Table_b2b68fb4c3c14505b960a7ec8623b0e2 [id=285bec7abdeb418fb4bd6cab90dbea85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.538026 24037 tablet_service.cc:1467] Processing CreateTablet for tablet fa1703fbb5bc42d1990d8064a14f2519 (DEFAULT_TABLE table=default.Table_15baf520893a4ad28df2c15c35649e15 [id=082704d7828a475696368c6f2c34afb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.537813 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 89fefde55e514e5fabae3f4e21f44ca3 (DEFAULT_TABLE table=default.Table_417bf79bff5e4e01ab05528227bf9081 [id=d13560bbbb3749ac9fb456bb82cd6479]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.538372 24031 tablet_service.cc:1467] Processing CreateTablet for tablet bb9e64d0f9384d1a9ea220ce5e982262 (DEFAULT_TABLE table=default.Table_47d67e47e5a24496aaa6876575d954f5 [id=1b388cae110643c6aafe24cebf4af00f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.538836 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 2c28c47b4f814573971227b2208d1f83 (DEFAULT_TABLE table=default.Table_2df607ff8993422187123671e2dcede3 [id=b5dbd75090c54c15a6eaad1dab9c5736]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.538926 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0cd4a9137f44f2be21056164a35afc (DEFAULT_TABLE table=default.Table_89e8be3491994ad4adc3c32d36c23d27 [id=9fa5384988334b57b1be8ddbb6f0c1c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.531188 23891 tablet_service.cc:1467] Processing CreateTablet for tablet f82e7b6c450c456b8547dd7c2ffc5c9a (DEFAULT_TABLE table=default.Table_5f3b7adae72c466fa28ec5465a266bcc [id=646be2f6f121446a838aa9a400095781]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.539104 23751 tablet_service.cc:1467] Processing CreateTablet for tablet ed5916a2229545b4ba13515219d459cb (DEFAULT_TABLE table=default.Table_6cde1d82c6b1472a843c94cccec9ee6c [id=00551fb59aa04e23a8c7cb96f413164d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.538115 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5604212dd32c430ab35f8c50fb9d7023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.539269 23759 tablet_service.cc:1467] Processing CreateTablet for tablet bb9e64d0f9384d1a9ea220ce5e982262 (DEFAULT_TABLE table=default.Table_47d67e47e5a24496aaa6876575d954f5 [id=1b388cae110643c6aafe24cebf4af00f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.539481 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 89fefde55e514e5fabae3f4e21f44ca3 (DEFAULT_TABLE table=default.Table_417bf79bff5e4e01ab05528227bf9081 [id=d13560bbbb3749ac9fb456bb82cd6479]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.539609 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fefde55e514e5fabae3f4e21f44ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.539640 23752 tablet_service.cc:1467] Processing CreateTablet for tablet 86990bef52294256a3c4604415551fb8 (DEFAULT_TABLE table=default.Table_8fd21899dd67434986e4e8fb3b799ce1 [id=d3ad9c99da954d3dac95473eb64d00dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.539803 23760 tablet_service.cc:1467] Processing CreateTablet for tablet fa1703fbb5bc42d1990d8064a14f2519 (DEFAULT_TABLE table=default.Table_15baf520893a4ad28df2c15c35649e15 [id=082704d7828a475696368c6f2c34afb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.540021 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 2c28c47b4f814573971227b2208d1f83 (DEFAULT_TABLE table=default.Table_2df607ff8993422187123671e2dcede3 [id=b5dbd75090c54c15a6eaad1dab9c5736]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.541728 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9e64d0f9384d1a9ea220ce5e982262. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.542860 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c28c47b4f814573971227b2208d1f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.548183 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1703fbb5bc42d1990d8064a14f2519. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.551654 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6e5de9033944979c30e3bd4d1e24fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.555526 24021 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b8d81f071d74b218fcfd4f4e8c20bab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:53.555737 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 751808dfc1964b539f6f836d786156aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:53.556943 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_08cf295230eb40d7a7807bbd049f127b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:53.557817 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7a7cec2e154d6fa095d68d1ba3308d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.558419 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 711a38d7c47749f8a7f59e52b81b51ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.558741 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet dd3078cc1f6247609f8bd017ed076b45 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.559072 23887 tablet_service.cc:1467] Processing CreateTablet for tablet f9f29ed2608a4966ba878ad2dae45975 (DEFAULT_TABLE table=default.Table_fbc10d796e98456ea3d7899ece9dcfc4 [id=e640a7b706b64dc5a6de4ba14584cee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.559185 23900 tablet_service.cc:1467] Processing CreateTablet for tablet e80acef39ff34f4c95f704df55d379a8 (DEFAULT_TABLE table=default.Table_c19c35e94c9748758a471eeb0896e0a9 [id=af1dae0c2e3a41ae92d48b6fdd72465d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.559895 23895 tablet_service.cc:1467] Processing CreateTablet for tablet ac7a7cec2e154d6fa095d68d1ba3308d (DEFAULT_TABLE table=default.Table_8cbe6e485a474bdba481239cfa55ee8a [id=ed0cc9cd5fbb448e8c68f94f395e8fcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.560649 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 2459cb52e4f74d67b6d3b5bd37f96a5d (DEFAULT_TABLE table=default.Table_b23dfc41a96043ed98c03260ce4d0d32 [id=93352a34dca1412c8f88e647fed14260]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.560911 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c4456172d24ce885e6a8503f2fad85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.561198 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 9672f5e61436482abcfc4e6bc033c49a (DEFAULT_TABLE table=default.Table_be653bb574ef4b5fba3f88ab616996be [id=7959f5e1f8dc43e59c4565810bc13bc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.561678 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 17c4456172d24ce885e6a8503f2fad85 (DEFAULT_TABLE table=default.Table_cea1e21aa3c3460aadbb92dd84a0acd7 [id=af616a92fcc045eb92c27fa250591f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.561949 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1364eb9eb640bdb315f6e2aa24be48 (DEFAULT_TABLE table=default.Table_444fa6f47a2d4a21b14a426d5e90b33b [id=4650b53969234e2e8267990ec8228fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.563208 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6e5de9033944979c30e3bd4d1e24fd (DEFAULT_TABLE table=default.Table_b13bfe8c518a4f27b9c2006d6f35b5c7 [id=f36f139558ba426b9c3a71ad07ed7fef]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:53.564046 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08cf295230eb40d7a7807bbd049f127b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.565085 23902 tablet_service.cc:1467] Processing CreateTablet for tablet eaea443ae9044b76b48b3b5f92a9ae7e (DEFAULT_TABLE table=default.Table_b5566be23d3744348216cab2e9a0c6bf [id=6fd6e744b27b4a6fbfc1fab435a68c7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.565800 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2459cb52e4f74d67b6d3b5bd37f96a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.568495 23754 tablet_service.cc:1514] Processing DeleteTablet for tablet dd3078cc1f6247609f8bd017ed076b45 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.569608 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 88acb3b0c2b7480e917ed68b7646b47d
I20240502 08:16:53.570518 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5604212dd32c430ab35f8c50fb9d7023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.570658 23754 tablet_service.cc:1514] Processing DeleteTablet for tablet 88acb3b0c2b7480e917ed68b7646b47d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.573544 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_066d340c016f4dfea60385ea0b28b7a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:53.573822 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed5916a2229545b4ba13515219d459cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.574018 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86990bef52294256a3c4604415551fb8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:53.574142 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_066d340c016f4dfea60385ea0b28b7a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.574303 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c28c47b4f814573971227b2208d1f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.574517 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9e64d0f9384d1a9ea220ce5e982262. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.577772 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9672f5e61436482abcfc4e6bc033c49a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.579346 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fefde55e514e5fabae3f4e21f44ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.583583 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1703fbb5bc42d1990d8064a14f2519. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.587100 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 765fdb66f12941f0ac16223e931e576e
I20240502 08:16:53.592154 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet 765fdb66f12941f0ac16223e931e576e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.594630 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e80acef39ff34f4c95f704df55d379a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.595170 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 552e46f69230474189278911cfeec6c7
I20240502 08:16:53.596444 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet df266081ed8345dda989b3050f605685
I20240502 08:16:53.597283 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet 552e46f69230474189278911cfeec6c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.597594 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet df266081ed8345dda989b3050f605685 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.605731 25357 raft_consensus.cc:482] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:53.605902 25357 raft_consensus.cc:504] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.606724 25357 leader_election.cc:288] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:53.622393 23897 tablet_service.cc:1467] Processing CreateTablet for tablet 5604212dd32c430ab35f8c50fb9d7023 (DEFAULT_TABLE table=default.Table_b2b68fb4c3c14505b960a7ec8623b0e2 [id=285bec7abdeb418fb4bd6cab90dbea85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.624100 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 88acb3b0c2b7480e917ed68b7646b47d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:53.630230 25357 raft_consensus.cc:482] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:53.630383 25357 raft_consensus.cc:504] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.631086 25357 leader_election.cc:288] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:53.632021 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_853fc43b2ae744be88b3f417db4ae6db" } new_table_name: "default.Table_976507364139460a90d2772b95f079f0" modify_external_catalogs: true
I20240502 08:16:53.632766 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_96756fabe5bb4bb0b8f9b6b105bf9501"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.633365 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96756fabe5bb4bb0b8f9b6b105bf9501 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.633774 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fbada8fc944c43eb8921c9296435a394" } new_table_name: "default.Table_cbe61670c678416995bf27144ab702d9" modify_external_catalogs: true
I20240502 08:16:53.634619 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_019682941bfd4af4af92e990f5801287"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:53.634781 25296 raft_consensus.cc:482] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:53.634932 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_722b7dc3bd6f4bdcb2ffc44d2b77e0a1" } new_table_name: "default.Table_0ef99b98ca9047769c22d238d860cd21" modify_external_catalogs: true
I20240502 08:16:53.634938 25296 raft_consensus.cc:504] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:53.635175 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_019682941bfd4af4af92e990f5801287 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.635691 25296 leader_election.cc:288] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:53.636132 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fffdb860d1084781ac9c8ab11a5194b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:53.636262 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7d7936b041894d9090bff1b6341088fb" } modify_external_catalogs: true
I20240502 08:16:53.636396 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7d7936b041894d9090bff1b6341088fb" } modify_external_catalogs: true
W20240502 08:16:53.636629 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fffdb860d1084781ac9c8ab11a5194b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.633527 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7c85926073de4625bef40b9aa50bb805"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.641892 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c85926073de4625bef40b9aa50bb805 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.642433 25227 raft_consensus.cc:482] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:53.642568 25227 raft_consensus.cc:504] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.643343 24057 raft_consensus.cc:1261] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Refusing update from remote peer 5b89a1f0485a44b7ac1710e2369a809b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 08:16:53.643579 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b8d81f071d74b218fcfd4f4e8c20bab" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:53.643929 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0224317e65747e1add533d33472f7f3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:53.644387 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 765fdb66f12941f0ac16223e931e576e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:53.644282 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fae2341935c6408082d8654e1d8a3f85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.644800 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fae2341935c6408082d8654e1d8a3f85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.644850 25227 leader_election.cc:288] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:53.646155 23918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b8d81f071d74b218fcfd4f4e8c20bab" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:53.647238 23993 leader_election.cc:341] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:53.647753 23996 leader_election.cc:341] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:53.649353 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet df266081ed8345dda989b3050f605685 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:53.650125 23723 leader_election.cc:341] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:53.647861 23996 leader_election.cc:302] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:53.651679 25297 raft_consensus.cc:2726] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:53.655638 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e746013fde4de4862f213792e235ed" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:53.656028 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e7432f1df94431a83cf68356e3e78a3" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:53.656193 25357 consensus_queue.cc:1035] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:53.657104 23861 leader_election.cc:341] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:53.658996 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 750) took 5522 ms. Trace:
I20240502 08:16:53.660120 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet 552e46f69230474189278911cfeec6c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:53.661525 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f0695a0f3e5c48c7aa4f998a18be36e1" } new_table_name: "default.Table_2e0fc2a0993e4208b05240b1ddc4d2ad" modify_external_catalogs: true
W20240502 08:16:53.665489 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c45aa0dd26524e0fba27e4eaf17d704e consensus state has no leader
W20240502 08:16:53.668118 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5704dff6114142d3b27c365ab99e2277 consensus state has no leader
I20240502 08:16:53.672660 24107 heartbeater.cc:347] Connected to a master server at 127.22.225.125:36189
I20240502 08:16:53.675381 25012 ts_tablet_manager.cc:1888] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:53.675963 25012 log.cc:1196] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/9c072eb45e1b41eeb5aafa6d5a786944
I20240502 08:16:53.677203 25012 ts_tablet_manager.cc:1909] T 9c072eb45e1b41eeb5aafa6d5a786944 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:53.678367 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 722) took 5945 ms. Trace:
I20240502 08:16:53.678661 25012 rpcz_store.cc:276] 0502 08:16:47.733282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.733369 (+ 87us) service_pool.cc:225] Handling call
0502 08:16:52.980852 (+5247483us) ts_tablet_manager.cc:1120] Deleting tablet 9c072eb45e1b41eeb5aafa6d5a786944
0502 08:16:52.980860 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:53.334349 (+353489us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:53.674780 (+340431us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:53.678290 (+ 3510us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5247288}
I20240502 08:16:53.692646 24174 tablet_bootstrap.cc:492] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:53.693248 24174 ts_tablet_manager.cc:1367] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.217s user 0.003s sys 0.003s
I20240502 08:16:53.695123 24173 tablet_bootstrap.cc:492] T 1da01e2c2f054ccfb53383e70e16da84 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:53.696856 24173 ts_tablet_manager.cc:1367] T 1da01e2c2f054ccfb53383e70e16da84 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.292s user 0.006s sys 0.028s
I20240502 08:16:53.700253 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0ca7df7eee144c08a8aa1c5cc0f008c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.700939 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ca7df7eee144c08a8aa1c5cc0f008c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.702509 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0c12355d49954405ae0e2b245ccdbc6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.703037 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c12355d49954405ae0e2b245ccdbc6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.703872 24175 tablet_bootstrap.cc:492] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:53.704238 24175 ts_tablet_manager.cc:1367] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.186s user 0.002s sys 0.002s
I20240502 08:16:53.706671 24175 raft_consensus.cc:348] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.707401 24175 raft_consensus.cc:374] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.707636 24175 raft_consensus.cc:729] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.706060 24173 raft_consensus.cc:348] T 1da01e2c2f054ccfb53383e70e16da84 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.708397 24175 consensus_queue.cc:260] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.708897 24173 raft_consensus.cc:374] T 1da01e2c2f054ccfb53383e70e16da84 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.709751 24173 raft_consensus.cc:729] T 1da01e2c2f054ccfb53383e70e16da84 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.713485 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c538b531b58147b89d720628ee7556dd" } new_table_name: "default.Table_d9f68c6b26d442bbbd485bb89ee5564c" modify_external_catalogs: true
I20240502 08:16:53.712041 24173 consensus_queue.cc:260] T 1da01e2c2f054ccfb53383e70e16da84 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.720460 24173 ts_tablet_manager.cc:1398] T 1da01e2c2f054ccfb53383e70e16da84 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.023s user 0.003s sys 0.002s
W20240502 08:16:53.722215 23993 consensus_peers.cc:485] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:53.725437 24173 tablet_bootstrap.cc:492] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:53.737877 25012 tablet_replica.cc:330] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:53.738602 25012 raft_consensus.cc:2229] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:53.739096 25012 raft_consensus.cc:2258] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:53.741729 25012 ts_tablet_manager.cc:1875] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:53.743029 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0902a73da6e34e8e9cd59bb53a36ec0e consensus state has no leader
I20240502 08:16:53.742866 24174 raft_consensus.cc:348] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.744246 24174 raft_consensus.cc:374] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.744482 24174 raft_consensus.cc:729] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.745249 24174 consensus_queue.cc:260] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.750650 24173 tablet_bootstrap.cc:654] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.771617 25296 raft_consensus.cc:482] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:53.771813 25296 raft_consensus.cc:504] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.772598 25296 leader_election.cc:288] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:53.773146 25357 raft_consensus.cc:482] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:53.773301 25357 raft_consensus.cc:504] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.773763 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001e3f4f72f64dc69dbd521724912f07" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:53.774766 25357 leader_election.cc:288] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:53.774827 23723 leader_election.cc:341] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:53.775343 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2126bf2d4ab44628b52481dc2ec3a99" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:53.782788 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_90c5920a0df74de598d87d8ce85c91ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.783490 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90c5920a0df74de598d87d8ce85c91ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.659874 24181 rpcz_store.cc:276] 0502 08:16:48.136303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.136364 (+ 61us) service_pool.cc:225] Handling call
0502 08:16:52.962438 (+4826074us) ts_tablet_manager.cc:1120] Deleting tablet 5a1e208733844829b13222e00a432439
0502 08:16:52.962446 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:53.064026 (+101580us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x616000145c54. stack: 00007f9184bfa756 00007f9184bfa510 00007f9183c2d48d 0000000000955804 00000000009557cd 000000000095519a 00007f9190468a00 00007f9191742853 00007f91941d557c 00007f91942078d4 00007f91941e8bc0 00007f91941e894d 0000000000a60bcc 00007f9184c40b74 00007f9184c4419b 00007f9184c44018
0502 08:16:53.236862 (+172836us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:53.500169 (+263307us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:53.658575 (+158406us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":232142592,"tablet-delete.queue_time_us":4825880}
I20240502 08:16:53.785090 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ddf5aa41088d4de4bcf2768d82c7d1fb" } modify_external_catalogs: true
I20240502 08:16:53.785295 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ddf5aa41088d4de4bcf2768d82c7d1fb" } modify_external_catalogs: true
I20240502 08:16:53.785843 24181 tablet_replica.cc:330] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:53.786749 24181 raft_consensus.cc:2229] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:53.787535 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 5a1e208733844829b13222e00a432439 (table default.Table_7c04460336ea48e79db571d95cdb1bcc [id=8ea9f7fff43846dfa8310549e8e57f2f]) successfully deleted
W20240502 08:16:53.789741 23597 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 0902a73da6e34e8e9cd59bb53a36ec0e (table default.Table_0ef99b98ca9047769c22d238d860cd21 [id=3f8d5d0f6bf04fc0a5f354c3bb24f146], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0902a73da6e34e8e9cd59bb53a36ec0e consensus state has no leader
I20240502 08:16:53.790068 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8cbe6e485a474bdba481239cfa55ee8a" } modify_external_catalogs: true
I20240502 08:16:53.790073 24175 ts_tablet_manager.cc:1398] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.086s user 0.003s sys 0.000s
I20240502 08:16:53.790269 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8cbe6e485a474bdba481239cfa55ee8a" } modify_external_catalogs: true
I20240502 08:16:53.790911 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaea443ae9044b76b48b3b5f92a9ae7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.791147 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6e5de9033944979c30e3bd4d1e24fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.793682 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2459cb52e4f74d67b6d3b5bd37f96a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.793989 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1daa7305e0054fbb971864ad8449c33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.794180 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f29ed2608a4966ba878ad2dae45975. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.794386 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7a7cec2e154d6fa095d68d1ba3308d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.794579 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9672f5e61436482abcfc4e6bc033c49a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.795123 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 552e46f69230474189278911cfeec6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.795364 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c4456172d24ce885e6a8503f2fad85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.795747 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.795943 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5604212dd32c430ab35f8c50fb9d7023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.796111 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968a301fd817452a8b4552e959d2e371. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.800022 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_963bbb5ed3184e6a95549deb63ffb35a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:53.800518 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1364eb9eb640bdb315f6e2aa24be48. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:53.800670 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_963bbb5ed3184e6a95549deb63ffb35a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.801530 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f82e7b6c450c456b8547dd7c2ffc5c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.808177 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e80acef39ff34f4c95f704df55d379a8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:53.809623 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac50774b2dd241de9cae8096578d183e consensus state has no leader
I20240502 08:16:53.810154 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_93b89623977c4ede951f10ea31e1762b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.810778 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93b89623977c4ede951f10ea31e1762b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.816815 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711a38d7c47749f8a7f59e52b81b51ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.824735 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_787b2d1011974a8d95e16bf54ec92ffd" } modify_external_catalogs: true
I20240502 08:16:53.825022 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_787b2d1011974a8d95e16bf54ec92ffd" } modify_external_catalogs: true
I20240502 08:16:53.826498 25357 raft_consensus.cc:482] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:53.826635 25357 raft_consensus.cc:504] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.827281 25357 leader_election.cc:288] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:53.829886 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7209bd5b466e4f3a8ab31704b1e83f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.834954 23887 tablet_service.cc:1467] Processing CreateTablet for tablet bb9e64d0f9384d1a9ea220ce5e982262 (DEFAULT_TABLE table=default.Table_47d67e47e5a24496aaa6876575d954f5 [id=1b388cae110643c6aafe24cebf4af00f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.835796 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9e64d0f9384d1a9ea220ce5e982262. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.836510 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 86990bef52294256a3c4604415551fb8 (DEFAULT_TABLE table=default.Table_8fd21899dd67434986e4e8fb3b799ce1 [id=d3ad9c99da954d3dac95473eb64d00dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.836850 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86990bef52294256a3c4604415551fb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.839447 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f30d039709f9465e96e1280298ae2aaf" } modify_external_catalogs: true
I20240502 08:16:53.839658 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f30d039709f9465e96e1280298ae2aaf" } modify_external_catalogs: true
W20240502 08:16:53.840276 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07a9d13d3a2d4e6dbc04998c3dd7e1a8 consensus state has no leader
I20240502 08:16:53.841316 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 2c28c47b4f814573971227b2208d1f83 (DEFAULT_TABLE table=default.Table_2df607ff8993422187123671e2dcede3 [id=b5dbd75090c54c15a6eaad1dab9c5736]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.841471 23895 tablet_service.cc:1467] Processing CreateTablet for tablet ed5916a2229545b4ba13515219d459cb (DEFAULT_TABLE table=default.Table_6cde1d82c6b1472a843c94cccec9ee6c [id=00551fb59aa04e23a8c7cb96f413164d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.841658 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c28c47b4f814573971227b2208d1f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.843245 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 89fefde55e514e5fabae3f4e21f44ca3 (DEFAULT_TABLE table=default.Table_417bf79bff5e4e01ab05528227bf9081 [id=d13560bbbb3749ac9fb456bb82cd6479]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.844128 23888 tablet_service.cc:1467] Processing CreateTablet for tablet fa1703fbb5bc42d1990d8064a14f2519 (DEFAULT_TABLE table=default.Table_15baf520893a4ad28df2c15c35649e15 [id=082704d7828a475696368c6f2c34afb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.844555 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fefde55e514e5fabae3f4e21f44ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.846621 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0cd4a9137f44f2be21056164a35afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.848301 23904 tablet_service.cc:1514] Processing DeleteTablet for tablet 88acb3b0c2b7480e917ed68b7646b47d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.848538 23904 tablet_service.cc:1514] Processing DeleteTablet for tablet 765fdb66f12941f0ac16223e931e576e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.848863 23904 tablet_service.cc:1514] Processing DeleteTablet for tablet 552e46f69230474189278911cfeec6c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.849107 23904 tablet_service.cc:1514] Processing DeleteTablet for tablet df266081ed8345dda989b3050f605685 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:53.849439 23602 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 001e3f4f72f64dc69dbd521724912f07 (table default.Table_d7cff78803d44e5aba3b50cd8ff11e92 [id=5c21117955d946418c656330984ae747]). Expected version 3 got 0
I20240502 08:16:53.850155 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1703fbb5bc42d1990d8064a14f2519. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.853924 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 3922e528a97a4287a0122a5b51035dcb (DEFAULT_TABLE table=default.Table_606b0e91907c4c5384fab7ba4980f048 [id=0be2d1db3a8d4709bf7c21ddb6e735d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.854365 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3922e528a97a4287a0122a5b51035dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.854604 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 2014ed6b794f4c65b0a4061b8f3cf601 (DEFAULT_TABLE table=default.Table_4cc1a3bc9a5c4574958aaa494a0dad5d [id=5afc2bf0b42946be829df946b0f79458]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.854952 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2014ed6b794f4c65b0a4061b8f3cf601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.855186 23768 tablet_service.cc:1467] Processing CreateTablet for tablet e7fe8765fb8c4cd184e7c3b3444b266d (DEFAULT_TABLE table=default.Table_d5b106e54e3b4396ac55b919e56ba322 [id=529cacef92454873b2f4c1d7cf73267d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.855530 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7fe8765fb8c4cd184e7c3b3444b266d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.862346 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb816c486a04481afe0832895acfc90 (DEFAULT_TABLE table=default.Table_655c02f28fab4cfaacd923e876e91078 [id=35ee91b3344f4048a75d14f749a87185]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.862784 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb816c486a04481afe0832895acfc90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.866329 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 17d6f4f6e9ee46c28b7e7513f791c81f (DEFAULT_TABLE table=default.Table_7d7936b041894d9090bff1b6341088fb [id=599d48758ccb48bbad14e8037d9efe5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.866712 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17d6f4f6e9ee46c28b7e7513f791c81f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.866941 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_bcfbfe93e2a14297828b5837679bfd8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.867666 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bcfbfe93e2a14297828b5837679bfd8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.827879 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3078cc1f6247609f8bd017ed076b45" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:53.869832 24057 raft_consensus.cc:2454] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:53.863782 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb816c486a04481afe0832895acfc90 (DEFAULT_TABLE table=default.Table_655c02f28fab4cfaacd923e876e91078 [id=35ee91b3344f4048a75d14f749a87185]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.870172 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5dcb3ffd44ea42de93aad8dcced74ed4" } new_table_name: "default.Table_5301ffccbac743618a78289c2c1415e1" modify_external_catalogs: true
I20240502 08:16:53.870373 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb816c486a04481afe0832895acfc90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.871094 23861 leader_election.cc:302] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:53.871171 24175 tablet_bootstrap.cc:492] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:53.872144 25357 raft_consensus.cc:2781] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:53.872222 25357 raft_consensus.cc:482] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:53.872175 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e29dfece3b88466ba6e6f6db0c1a31fd" } new_table_name: "default.Table_382a0f031cf1457c8b008f9123f8af74" modify_external_catalogs: true
I20240502 08:16:53.872277 25357 raft_consensus.cc:3037] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:53.872396 24175 tablet_bootstrap.cc:654] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.874866 25357 raft_consensus.cc:504] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.875563 25357 leader_election.cc:288] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:53.878588 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 558a3a4a8383439bb64af1d3eaa64572 (DEFAULT_TABLE table=default.Table_fc9f3ca03af3472d9628fdfe64a89450 [id=935942d3df41464789cb1513641a9e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.878822 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 30056c688ae1443a8d2b50501ecbe620 (DEFAULT_TABLE table=default.Table_cd64f661d15b4fddbd06f615b876bfc6 [id=07b649d73cbf43c78db62765ca02c2d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.879070 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558a3a4a8383439bb64af1d3eaa64572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.880962 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30056c688ae1443a8d2b50501ecbe620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.882408 24174 ts_tablet_manager.cc:1398] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.189s user 0.004s sys 0.000s
I20240502 08:16:53.882755 24174 tablet_bootstrap.cc:492] T 6e7432f1df94431a83cf68356e3e78a3 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:53.884353 24174 tablet_bootstrap.cc:654] T 6e7432f1df94431a83cf68356e3e78a3 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.885392 24181 raft_consensus.cc:2258] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:53.885798 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 643ce929e74c499da835fbe9b63b8880 (table default.Table_a0ebd3d8bd13425ea80ffd0c428e20da [id=1c9043e3bf434bae83ea36aaef230b6f]) successfully deleted
W20240502 08:16:53.886446 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 48d96e42813f4169969dd5931e169923 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48d96e42813f4169969dd5931e169923 already in progress: creating tablet
I20240502 08:16:53.888998 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 4f1736b496354e2dadd844242d3867f8 (table default.Table_02f448a8a9844def8ecda8716fdc352e [id=79a03d22b7e34f76bb3242c640129a13]) successfully deleted
W20240502 08:16:53.889474 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet ba3aa2621fdb43fd8299f81af486ef1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba3aa2621fdb43fd8299f81af486ef1e already in progress: creating tablet
W20240502 08:16:53.889851 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 202fa612a085450ca9c1225434cb7fe4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 202fa612a085450ca9c1225434cb7fe4 already in progress: creating tablet
I20240502 08:16:53.890106 24181 ts_tablet_manager.cc:1875] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:53.891280 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 4baa2bc2752c450fa212471c150cf098 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4baa2bc2752c450fa212471c150cf098 already in progress: creating tablet
W20240502 08:16:53.891602 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet d36e7cd33f9a4edabe690e75602590b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d36e7cd33f9a4edabe690e75602590b9 already in progress: creating tablet
W20240502 08:16:53.891940 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 25c8a6aaabe2413d8ce88f54ce45a4c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25c8a6aaabe2413d8ce88f54ce45a4c4 already in progress: creating tablet
I20240502 08:16:53.893224 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet e686db13bb7541c69ae88f0fb7e27061 (table default.Table_178527d31d7044fc98fa41886d651049 [id=6dc275aed42b42118a70099d3eae3c21]) successfully deleted
I20240502 08:16:53.907231 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 3922e528a97a4287a0122a5b51035dcb (DEFAULT_TABLE table=default.Table_606b0e91907c4c5384fab7ba4980f048 [id=0be2d1db3a8d4709bf7c21ddb6e735d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.907701 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3922e528a97a4287a0122a5b51035dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.908018 24028 tablet_service.cc:1467] Processing CreateTablet for tablet e7fe8765fb8c4cd184e7c3b3444b266d (DEFAULT_TABLE table=default.Table_d5b106e54e3b4396ac55b919e56ba322 [id=529cacef92454873b2f4c1d7cf73267d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.908381 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7fe8765fb8c4cd184e7c3b3444b266d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.860566 24022 tablet_service.cc:1467] Processing CreateTablet for tablet 2014ed6b794f4c65b0a4061b8f3cf601 (DEFAULT_TABLE table=default.Table_4cc1a3bc9a5c4574958aaa494a0dad5d [id=5afc2bf0b42946be829df946b0f79458]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.917946 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2014ed6b794f4c65b0a4061b8f3cf601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.918251 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 17d6f4f6e9ee46c28b7e7513f791c81f (DEFAULT_TABLE table=default.Table_7d7936b041894d9090bff1b6341088fb [id=599d48758ccb48bbad14e8037d9efe5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.918589 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17d6f4f6e9ee46c28b7e7513f791c81f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.919399 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3078cc1f6247609f8bd017ed076b45" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:53.919596 24057 raft_consensus.cc:3037] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:53.920030 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 558a3a4a8383439bb64af1d3eaa64572 (DEFAULT_TABLE table=default.Table_fc9f3ca03af3472d9628fdfe64a89450 [id=935942d3df41464789cb1513641a9e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.920354 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558a3a4a8383439bb64af1d3eaa64572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.924319 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 30056c688ae1443a8d2b50501ecbe620 (DEFAULT_TABLE table=default.Table_cd64f661d15b4fddbd06f615b876bfc6 [id=07b649d73cbf43c78db62765ca02c2d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.924619 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30056c688ae1443a8d2b50501ecbe620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.943084 25227 raft_consensus.cc:482] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:53.943043 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e746013fde4de4862f213792e235ed" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:53.943226 25227 raft_consensus.cc:504] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.943755 25227 leader_election.cc:288] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:53.944535 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577b149f52e44d5d9a9f6b9d85a6864d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:53.944864 23787 raft_consensus.cc:2454] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:53.946301 24174 tablet_bootstrap.cc:492] T 6e7432f1df94431a83cf68356e3e78a3 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:53.946455 24174 ts_tablet_manager.cc:1367] T 6e7432f1df94431a83cf68356e3e78a3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.000s
I20240502 08:16:53.946872 23993 leader_election.cc:302] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:53.947185 24174 raft_consensus.cc:348] T 6e7432f1df94431a83cf68356e3e78a3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.947357 24174 raft_consensus.cc:374] T 6e7432f1df94431a83cf68356e3e78a3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.947409 24174 raft_consensus.cc:729] T 6e7432f1df94431a83cf68356e3e78a3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.947616 24174 consensus_queue.cc:260] T 6e7432f1df94431a83cf68356e3e78a3 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.948064 24180 ts_tablet_manager.cc:1888] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:16:53.948164 24180 log.cc:1196] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/e134e90efb4a48bb82b53a80bda708b3
I20240502 08:16:53.948199 24174 ts_tablet_manager.cc:1398] T 6e7432f1df94431a83cf68356e3e78a3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:53.948417 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577b149f52e44d5d9a9f6b9d85a6864d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:53.948525 24174 tablet_bootstrap.cc:492] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:53.948567 24180 ts_tablet_manager.cc:1909] T e134e90efb4a48bb82b53a80bda708b3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:53.948720 25227 raft_consensus.cc:2781] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:53.948807 25227 raft_consensus.cc:482] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:53.948871 25227 raft_consensus.cc:3037] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:53.950090 24174 tablet_bootstrap.cc:654] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.955192 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e7432f1df94431a83cf68356e3e78a3" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:53.955808 23787 raft_consensus.cc:2454] T 6e7432f1df94431a83cf68356e3e78a3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:53.956753 25357 raft_consensus.cc:482] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:53.956885 25357 raft_consensus.cc:504] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.957453 25357 leader_election.cc:288] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:53.957578 25357 raft_consensus.cc:482] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:53.957659 24175 tablet_bootstrap.cc:492] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:53.957795 24175 ts_tablet_manager.cc:1367] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.087s user 0.004s sys 0.000s
I20240502 08:16:53.957677 25357 raft_consensus.cc:504] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:53.958573 23861 leader_election.cc:341] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: d9fcc7accb1f45ad9cf55b8f64d21b91
I20240502 08:16:53.958806 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed5916a2229545b4ba13515219d459cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.962108 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 17d6f4f6e9ee46c28b7e7513f791c81f
I20240502 08:16:53.962884 24174 tablet_bootstrap.cc:492] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:53.962852 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d3cd93eb7315400aa078bbb480182814" } new_table_name: "default.Table_ce9cf2a3f4b44ffaa0b92f8fc3e09bf7" modify_external_catalogs: true
I20240502 08:16:53.963019 24174 ts_tablet_manager.cc:1367] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240502 08:16:53.963711 24174 raft_consensus.cc:348] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.963907 24174 raft_consensus.cc:374] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.963766 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1060ebe6de4a49499e58dfa9a9ab5e20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:53.963955 24174 raft_consensus.cc:729] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.964145 24174 consensus_queue.cc:260] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:53.964320 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1060ebe6de4a49499e58dfa9a9ab5e20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.965440 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 518cb4b86ea34e95bef51c5c2a64d89b
I20240502 08:16:53.965804 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 7a0ff1c3818a49d2b13f57ea6a81c3d6
I20240502 08:16:53.966535 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e46226aed42b4c30869f1eeb5016c9e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.967015 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e46226aed42b4c30869f1eeb5016c9e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.971037 24174 ts_tablet_manager.cc:1398] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:16:53.971721 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 9c072eb45e1b41eeb5aafa6d5a786944 (table default.Table_20c5ace604d64b6cabead6037cdc04e1 [id=2e5831a5849049c8bd445667b01b1bd8]) successfully deleted
I20240502 08:16:53.971983 24174 tablet_bootstrap.cc:492] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:53.975200 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_bac8a6673ffc44e7825538aa4257329e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:53.975790 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bac8a6673ffc44e7825538aa4257329e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:53.977386 24174 tablet_bootstrap.cc:654] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.949364 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 801) took 5491 ms. Trace:
I20240502 08:16:53.981067 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet 518cb4b86ea34e95bef51c5c2a64d89b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.983474 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a0ff1c3818a49d2b13f57ea6a81c3d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:53.983963 24174 tablet_bootstrap.cc:492] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:53.984098 24174 ts_tablet_manager.cc:1367] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 08:16:53.979368 24180 rpcz_store.cc:276] 0502 08:16:48.457659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.457729 (+ 70us) service_pool.cc:225] Handling call
0502 08:16:53.487564 (+5029835us) ts_tablet_manager.cc:1120] Deleting tablet e134e90efb4a48bb82b53a80bda708b3
0502 08:16:53.487572 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:53.729172 (+241600us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:53.947953 (+218781us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:53.949344 (+ 1391us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5029613}
I20240502 08:16:53.984902 24174 raft_consensus.cc:348] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.985088 24174 raft_consensus.cc:374] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.985141 24174 raft_consensus.cc:729] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.985340 24174 consensus_queue.cc:260] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:53.985482 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet e134e90efb4a48bb82b53a80bda708b3 (table default.Table_af464bf1ebe54d7281005831aa79b8bf [id=47682c26a85842c7b8991bf3828a9336]) successfully deleted
I20240502 08:16:53.989250 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 804) took 5486 ms. Trace:
I20240502 08:16:53.991518 24174 ts_tablet_manager.cc:1398] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:16:53.991838 24174 tablet_bootstrap.cc:492] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:53.990352 24180 rpcz_store.cc:276] 0502 08:16:48.502169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.502349 (+ 180us) service_pool.cc:225] Handling call
0502 08:16:53.986753 (+5484404us) ts_tablet_manager.cc:1120] Deleting tablet 571886fdbc2c43e19c790b2b7830ec27
0502 08:16:53.986764 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:53.988330 (+ 1566us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5483804}
W20240502 08:16:53.992893 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 571886fdbc2c43e19c790b2b7830ec27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 571886fdbc2c43e19c790b2b7830ec27 already in progress: creating tablet
I20240502 08:16:53.993471 24174 tablet_bootstrap.cc:654] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.995004 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 805) took 5492 ms. Trace:
I20240502 08:16:53.995895 24180 rpcz_store.cc:276] 0502 08:16:48.502298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.502579 (+ 281us) service_pool.cc:225] Handling call
0502 08:16:53.993821 (+5491242us) ts_tablet_manager.cc:1120] Deleting tablet e37d58fbcf6b476fb8c4558b3763c94f
0502 08:16:53.993836 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:53.994975 (+ 1139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5490422}
W20240502 08:16:53.998340 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e37d58fbcf6b476fb8c4558b3763c94f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e37d58fbcf6b476fb8c4558b3763c94f already in progress: creating tablet
I20240502 08:16:53.999961 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 812) took 5468 ms. Trace:
I20240502 08:16:54.000550 24180 rpcz_store.cc:276] 0502 08:16:48.531551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.531601 (+ 50us) service_pool.cc:225] Handling call
0502 08:16:53.998818 (+5467217us) ts_tablet_manager.cc:1120] Deleting tablet 0e559ce43d2c440fa50715f25cc2c7e9
0502 08:16:53.998832 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:53.999765 (+ 933us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5466869}
W20240502 08:16:54.002414 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 0e559ce43d2c440fa50715f25cc2c7e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e559ce43d2c440fa50715f25cc2c7e9 already in progress: creating tablet
I20240502 08:16:54.003129 24174 tablet_bootstrap.cc:492] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:54.003249 24174 ts_tablet_manager.cc:1367] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 08:16:54.004643 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 816) took 5340 ms. Trace:
I20240502 08:16:54.005033 24174 raft_consensus.cc:348] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.005219 24174 raft_consensus.cc:374] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.005662 24174 raft_consensus.cc:729] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.005147 24180 rpcz_store.cc:276] 0502 08:16:48.664055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.666634 (+ 2579us) service_pool.cc:225] Handling call
0502 08:16:54.003219 (+5336585us) ts_tablet_manager.cc:1120] Deleting tablet 982333cbce18439da3b35693b3f6ca0a
0502 08:16:54.003232 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.004626 (+ 1394us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5335618}
I20240502 08:16:54.006736 25296 raft_consensus.cc:482] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:54.006842 25296 raft_consensus.cc:504] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:54.007489 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 982333cbce18439da3b35693b3f6ca0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 982333cbce18439da3b35693b3f6ca0a already in progress: creating tablet
I20240502 08:16:54.007566 25296 leader_election.cc:288] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:54.007743 24174 consensus_queue.cc:260] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.008358 24174 ts_tablet_manager.cc:1398] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 08:16:54.008626 24174 tablet_bootstrap.cc:492] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:54.009441 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 815) took 5346 ms. Trace:
I20240502 08:16:54.011255 24174 tablet_bootstrap.cc:654] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.010401 24180 rpcz_store.cc:276] 0502 08:16:48.662865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.666907 (+ 4042us) service_pool.cc:225] Handling call
0502 08:16:54.007838 (+5340931us) ts_tablet_manager.cc:1120] Deleting tablet e9133e69e1724a26bc3650ea658ca349
0502 08:16:54.008030 (+ 192us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.009173 (+ 1143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5340591}
W20240502 08:16:54.012969 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e9133e69e1724a26bc3650ea658ca349 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9133e69e1724a26bc3650ea658ca349 already in progress: creating tablet
I20240502 08:16:54.015836 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet 17d6f4f6e9ee46c28b7e7513f791c81f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:54.019312 25227 raft_consensus.cc:504] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.019878 25227 leader_election.cc:288] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:54.020787 23996 leader_election.cc:341] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:54.021723 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577b149f52e44d5d9a9f6b9d85a6864d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:54.025967 23996 consensus_peers.cc:485] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:16:54.026350 23996 leader_election.cc:341] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:54.029449 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 17d6f4f6e9ee46c28b7e7513f791c81f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:54.037119 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1d946bd6de48f1a8d8fe5f179d22b6" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:54.037451 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 518cb4b86ea34e95bef51c5c2a64d89b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:54.039041 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a0ff1c3818a49d2b13f57ea6a81c3d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:54.042732 24173 tablet_bootstrap.cc:492] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:54.042867 24173 ts_tablet_manager.cc:1367] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.319s user 0.011s sys 0.009s
I20240502 08:16:54.020635 24174 tablet_bootstrap.cc:492] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:54.046347 24174 ts_tablet_manager.cc:1367] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.001s
I20240502 08:16:54.022367 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577b149f52e44d5d9a9f6b9d85a6864d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:54.046582 23787 raft_consensus.cc:3037] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.048038 24057 raft_consensus.cc:2454] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:54.048496 25357 leader_election.cc:288] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:54.048645 25357 raft_consensus.cc:482] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:54.048744 25357 raft_consensus.cc:504] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.049096 23787 raft_consensus.cc:2454] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:54.049257 25357 leader_election.cc:288] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:54.038542 23723 leader_election.cc:341] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:54.038744 25296 consensus_queue.cc:1035] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:54.049685 24175 raft_consensus.cc:348] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.049849 24175 raft_consensus.cc:374] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.049896 24175 raft_consensus.cc:729] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.050333 23993 leader_election.cc:302] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:54.050726 25297 raft_consensus.cc:2781] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:54.050823 25297 raft_consensus.cc:686] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:54.051003 25297 consensus_queue.cc:237] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.056229 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet ac7a7cec2e154d6fa095d68d1ba3308d
I20240502 08:16:54.058869 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ec2cb5eeb2c34373b60522a9725b77e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:54.059394 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec2cb5eeb2c34373b60522a9725b77e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.061198 24174 raft_consensus.cc:348] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.061384 24174 raft_consensus.cc:374] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.061434 24174 raft_consensus.cc:729] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.061657 24174 consensus_queue.cc:260] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.062116 24174 ts_tablet_manager.cc:1398] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 08:16:54.066685 24173 raft_consensus.cc:348] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.066854 24173 raft_consensus.cc:374] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.066900 24173 raft_consensus.cc:729] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.067054 24173 consensus_queue.cc:260] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.067539 24173 ts_tablet_manager.cc:1398] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.025s user 0.000s sys 0.000s
I20240502 08:16:54.050079 24175 consensus_queue.cc:260] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.071241 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet ac7a7cec2e154d6fa095d68d1ba3308d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:54.072224 24173 tablet_bootstrap.cc:492] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:54.073460 24173 tablet_bootstrap.cc:654] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.073696 25317 raft_consensus.cc:482] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:54.073833 25317 raft_consensus.cc:504] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.074554 25317 leader_election.cc:288] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:54.075193 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0224317e65747e1add533d33472f7f3" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:54.075376 23787 raft_consensus.cc:2454] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:54.075783 23993 leader_election.cc:302] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:54.076151 25317 raft_consensus.cc:2781] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:54.076208 25317 raft_consensus.cc:482] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:54.076278 25317 raft_consensus.cc:3037] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.084805 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba1baa2279045b3adc58d187a12cbf3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:54.084990 23916 raft_consensus.cc:3037] T 1ba1baa2279045b3adc58d187a12cbf3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.088092 23916 raft_consensus.cc:2454] T 1ba1baa2279045b3adc58d187a12cbf3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:54.091759 23918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577b149f52e44d5d9a9f6b9d85a6864d" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:54.093520 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6746c25111d04f20a6f30ce0838d71f5" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:54.095582 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0224317e65747e1add533d33472f7f3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:54.096307 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001e3f4f72f64dc69dbd521724912f07" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:54.097507 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1d946bd6de48f1a8d8fe5f179d22b6" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:54.099300 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2126bf2d4ab44628b52481dc2ec3a99" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:54.100406 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3078cc1f6247609f8bd017ed076b45" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:54.100621 23787 raft_consensus.cc:2454] T dd3078cc1f6247609f8bd017ed076b45 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:54.105404 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3078cc1f6247609f8bd017ed076b45" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:54.105649 23787 raft_consensus.cc:3037] T dd3078cc1f6247609f8bd017ed076b45 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.106806 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet ac7a7cec2e154d6fa095d68d1ba3308d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:54.108410 23787 raft_consensus.cc:2454] T dd3078cc1f6247609f8bd017ed076b45 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:54.109735 25407 raft_consensus.cc:482] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:54.113406 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c49da1e0f0435ba241a6a66526e790" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:54.117779 25396 consensus_queue.cc:1035] T 96fcb895c8f449deb0190de75af18c3c P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:54.119513 25396 consensus_queue.cc:1035] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:54.120373 23723 leader_election.cc:341] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:54.120476 23723 leader_election.cc:302] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:54.120909 25396 raft_consensus.cc:2726] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:54.123517 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 817) took 5450 ms. Trace:
I20240502 08:16:54.123598 24180 rpcz_store.cc:276] 0502 08:16:48.673458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.673513 (+ 55us) service_pool.cc:225] Handling call
0502 08:16:54.013176 (+5339663us) ts_tablet_manager.cc:1120] Deleting tablet 725982e4712e442d8ff1fd3f796e18f5
0502 08:16:54.123393 (+110217us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.123498 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5339456}
I20240502 08:16:54.123932 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 819) took 5376 ms. Trace:
I20240502 08:16:54.124004 24180 rpcz_store.cc:276] 0502 08:16:48.747310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.747440 (+ 130us) service_pool.cc:225] Handling call
0502 08:16:54.123838 (+5376398us) ts_tablet_manager.cc:1120] Deleting tablet b123bb81d0b44d4ea6443536168d085c
0502 08:16:54.123847 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.123920 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5376134}
W20240502 08:16:54.124078 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 725982e4712e442d8ff1fd3f796e18f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 725982e4712e442d8ff1fd3f796e18f5 already in progress: creating tablet
I20240502 08:16:54.124265 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 818) took 5395 ms. Trace:
I20240502 08:16:54.124325 24180 rpcz_store.cc:276] 0502 08:16:48.728483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.749790 (+ 21307us) service_pool.cc:225] Handling call
0502 08:16:54.124182 (+5374392us) ts_tablet_manager.cc:1120] Deleting tablet 9a2cbe751dc4450eaa3eb60268ad510d
0502 08:16:54.124192 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.124252 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5374188}
I20240502 08:16:54.124584 24180 tablet_replica.cc:330] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
W20240502 08:16:54.124614 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet b123bb81d0b44d4ea6443536168d085c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b123bb81d0b44d4ea6443536168d085c already in progress: creating tablet
I20240502 08:16:54.124775 24180 raft_consensus.cc:2229] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
W20240502 08:16:54.125082 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 9a2cbe751dc4450eaa3eb60268ad510d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a2cbe751dc4450eaa3eb60268ad510d already in progress: creating tablet
I20240502 08:16:54.125123 24180 raft_consensus.cc:2258] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.125720 24180 ts_tablet_manager.cc:1875] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.129848 24174 tablet_bootstrap.cc:492] T 5d04312bd81f46ed8c49c42b1af1aec9 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:54.131413 24174 tablet_bootstrap.cc:654] T 5d04312bd81f46ed8c49c42b1af1aec9 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.136690 24180 ts_tablet_manager.cc:1888] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:16:54.139503 24180 log.cc:1196] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/642744194e22461f8316a78689fd99ff
I20240502 08:16:54.140132 24180 ts_tablet_manager.cc:1909] T 642744194e22461f8316a78689fd99ff P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
W20240502 08:16:54.141842 23434 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 85585.9us
I20240502 08:16:54.142201 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 888) took 4475 ms. Trace:
I20240502 08:16:54.142289 24180 rpcz_store.cc:276] 0502 08:16:49.666314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.818645 (+2152331us) service_pool.cc:225] Handling call
0502 08:16:54.124544 (+2305899us) ts_tablet_manager.cc:1120] Deleting tablet 642744194e22461f8316a78689fd99ff
0502 08:16:54.124552 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.129055 (+ 4503us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.136571 (+ 7516us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.142181 (+ 5610us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2305750}
I20240502 08:16:54.142618 24180 tablet_replica.cc:330] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:54.142791 24180 raft_consensus.cc:2229] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:54.142848 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 642744194e22461f8316a78689fd99ff (table default.Table_8bed20f633e348679b0087687298cc42 [id=18ab2c0716464d82a0e33bd99bd0735d]) successfully deleted
I20240502 08:16:54.142975 24180 raft_consensus.cc:2258] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.144948 24180 ts_tablet_manager.cc:1875] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.146689 24174 tablet_bootstrap.cc:492] T 5d04312bd81f46ed8c49c42b1af1aec9 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:54.146821 24174 ts_tablet_manager.cc:1367] T 5d04312bd81f46ed8c49c42b1af1aec9 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.003s
I20240502 08:16:54.147591 24174 raft_consensus.cc:348] T 5d04312bd81f46ed8c49c42b1af1aec9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.147809 24174 raft_consensus.cc:374] T 5d04312bd81f46ed8c49c42b1af1aec9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.147879 24174 raft_consensus.cc:729] T 5d04312bd81f46ed8c49c42b1af1aec9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.148126 24174 consensus_queue.cc:260] T 5d04312bd81f46ed8c49c42b1af1aec9 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.148783 24174 ts_tablet_manager.cc:1398] T 5d04312bd81f46ed8c49c42b1af1aec9 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:54.149159 24174 tablet_bootstrap.cc:492] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:54.151122 24174 tablet_bootstrap.cc:654] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.151602 24181 ts_tablet_manager.cc:1888] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:54.151990 24181 log.cc:1196] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/c1e746013fde4de4862f213792e235ed
I20240502 08:16:54.153131 25296 consensus_queue.cc:1035] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:54.153806 24181 ts_tablet_manager.cc:1909] T c1e746013fde4de4862f213792e235ed P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
W20240502 08:16:54.153834 23723 leader_election.cc:341] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:54.153985 23723 leader_election.cc:302] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:54.154278 25296 raft_consensus.cc:2726] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:54.157935 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 752) took 6021 ms. Trace:
I20240502 08:16:54.158035 24181 rpcz_store.cc:276] 0502 08:16:48.136557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.136597 (+ 40us) service_pool.cc:225] Handling call
0502 08:16:53.785556 (+5648959us) ts_tablet_manager.cc:1120] Deleting tablet c1e746013fde4de4862f213792e235ed
0502 08:16:53.785626 (+ 70us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.030106 (+244480us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.150677 (+120571us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.157912 (+ 7235us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15561,"tablet-delete.queue_time_us":5648765}
I20240502 08:16:54.158393 24181 tablet_replica.cc:330] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
W20240502 08:16:54.158982 23723 leader_election.cc:341] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:54.158960 25296 consensus_queue.cc:1035] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:54.159160 23723 leader_election.cc:302] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:54.159394 25296 raft_consensus.cc:2726] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:54.159516 23723 leader_election.cc:341] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:54.159611 23723 leader_election.cc:302] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:54.159821 25296 raft_consensus.cc:2726] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:54.159937 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet c1e746013fde4de4862f213792e235ed (table default.Table_42d6187c77b347e58022b68ed044f3fe [id=ecfe360ec5554bdcac562a1c614e1672]) successfully deleted
W20240502 08:16:54.160059 23723 leader_election.cc:341] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:54.160154 23723 leader_election.cc:302] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:54.160372 25296 raft_consensus.cc:2726] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:54.160600 25296 consensus_queue.cc:1035] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:54.161046 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a4104f4cb64889ad66810788ad6b29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:54.161509 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 3922e528a97a4287a0122a5b51035dcb (DEFAULT_TABLE table=default.Table_606b0e91907c4c5384fab7ba4980f048 [id=0be2d1db3a8d4709bf7c21ddb6e735d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.161923 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3922e528a97a4287a0122a5b51035dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.165638 23891 tablet_service.cc:1467] Processing CreateTablet for tablet e7fe8765fb8c4cd184e7c3b3444b266d (DEFAULT_TABLE table=default.Table_d5b106e54e3b4396ac55b919e56ba322 [id=529cacef92454873b2f4c1d7cf73267d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.166131 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7fe8765fb8c4cd184e7c3b3444b266d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.172734 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 2014ed6b794f4c65b0a4061b8f3cf601 (DEFAULT_TABLE table=default.Table_4cc1a3bc9a5c4574958aaa494a0dad5d [id=5afc2bf0b42946be829df946b0f79458]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.173220 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2014ed6b794f4c65b0a4061b8f3cf601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.175930 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb816c486a04481afe0832895acfc90 (DEFAULT_TABLE table=default.Table_655c02f28fab4cfaacd923e876e91078 [id=35ee91b3344f4048a75d14f749a87185]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.176321 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb816c486a04481afe0832895acfc90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.182335 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 17d6f4f6e9ee46c28b7e7513f791c81f (DEFAULT_TABLE table=default.Table_7d7936b041894d9090bff1b6341088fb [id=599d48758ccb48bbad14e8037d9efe5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.182695 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17d6f4f6e9ee46c28b7e7513f791c81f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.183660 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e4ac2b5073e34cd98e0246620fc1515d
I20240502 08:16:54.185891 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 558a3a4a8383439bb64af1d3eaa64572 (DEFAULT_TABLE table=default.Table_fc9f3ca03af3472d9628fdfe64a89450 [id=935942d3df41464789cb1513641a9e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.186242 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558a3a4a8383439bb64af1d3eaa64572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.186204 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cd64f661d15b4fddbd06f615b876bfc6" } new_table_name: "default.Table_516047691aa3440bb18be9a6213fe9c1" modify_external_catalogs: true
W20240502 08:16:54.187652 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0b0d4871dd24607854100a4ef762e25 consensus state has no leader
I20240502 08:16:54.188083 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0d515c0c1c334feb9d4252e90ba9ad4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:54.188871 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1fc2270a5461448ca0245bd875f4245a consensus state has no leader
I20240502 08:16:54.189167 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0f71ff280bad45758c41f3056333ed38" } new_table_name: "default.Table_fd40ada5753247409cbacbce80cf2233" modify_external_catalogs: true
W20240502 08:16:54.190281 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 12bd0a3bf7d24d939288c3eca83e1540 consensus state has no leader
I20240502 08:16:54.190578 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2d4fafaec4f240edae01ff9a29566924"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:54.191519 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b59e0701b4d042f0b4d6277ade5ab169"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:54.192135 25418 raft_consensus.cc:482] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:54.192272 25418 raft_consensus.cc:504] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.192373 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet e2126bf2d4ab44628b52481dc2ec3a99 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:54.192931 25418 leader_election.cc:288] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:54.193017 24175 ts_tablet_manager.cc:1398] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.235s user 0.002s sys 0.000s
I20240502 08:16:54.192942 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fc4c2996ae4c4e8ea466bf9625b823f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:54.193333 24175 tablet_bootstrap.cc:492] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:54.193867 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d40724ccadc148cd91ea36278515b65c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:54.196370 24175 tablet_bootstrap.cc:654] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.198343 24022 tablet_service.cc:1514] Processing DeleteTablet for tablet e4ac2b5073e34cd98e0246620fc1515d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:54.200567 25012 ts_tablet_manager.cc:1888] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:54.201802 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2df607ff8993422187123671e2dcede3" } modify_external_catalogs: true
W20240502 08:16:54.202390 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d40724ccadc148cd91ea36278515b65c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:54.202165 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d4fafaec4f240edae01ff9a29566924 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:54.202821 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b59e0701b4d042f0b4d6277ade5ab169 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:54.206751 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d515c0c1c334feb9d4252e90ba9ad4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.207509 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 30056c688ae1443a8d2b50501ecbe620 (DEFAULT_TABLE table=default.Table_cd64f661d15b4fddbd06f615b876bfc6 [id=07b649d73cbf43c78db62765ca02c2d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.207882 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30056c688ae1443a8d2b50501ecbe620. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.202683 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2df607ff8993422187123671e2dcede3" } modify_external_catalogs: true
W20240502 08:16:54.210851 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc4c2996ae4c4e8ea466bf9625b823f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.212771 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 17d6f4f6e9ee46c28b7e7513f791c81f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:54.200708 25012 log.cc:1196] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/1e5de793e74247dd9e73b7b1d58027d9
I20240502 08:16:54.213404 25012 ts_tablet_manager.cc:1909] T 1e5de793e74247dd9e73b7b1d58027d9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:54.214238 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 723) took 6480 ms. Trace:
I20240502 08:16:54.214314 25012 rpcz_store.cc:276] 0502 08:16:47.733812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.733855 (+ 43us) service_pool.cc:225] Handling call
0502 08:16:53.737522 (+6003667us) ts_tablet_manager.cc:1120] Deleting tablet 1e5de793e74247dd9e73b7b1d58027d9
0502 08:16:53.737729 (+ 207us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:53.876012 (+138283us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.200466 (+324454us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.214218 (+ 13752us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6003416}
I20240502 08:16:54.214653 25012 tablet_replica.cc:330] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:54.214828 25012 raft_consensus.cc:2229] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:54.214963 25012 raft_consensus.cc:2258] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.215471 25012 ts_tablet_manager.cc:1875] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.218214 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet e4ac2b5073e34cd98e0246620fc1515d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:54.110191 25407 raft_consensus.cc:504] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.219978 25407 leader_election.cc:288] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:54.220559 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 59733us
I20240502 08:16:54.221297 25012 ts_tablet_manager.cc:1888] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:54.221423 25012 log.cc:1196] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/d9fcc7accb1f45ad9cf55b8f64d21b91
I20240502 08:16:54.222808 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f02efa659e457ab57bc32ed3f66bb9" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:54.223065 24053 raft_consensus.cc:2454] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:54.224242 23723 leader_election.cc:302] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:54.224802 25158 raft_consensus.cc:2781] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:54.224872 25158 raft_consensus.cc:482] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:54.224925 25158 raft_consensus.cc:3037] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.226922 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a0ff1c3818a49d2b13f57ea6a81c3d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:54.227497 25158 raft_consensus.cc:504] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.228457 25158 leader_election.cc:288] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:54.229614 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f02efa659e457ab57bc32ed3f66bb9" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:54.229866 24053 raft_consensus.cc:3037] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:54.230970 23861 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 185418us
I20240502 08:16:54.231725 23861 leader_election.cc:302] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:54.231842 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c49da1e0f0435ba241a6a66526e790" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:54.232313 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 518cb4b86ea34e95bef51c5c2a64d89b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:54.232389 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0224317e65747e1add533d33472f7f3" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:54.232319 25407 raft_consensus.cc:2781] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:54.232594 25407 raft_consensus.cc:686] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:16:54.232831 25407 consensus_queue.cc:237] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.236459 24174 tablet_bootstrap.cc:492] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:54.237143 24174 ts_tablet_manager.cc:1367] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.088s user 0.004s sys 0.000s
I20240502 08:16:54.237776 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet ac7a7cec2e154d6fa095d68d1ba3308d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:54.238291 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet e4ac2b5073e34cd98e0246620fc1515d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:53 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:54.239666 24174 raft_consensus.cc:348] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.239840 24174 raft_consensus.cc:374] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.239884 24174 raft_consensus.cc:729] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.240058 24174 consensus_queue.cc:260] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.240518 24174 ts_tablet_manager.cc:1398] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:16:54.240764 24174 tablet_bootstrap.cc:492] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:54.238723 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 1da01e2c2f054ccfb53383e70e16da84 with delete_type TABLET_DATA_DELETED (replaced by tablet 3e36be367f9b4c568ce0f3278b6297d9) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:54.243356 23858 leader_election.cc:341] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 5bcc7169796e4ee0910f5e7084849012
W20240502 08:16:54.245246 23858 leader_election.cc:341] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 5bcc7169796e4ee0910f5e7084849012
W20240502 08:16:54.248646 23858 consensus_peers.cc:485] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd4a31e5aa834adeb9f59267247c6980. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:54.250690 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7a0267df98fe45a583a32684e35f9416" } modify_external_catalogs: true
W20240502 08:16:54.246978 23861 leader_election.cc:341] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 98a96292b28a4b0aad88fe0a96730a68
I20240502 08:16:54.247056 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1da01e2c2f054ccfb53383e70e16da84" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:54.252107 24057 raft_consensus.cc:2454] T 1da01e2c2f054ccfb53383e70e16da84 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
W20240502 08:16:54.252192 23861 consensus_peers.cc:485] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:54.244495 25012 ts_tablet_manager.cc:1909] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:54.252466 23861 leader_election.cc:302] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:54.252729 25357 raft_consensus.cc:2781] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:54.252815 25357 raft_consensus.cc:482] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:54.252894 25357 raft_consensus.cc:3037] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.252187 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7a0267df98fe45a583a32684e35f9416" } modify_external_catalogs: true
I20240502 08:16:54.254488 23858 leader_election.cc:302] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:54.254937 23858 consensus_peers.cc:485] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5bcc7169796e4ee0910f5e7084849012. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:54.255056 23858 leader_election.cc:302] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:54.255312 25403 raft_consensus.cc:2726] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:54.242043 24174 tablet_bootstrap.cc:654] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.255545 25403 raft_consensus.cc:2781] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:54.255610 25403 raft_consensus.cc:482] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:54.255664 25403 raft_consensus.cc:3037] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.258663 25403 raft_consensus.cc:504] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.259120 25357 raft_consensus.cc:504] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.259392 25403 leader_election.cc:288] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:54.259735 25357 leader_election.cc:288] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:54.260686 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 724) took 6526 ms. Trace:
I20240502 08:16:54.260807 25012 rpcz_store.cc:276] 0502 08:16:47.734264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:47.734308 (+ 44us) service_pool.cc:225] Handling call
0502 08:16:54.214607 (+6480299us) ts_tablet_manager.cc:1120] Deleting tablet d9fcc7accb1f45ad9cf55b8f64d21b91
0502 08:16:54.214616 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.218417 (+ 3801us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.221208 (+ 2791us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.260665 (+ 39457us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6480122}
I20240502 08:16:54.264009 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 725) took 6209 ms. Trace:
I20240502 08:16:54.264156 25012 rpcz_store.cc:276] 0502 08:16:48.054096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.054158 (+ 62us) service_pool.cc:225] Handling call
0502 08:16:54.263681 (+6209523us) ts_tablet_manager.cc:1120] Deleting tablet 13cbab4352bb44f389d70d4379e92457
0502 08:16:54.263701 (+ 20us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.263975 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6208881}
I20240502 08:16:54.264703 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 726) took 6210 ms. Trace:
I20240502 08:16:54.264832 25012 rpcz_store.cc:276] 0502 08:16:48.054655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.054703 (+ 48us) service_pool.cc:225] Handling call
0502 08:16:54.264584 (+6209881us) ts_tablet_manager.cc:1120] Deleting tablet 21d2db2158d4447f92eedc5780b46056
0502 08:16:54.264595 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.264686 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6209719}
I20240502 08:16:54.266356 25317 raft_consensus.cc:504] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.267537 25317 leader_election.cc:288] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:54.268402 23996 leader_election.cc:341] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:54.269351 24053 raft_consensus.cc:2454] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:54.269881 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 727) took 6214 ms. Trace:
I20240502 08:16:54.269917 23723 leader_election.cc:302] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:54.269974 25012 rpcz_store.cc:276] 0502 08:16:48.055083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.055131 (+ 48us) service_pool.cc:225] Handling call
0502 08:16:54.269758 (+6214627us) ts_tablet_manager.cc:1120] Deleting tablet 0ae92276246c4d39b194d7e9b868fc02
0502 08:16:54.269767 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.269862 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6214470}
I20240502 08:16:54.270296 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 728) took 6214 ms. Trace:
I20240502 08:16:54.270293 25158 raft_consensus.cc:2781] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:54.270359 25012 rpcz_store.cc:276] 0502 08:16:48.055474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.055522 (+ 48us) service_pool.cc:225] Handling call
0502 08:16:54.270210 (+6214688us) ts_tablet_manager.cc:1120] Deleting tablet 21d2db2158d4447f92eedc5780b46056
0502 08:16:54.270218 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.270283 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6214541}
I20240502 08:16:54.270418 25158 raft_consensus.cc:686] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:54.270679 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e7432f1df94431a83cf68356e3e78a3" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:54.270664 25158 consensus_queue.cc:237] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.271255 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1da01e2c2f054ccfb53383e70e16da84" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:54.271404 24057 raft_consensus.cc:3037] T 1da01e2c2f054ccfb53383e70e16da84 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:54.272672 23861 leader_election.cc:341] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:54.273551 24174 tablet_bootstrap.cc:492] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:54.273705 24174 ts_tablet_manager.cc:1367] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.000s
I20240502 08:16:54.273036 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0224317e65747e1add533d33472f7f3" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:54.276353 23996 leader_election.cc:341] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:54.278347 24174 raft_consensus.cc:348] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.278553 24174 raft_consensus.cc:374] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.278614 24174 raft_consensus.cc:729] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.278645 25413 raft_consensus.cc:482] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:54.278713 24180 ts_tablet_manager.cc:1888] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:54.278795 24180 log.cc:1196] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/c1e746013fde4de4862f213792e235ed
I20240502 08:16:54.278762 25413 raft_consensus.cc:504] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.279135 24180 ts_tablet_manager.cc:1909] T c1e746013fde4de4862f213792e235ed P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:54.279343 25413 leader_election.cc:288] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:54.279413 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 893) took 4608 ms. Trace:
I20240502 08:16:54.279479 24180 rpcz_store.cc:276] 0502 08:16:49.670511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.818805 (+2148294us) service_pool.cc:225] Handling call
0502 08:16:54.142574 (+2323769us) ts_tablet_manager.cc:1120] Deleting tablet c1e746013fde4de4862f213792e235ed
0502 08:16:54.142583 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.246918 (+104335us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.278604 (+ 31686us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.279394 (+ 790us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2323646}
I20240502 08:16:54.278795 24174 consensus_queue.cc:260] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.255435 23858 leader_election.cc:302] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:54.280056 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet c1e746013fde4de4862f213792e235ed (table default.Table_42d6187c77b347e58022b68ed044f3fe [id=ecfe360ec5554bdcac562a1c614e1672]) successfully deleted
I20240502 08:16:54.280236 25407 raft_consensus.cc:2726] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:54.280347 24174 ts_tablet_manager.cc:1398] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 08:16:54.266714 24175 tablet_bootstrap.cc:492] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:54.280581 24175 ts_tablet_manager.cc:1367] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.087s user 0.010s sys 0.002s
I20240502 08:16:54.280663 24174 tablet_bootstrap.cc:492] T 7ed919909289410e867881baab841f53 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:54.281692 24175 raft_consensus.cc:348] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.283581 24057 raft_consensus.cc:2454] T 1da01e2c2f054ccfb53383e70e16da84 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:54.285072 24175 raft_consensus.cc:374] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.285439 24175 raft_consensus.cc:729] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
W20240502 08:16:54.286132 23858 leader_election.cc:341] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: d9fcc7accb1f45ad9cf55b8f64d21b91
I20240502 08:16:54.287068 24175 consensus_queue.cc:260] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.287779 24174 tablet_bootstrap.cc:654] T 7ed919909289410e867881baab841f53 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.288725 24175 ts_tablet_manager.cc:1398] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240502 08:16:54.289321 24181 raft_consensus.cc:2229] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:54.289456 24181 pending_rounds.cc:62] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5: Trying to abort 1 pending ops.
I20240502 08:16:54.289496 24181 pending_rounds.cc:68] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7023156480904564736 op_type: NO_OP noop_request { }
I20240502 08:16:54.289640 24181 raft_consensus.cc:2866] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:54.289714 24181 raft_consensus.cc:2258] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.290520 24181 ts_tablet_manager.cc:1875] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:54.291718 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 30056c688ae1443a8d2b50501ecbe620 consensus state has no leader
I20240502 08:16:54.279764 24180 tablet_replica.cc:330] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
W20240502 08:16:54.297436 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d609a55e47ad420098d279835e9aa090 consensus state has no leader
I20240502 08:16:54.297546 24022 tablet_service.cc:1514] Processing DeleteTablet for tablet 1da01e2c2f054ccfb53383e70e16da84 with delete_type TABLET_DATA_DELETED (replaced by tablet 3e36be367f9b4c568ce0f3278b6297d9) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:54.297752 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_50a5854b27c94d7a9a5f626cc6c8327e" } modify_external_catalogs: true
I20240502 08:16:54.297915 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_50a5854b27c94d7a9a5f626cc6c8327e" } modify_external_catalogs: true
I20240502 08:16:54.297950 24180 raft_consensus.cc:2229] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:54.298089 24181 ts_tablet_manager.cc:1888] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20240502 08:16:54.298183 24181 log.cc:1196] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/5bcc7169796e4ee0910f5e7084849012
I20240502 08:16:54.298259 24180 raft_consensus.cc:2258] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.298527 24181 ts_tablet_manager.cc:1909] T 5bcc7169796e4ee0910f5e7084849012 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:54.298738 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7c85926073de4625bef40b9aa50bb805" } modify_external_catalogs: true
I20240502 08:16:54.298795 24180 ts_tablet_manager.cc:1875] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.298815 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 748) took 6163 ms. Trace:
I20240502 08:16:54.298866 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7c85926073de4625bef40b9aa50bb805" } modify_external_catalogs: true
I20240502 08:16:54.298874 24181 rpcz_store.cc:276] 0502 08:16:48.135577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.136844 (+ 1267us) service_pool.cc:225] Handling call
0502 08:16:54.158348 (+6021504us) ts_tablet_manager.cc:1120] Deleting tablet 5bcc7169796e4ee0910f5e7084849012
0502 08:16:54.158356 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.294055 (+135699us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.297972 (+ 3917us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.298797 (+ 825us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":130709,"tablet-delete.queue_time_us":6021334}
I20240502 08:16:54.300900 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 5bcc7169796e4ee0910f5e7084849012 (table default.Table_bc9b016da1ce42b6a0509fbb664c8548 [id=8c0e32da69b34dd0b2a3c68e4ba7a166]) successfully deleted
I20240502 08:16:54.304175 24181 tablet_replica.cc:330] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:54.304396 24181 raft_consensus.cc:2229] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:54.304761 24181 raft_consensus.cc:2258] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.287554 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 730) took 6230 ms. Trace:
I20240502 08:16:54.308912 25012 rpcz_store.cc:276] 0502 08:16:48.056683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.056731 (+ 48us) service_pool.cc:225] Handling call
0502 08:16:54.270523 (+6213792us) ts_tablet_manager.cc:1120] Deleting tablet 40964817cf2a4712be42f40e37639bc9
0502 08:16:54.270530 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.287534 (+ 17004us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6213642}
I20240502 08:16:54.309358 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 731) took 6252 ms. Trace:
I20240502 08:16:54.290513 23858 leader_election.cc:302] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:54.309437 25012 rpcz_store.cc:276] 0502 08:16:48.057073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.057114 (+ 41us) service_pool.cc:225] Handling call
0502 08:16:54.309216 (+6252102us) ts_tablet_manager.cc:1120] Deleting tablet 4a0684db8dd947d79343afa5a2090614
0502 08:16:54.309228 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.309338 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6251943}
W20240502 08:16:54.309602 23858 raft_consensus.cc:2670] T d9fcc7accb1f45ad9cf55b8f64d21b91 P 5b89a1f0485a44b7ac1710e2369a809b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:16:54.309818 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 732) took 6252 ms. Trace:
I20240502 08:16:54.309896 25012 rpcz_store.cc:276] 0502 08:16:48.115559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.115687 (+ 128us) service_pool.cc:225] Handling call
0502 08:16:54.309710 (+6194023us) ts_tablet_manager.cc:1120] Deleting tablet a4d151770ee44130adfd822518b9b017
0502 08:16:54.309720 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.309802 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6193685}
I20240502 08:16:54.310195 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 734) took 6188 ms. Trace:
I20240502 08:16:54.310261 25012 rpcz_store.cc:276] 0502 08:16:48.123154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.123746 (+ 592us) service_pool.cc:225] Handling call
0502 08:16:54.310098 (+6186352us) ts_tablet_manager.cc:1120] Deleting tablet 28a947b7e7484120b3f4069f6eef5520
0502 08:16:54.310106 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.310178 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6186171}
I20240502 08:16:54.310503 25012 tablet_replica.cc:330] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:54.310705 25012 raft_consensus.cc:2229] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:54.310859 25012 raft_consensus.cc:2258] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.311379 25012 ts_tablet_manager.cc:1875] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.312213 24180 ts_tablet_manager.cc:1888] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:54.312322 24180 log.cc:1196] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/e2126bf2d4ab44628b52481dc2ec3a99
I20240502 08:16:54.312644 24180 ts_tablet_manager.cc:1909] T e2126bf2d4ab44628b52481dc2ec3a99 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:54.312916 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 900) took 3583 ms. Trace:
I20240502 08:16:54.312989 24180 rpcz_store.cc:276] 0502 08:16:50.769523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.871633 (+1102110us) service_pool.cc:225] Handling call
0502 08:16:54.279721 (+2408088us) ts_tablet_manager.cc:1120] Deleting tablet e2126bf2d4ab44628b52481dc2ec3a99
0502 08:16:54.279729 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.309304 (+ 29575us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.312125 (+ 2821us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.312895 (+ 770us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2407917}
I20240502 08:16:54.313274 24180 tablet_replica.cc:330] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:54.313434 24180 raft_consensus.cc:2229] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:54.313436 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet e2126bf2d4ab44628b52481dc2ec3a99 (table default.Table_5b4e617d1c8949daa34aa358c1528f88 [id=92cbf164660d48bd974bf8a26b8dfb0d]) successfully deleted
I20240502 08:16:54.310262 23858 leader_election.cc:302] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:54.313843 25407 raft_consensus.cc:2726] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:54.314018 24175 tablet_bootstrap.cc:492] T 78ddfc7b74644a0bbf8222d5f90e6658 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:54.314697 24173 tablet_bootstrap.cc:492] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:54.315869 24173 ts_tablet_manager.cc:1367] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.244s user 0.009s sys 0.009s
I20240502 08:16:54.316771 24173 raft_consensus.cc:348] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.316983 24173 raft_consensus.cc:374] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.317062 24173 raft_consensus.cc:729] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.317274 24173 consensus_queue.cc:260] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.317889 24173 ts_tablet_manager.cc:1398] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:54.314303 24174 tablet_bootstrap.cc:492] T 7ed919909289410e867881baab841f53 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:54.324101 24174 ts_tablet_manager.cc:1367] T 7ed919909289410e867881baab841f53 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.001s
I20240502 08:16:54.324870 24174 raft_consensus.cc:348] T 7ed919909289410e867881baab841f53 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.325098 24174 raft_consensus.cc:374] T 7ed919909289410e867881baab841f53 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.325160 24174 raft_consensus.cc:729] T 7ed919909289410e867881baab841f53 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.325376 24174 consensus_queue.cc:260] T 7ed919909289410e867881baab841f53 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.326035 24174 ts_tablet_manager.cc:1398] T 7ed919909289410e867881baab841f53 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:54.313556 24180 raft_consensus.cc:2258] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:54.316504 23858 leader_election.cc:341] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 98a96292b28a4b0aad88fe0a96730a68
I20240502 08:16:54.326769 24174 tablet_bootstrap.cc:492] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:54.326786 23858 leader_election.cc:302] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:54.327036 25407 raft_consensus.cc:2726] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:54.328047 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 1e5de793e74247dd9e73b7b1d58027d9 (table default.Table_a286e67e441e43a1b4738cd219dbea98 [id=b56e3c0686244ad6b49afdd589c5ed34]) successfully deleted
I20240502 08:16:54.328183 24174 tablet_bootstrap.cc:654] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.328271 24175 tablet_bootstrap.cc:654] T 78ddfc7b74644a0bbf8222d5f90e6658 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.328433 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet d9fcc7accb1f45ad9cf55b8f64d21b91 (table default.Table_24d1bd1b4dcc4882884e13873cf896ff [id=9d91876e10fd42f883117eb1434d3a8f]) successfully deleted
W20240502 08:16:54.328649 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 13cbab4352bb44f389d70d4379e92457 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13cbab4352bb44f389d70d4379e92457 already in progress: creating tablet
W20240502 08:16:54.328814 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 21d2db2158d4447f92eedc5780b46056 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21d2db2158d4447f92eedc5780b46056 already in progress: creating tablet
W20240502 08:16:54.328984 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 0ae92276246c4d39b194d7e9b868fc02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ae92276246c4d39b194d7e9b868fc02 already in progress: creating tablet
I20240502 08:16:54.329097 24180 ts_tablet_manager.cc:1875] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:54.329140 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 21d2db2158d4447f92eedc5780b46056 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21d2db2158d4447f92eedc5780b46056 already in progress: creating tablet
W20240502 08:16:54.329537 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 40964817cf2a4712be42f40e37639bc9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40964817cf2a4712be42f40e37639bc9 already in progress: creating tablet
I20240502 08:16:54.329556 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1da01e2c2f054ccfb53383e70e16da84" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:54.329738 23786 raft_consensus.cc:3037] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:54.329774 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 4a0684db8dd947d79343afa5a2090614 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a0684db8dd947d79343afa5a2090614 already in progress: creating tablet
W20240502 08:16:54.329946 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet a4d151770ee44130adfd822518b9b017 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4d151770ee44130adfd822518b9b017 already in progress: creating tablet
W20240502 08:16:54.330121 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 28a947b7e7484120b3f4069f6eef5520 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28a947b7e7484120b3f4069f6eef5520 already in progress: creating tablet
I20240502 08:16:54.330106 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1da01e2c2f054ccfb53383e70e16da84" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:54.330446 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e7432f1df94431a83cf68356e3e78a3" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:54.330576 23783 raft_consensus.cc:3037] T 6e7432f1df94431a83cf68356e3e78a3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.332217 23786 raft_consensus.cc:2454] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:54.332872 23783 raft_consensus.cc:2454] T 6e7432f1df94431a83cf68356e3e78a3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:54.339505 24181 ts_tablet_manager.cc:1875] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.343394 23972 heartbeater.cc:347] Connected to a master server at 127.22.225.125:36189
I20240502 08:16:54.344558 23918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f02efa659e457ab57bc32ed3f66bb9" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:54.344664 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0224317e65747e1add533d33472f7f3" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:54.345307 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6746c25111d04f20a6f30ce0838d71f5" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:54.345495 23783 raft_consensus.cc:2454] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
W20240502 08:16:54.346583 23723 leader_election.cc:341] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:54.344333 23919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f02efa659e457ab57bc32ed3f66bb9" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:54.349467 24174 tablet_bootstrap.cc:492] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:54.349638 24174 ts_tablet_manager.cc:1367] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.004s
I20240502 08:16:54.350558 24174 raft_consensus.cc:348] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:54.350777 24174 raft_consensus.cc:374] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.350826 24174 raft_consensus.cc:729] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
W20240502 08:16:54.351075 23858 consensus_peers.cc:485] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 319fabf91458438ba5e40ab68b323055. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:54.351016 24174 consensus_queue.cc:260] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:54.351500 24174 ts_tablet_manager.cc:1398] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:54.351758 24174 tablet_bootstrap.cc:492] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:54.353418 24174 tablet_bootstrap.cc:654] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.355116 24180 ts_tablet_manager.cc:1888] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:54.355207 24180 log.cc:1196] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/a0224317e65747e1add533d33472f7f3
I20240502 08:16:54.355569 24180 ts_tablet_manager.cc:1909] T a0224317e65747e1add533d33472f7f3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:54.355916 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 905) took 3366 ms. Trace:
I20240502 08:16:54.356011 24180 rpcz_store.cc:276] 0502 08:16:50.989280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.878651 (+889371us) service_pool.cc:225] Handling call
0502 08:16:54.313236 (+2434585us) ts_tablet_manager.cc:1120] Deleting tablet a0224317e65747e1add533d33472f7f3
0502 08:16:54.313243 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.334863 (+ 21620us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.354991 (+ 20128us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.355891 (+ 900us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2434423}
I20240502 08:16:54.316850 23861 leader_election.cc:302] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:54.356485 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 906) took 3367 ms. Trace:
I20240502 08:16:54.356642 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet a0224317e65747e1add533d33472f7f3 (table default.Table_5782cc56973c43cfac93b2560c62c688 [id=9a3faa2c394841cfbfab3425282d84d8]) successfully deleted
I20240502 08:16:54.356659 24180 rpcz_store.cc:276] 0502 08:16:50.989369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.878828 (+889459us) service_pool.cc:225] Handling call
0502 08:16:54.356355 (+2477527us) ts_tablet_manager.cc:1120] Deleting tablet ee3d91d55ffd49ceb20d2c33408f5a5a
0502 08:16:54.356366 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.356467 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2477399}
I20240502 08:16:54.356825 25407 raft_consensus.cc:2781] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:54.356971 24180 tablet_replica.cc:330] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:54.356974 25407 raft_consensus.cc:686] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:16:54.357141 24180 raft_consensus.cc:2229] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
W20240502 08:16:54.357287 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet ee3d91d55ffd49ceb20d2c33408f5a5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee3d91d55ffd49ceb20d2c33408f5a5a already in progress: creating tablet
I20240502 08:16:54.357393 24180 raft_consensus.cc:2258] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.357208 25407 consensus_queue.cc:237] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:54.346145 23993 leader_election.cc:341] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: Tablet not RUNNING: STOPPED
I20240502 08:16:54.361174 23993 leader_election.cc:302] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:54.363260 24180 ts_tablet_manager.cc:1875] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.364408 25227 raft_consensus.cc:2726] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:16:54.364464 23993 leader_election.cc:302] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:54.364667 25227 consensus_queue.cc:1035] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:54.365202 25227 raft_consensus.cc:2781] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:54.365289 25227 raft_consensus.cc:482] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:54.365362 25227 raft_consensus.cc:3037] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.365336 25413 consensus_queue.cc:1035] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:54.369334 25227 raft_consensus.cc:504] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.371485 25227 leader_election.cc:288] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:54.371899 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6746c25111d04f20a6f30ce0838d71f5" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:54.372185 23788 raft_consensus.cc:3037] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.374735 23788 raft_consensus.cc:2454] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:54.375164 23993 leader_election.cc:302] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:54.375525 25405 raft_consensus.cc:2781] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:54.375684 25405 raft_consensus.cc:686] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:54.375972 25405 consensus_queue.cc:237] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.377866 24173 tablet_bootstrap.cc:492] T 78ddfc7b74644a0bbf8222d5f90e6658 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:54.392212 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet b59c2f93d6514c9cbf358ab96f1a06b5 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:54.402832 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9e72800839ec45fead8b8b2c5afe4799"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:54.403461 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e72800839ec45fead8b8b2c5afe4799 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.404785 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1fa44e7d541041d3a86bed4be93b5a03" } new_table_name: "default.Table_09d7b1b0dafa4bb6a29891b185e304d1" modify_external_catalogs: true
I20240502 08:16:54.410154 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ab1b6d7ec1d44a4f8559d75bb38fb8a1" } new_table_name: "default.Table_fa90dfc7938f455082fb034307a1ca25" modify_external_catalogs: true
W20240502 08:16:54.422303 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 517354df966c4338a01a425251cccdd2 consensus state has no leader
I20240502 08:16:54.422827 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_019682941bfd4af4af92e990f5801287" } new_table_name: "default.Table_d66d58dae2d54c2c8befedd495a51606" modify_external_catalogs: true
I20240502 08:16:54.424161 23858 leader_election.cc:302] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:54.424554 25357 raft_consensus.cc:2781] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:54.424691 25357 raft_consensus.cc:686] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
W20240502 08:16:54.424731 23723 leader_election.cc:341] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:54.427058 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c26660ee28674f03916ea5aecabf0a21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:54.427531 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6746c25111d04f20a6f30ce0838d71f5" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:54.427647 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c26660ee28674f03916ea5aecabf0a21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.427856 23923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6746c25111d04f20a6f30ce0838d71f5" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:54.428378 23996 consensus_peers.cc:485] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:16:54.428575 23996 leader_election.cc:341] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:54.428937 23996 leader_election.cc:341] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:54.429980 23434 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 50921us
W20240502 08:16:54.432430 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e99c6ea531ff4e9f92e130f9e21f0ff6 consensus state has no leader
I20240502 08:16:54.432902 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_4e5d657dc235492dad016da586febee0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:54.433419 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e5d657dc235492dad016da586febee0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.437791 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet 577b149f52e44d5d9a9f6b9d85a6864d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:54.440246 24174 tablet_bootstrap.cc:492] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:54.440485 24174 ts_tablet_manager.cc:1367] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.089s user 0.004s sys 0.003s
I20240502 08:16:54.441370 24174 raft_consensus.cc:348] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:54.441682 24174 raft_consensus.cc:374] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.441749 24174 raft_consensus.cc:729] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.441955 24174 consensus_queue.cc:260] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:54.442603 24174 ts_tablet_manager.cc:1398] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:54.442942 24174 tablet_bootstrap.cc:492] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:54.443259 24173 tablet_bootstrap.cc:654] T 78ddfc7b74644a0bbf8222d5f90e6658 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.444547 24174 tablet_bootstrap.cc:654] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:54.448765 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e8bb9a499b254ae69ba752b949f409c8 has no consensus state
I20240502 08:16:54.442353 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_41d744e17dee458382533bb204fd1197" } modify_external_catalogs: true
I20240502 08:16:54.449841 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_41d744e17dee458382533bb204fd1197" } modify_external_catalogs: true
I20240502 08:16:54.452903 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_44816133bdb24a48a436fa62d832ed30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:54.453469 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44816133bdb24a48a436fa62d832ed30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.455972 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8e550c4078d04f7db883f8ed5e051554"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:54.456487 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e550c4078d04f7db883f8ed5e051554 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.463795 25403 raft_consensus.cc:482] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:54.463989 25403 raft_consensus.cc:504] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.464761 25403 leader_election.cc:288] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:54.466142 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a4104f4cb64889ad66810788ad6b29" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:54.467305 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a4104f4cb64889ad66810788ad6b29" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:54.468068 23861 leader_election.cc:302] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:54.470616 25403 raft_consensus.cc:2726] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:54.471567 25012 ts_tablet_manager.cc:1888] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:54.473441 25012 log.cc:1196] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/4a57ed188315493198e95cefb6907ab6
I20240502 08:16:54.474856 25012 ts_tablet_manager.cc:1909] T 4a57ed188315493198e95cefb6907ab6 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:54.437258 25357 consensus_queue.cc:237] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.474776 25433 tablet.cc:1722] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240502 08:16:54.476482 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d04312bd81f46ed8c49c42b1af1aec9 with delete_type TABLET_DATA_DELETED (replaced by tablet b45f3233dae94be1a1d0d8fb426d7f2c) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:54.478955 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 735) took 6352 ms. Trace:
I20240502 08:16:54.479130 25012 rpcz_store.cc:276] 0502 08:16:48.128072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.128338 (+ 266us) service_pool.cc:225] Handling call
0502 08:16:54.310463 (+6182125us) ts_tablet_manager.cc:1120] Deleting tablet 4a57ed188315493198e95cefb6907ab6
0502 08:16:54.310472 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.369966 (+ 59494us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.471430 (+101464us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.478933 (+ 7503us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6181928}
I20240502 08:16:54.482401 24173 tablet_bootstrap.cc:492] T 78ddfc7b74644a0bbf8222d5f90e6658 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:54.482547 24173 ts_tablet_manager.cc:1367] T 78ddfc7b74644a0bbf8222d5f90e6658 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.105s user 0.001s sys 0.002s
I20240502 08:16:54.483340 24173 raft_consensus.cc:348] T 78ddfc7b74644a0bbf8222d5f90e6658 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.483552 24173 raft_consensus.cc:374] T 78ddfc7b74644a0bbf8222d5f90e6658 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.483610 24173 raft_consensus.cc:729] T 78ddfc7b74644a0bbf8222d5f90e6658 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.490911 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 4a57ed188315493198e95cefb6907ab6 (table default.Table_8c4dfc5c68e84b6ab815e9f333143678 [id=58ae4f5729f44a9c8be960be8a1a13a1]) successfully deleted
I20240502 08:16:54.491338 25012 tablet_replica.cc:330] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:54.492563 25012 raft_consensus.cc:2229] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:54.493485 25012 raft_consensus.cc:2258] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.494375 25012 ts_tablet_manager.cc:1875] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.499459 24181 ts_tablet_manager.cc:1888] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:54.499600 24181 log.cc:1196] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/aacdf506221044ffada0fec0c7d42d0b
I20240502 08:16:54.500021 24181 ts_tablet_manager.cc:1909] T aacdf506221044ffada0fec0c7d42d0b P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:54.500394 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 749) took 6364 ms. Trace:
I20240502 08:16:54.500499 24181 rpcz_store.cc:276] 0502 08:16:48.135716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.137054 (+ 1338us) service_pool.cc:225] Handling call
0502 08:16:54.304111 (+6167057us) ts_tablet_manager.cc:1120] Deleting tablet aacdf506221044ffada0fec0c7d42d0b
0502 08:16:54.304119 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.396184 (+ 92065us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.499318 (+103134us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.500372 (+ 1054us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6166906}
I20240502 08:16:54.500901 24181 tablet_replica.cc:330] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:54.501026 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet aacdf506221044ffada0fec0c7d42d0b (table default.Table_b2e1c08bfc404c7ebb2720631ad10495 [id=9480d59a76fc401e959a1e41caafc845]) successfully deleted
I20240502 08:16:54.501102 24181 raft_consensus.cc:2229] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:54.501294 24175 tablet_bootstrap.cc:492] T 78ddfc7b74644a0bbf8222d5f90e6658 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:54.501658 24175 ts_tablet_manager.cc:1367] T 78ddfc7b74644a0bbf8222d5f90e6658 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.188s user 0.007s sys 0.000s
I20240502 08:16:54.502308 24175 raft_consensus.cc:348] T 78ddfc7b74644a0bbf8222d5f90e6658 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.502563 24175 raft_consensus.cc:374] T 78ddfc7b74644a0bbf8222d5f90e6658 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.502658 24175 raft_consensus.cc:729] T 78ddfc7b74644a0bbf8222d5f90e6658 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.502925 24175 consensus_queue.cc:260] T 78ddfc7b74644a0bbf8222d5f90e6658 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.504060 24175 ts_tablet_manager.cc:1398] T 78ddfc7b74644a0bbf8222d5f90e6658 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:54.504352 24175 tablet_bootstrap.cc:492] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:54.516119 24180 ts_tablet_manager.cc:1888] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:54.516332 24180 log.cc:1196] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/e549cad1b3be4b28abb0bf5af720eb52
I20240502 08:16:54.516759 24180 ts_tablet_manager.cc:1909] T e549cad1b3be4b28abb0bf5af720eb52 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:54.517102 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 907) took 3504 ms. Trace:
I20240502 08:16:54.519773 24173 consensus_queue.cc:260] T 78ddfc7b74644a0bbf8222d5f90e6658 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.519620 24174 tablet_bootstrap.cc:492] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:54.520175 24174 ts_tablet_manager.cc:1367] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.077s user 0.002s sys 0.001s
I20240502 08:16:54.520078 24180 rpcz_store.cc:276] 0502 08:16:51.012586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.878957 (+866371us) service_pool.cc:225] Handling call
0502 08:16:54.356926 (+2477969us) ts_tablet_manager.cc:1120] Deleting tablet e549cad1b3be4b28abb0bf5af720eb52
0502 08:16:54.356936 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.449941 (+ 93005us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.516007 (+ 66066us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.517086 (+ 1079us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40192,"tablet-delete.queue_time_us":2477862}
I20240502 08:16:54.521206 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet e549cad1b3be4b28abb0bf5af720eb52 (table default.Table_b461b560f3094407a1766253cfc2ece7 [id=4b9348e7a75242788274eee8f4c94838]) successfully deleted
I20240502 08:16:54.523291 24175 tablet_bootstrap.cc:654] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.527671 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_00e3ffabca274246a81fb18ce0d772a4" } new_table_name: "default.Table_9845c5ca37bf4f4486319edc1e088aef" modify_external_catalogs: true
I20240502 08:16:54.529397 25296 raft_consensus.cc:482] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:54.529613 25296 raft_consensus.cc:504] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.530313 25296 leader_election.cc:288] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:54.530833 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59c2f93d6514c9cbf358ab96f1a06b5" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:54.531054 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59c2f93d6514c9cbf358ab96f1a06b5" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:54.531289 24057 raft_consensus.cc:2454] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
W20240502 08:16:54.531805 23723 leader_election.cc:341] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:54.532255 23723 leader_election.cc:302] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:54.532498 25296 raft_consensus.cc:2781] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:54.532568 25296 raft_consensus.cc:482] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:54.532627 25296 raft_consensus.cc:3037] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.532892 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0ef99b98ca9047769c22d238d860cd21" } new_table_name: "default.Table_a4b017025fc743b2a7d81ca07433a039" modify_external_catalogs: true
I20240502 08:16:54.534920 25296 raft_consensus.cc:504] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.535683 25296 leader_election.cc:288] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:54.538007 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59c2f93d6514c9cbf358ab96f1a06b5" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:54.538538 23723 leader_election.cc:341] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:54.537433 24174 raft_consensus.cc:348] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.540448 24174 raft_consensus.cc:374] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.540983 24174 raft_consensus.cc:729] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.541700 24174 consensus_queue.cc:260] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.542785 24174 ts_tablet_manager.cc:1398] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20240502 08:16:54.546483 24180 tablet_replica.cc:330] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:54.546697 24180 raft_consensus.cc:2229] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:54.546993 24180 raft_consensus.cc:2258] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.547578 24180 ts_tablet_manager.cc:1875] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.548683 25012 ts_tablet_manager.cc:1888] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:54.552955 24180 ts_tablet_manager.cc:1888] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:54.553061 24180 log.cc:1196] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/5a1e208733844829b13222e00a432439
I20240502 08:16:54.553215 24181 raft_consensus.cc:2258] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.553884 24181 ts_tablet_manager.cc:1875] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.554620 24173 ts_tablet_manager.cc:1398] T 78ddfc7b74644a0bbf8222d5f90e6658 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.072s user 0.000s sys 0.002s
I20240502 08:16:54.555752 25012 log.cc:1196] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/c17e1be6ef294769a01981f4d361be21
I20240502 08:16:54.556151 25012 ts_tablet_manager.cc:1909] T c17e1be6ef294769a01981f4d361be21 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:54.556983 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 736) took 6391 ms. Trace:
I20240502 08:16:54.557128 25012 rpcz_store.cc:276] 0502 08:16:48.165878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.173747 (+ 7869us) service_pool.cc:225] Handling call
0502 08:16:54.491269 (+6317522us) ts_tablet_manager.cc:1120] Deleting tablet c17e1be6ef294769a01981f4d361be21
0502 08:16:54.491280 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.540429 (+ 49149us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.548593 (+ 8164us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.556963 (+ 8370us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6317326}
I20240502 08:16:54.557741 25012 tablet_replica.cc:330] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:54.557858 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet c17e1be6ef294769a01981f4d361be21 (table default.Table_498b372c15a04b70b87bd67b543501cf [id=38e955d6bd644d24a7c45172ab60282b]) successfully deleted
I20240502 08:16:54.552780 25433 tablet.cc:1722] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:54.553990 24175 tablet_bootstrap.cc:492] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:54.558104 24175 ts_tablet_manager.cc:1367] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.054s user 0.000s sys 0.005s
I20240502 08:16:54.558719 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59c2f93d6514c9cbf358ab96f1a06b5" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:54.558847 24173 tablet_bootstrap.cc:492] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:54.558776 24175 raft_consensus.cc:348] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.558920 24175 raft_consensus.cc:374] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.558915 24057 raft_consensus.cc:3037] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.558964 24175 raft_consensus.cc:729] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.558132 25012 raft_consensus.cc:2229] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:54.559859 24175 consensus_queue.cc:260] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.560314 25012 pending_rounds.cc:62] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b: Trying to abort 1 pending ops.
I20240502 08:16:54.560402 25012 pending_rounds.cc:68] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156476816142336 op_type: NO_OP noop_request { }
I20240502 08:16:54.562364 24180 ts_tablet_manager.cc:1909] T 5a1e208733844829b13222e00a432439 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:54.566677 24174 tablet_bootstrap.cc:492] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:54.566843 25012 raft_consensus.cc:2866] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:54.567004 25012 raft_consensus.cc:2258] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.560791 24175 ts_tablet_manager.cc:1398] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:16:54.567961 24175 tablet_bootstrap.cc:492] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:54.568728 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0902a73da6e34e8e9cd59bb53a36ec0e consensus state has no leader
W20240502 08:16:54.568985 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 20a0519367ad4d6fae687aaeef0a8f44 consensus state has no leader
I20240502 08:16:54.569334 24175 tablet_bootstrap.cc:654] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.569300 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2bcd9c267ebf4fd3a2763c589938275f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:54.569993 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2bcd9c267ebf4fd3a2763c589938275f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.570407 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 908) took 3557 ms. Trace:
I20240502 08:16:54.570477 24180 rpcz_store.cc:276] 0502 08:16:51.012996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.879069 (+866073us) service_pool.cc:225] Handling call
0502 08:16:54.522677 (+2643608us) ts_tablet_manager.cc:1120] Deleting tablet 5a1e208733844829b13222e00a432439
0502 08:16:54.546416 (+ 23739us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.550334 (+ 3918us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.552884 (+ 2550us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.570389 (+ 17505us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2643499}
I20240502 08:16:54.570804 24174 tablet_bootstrap.cc:654] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.571408 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 5a1e208733844829b13222e00a432439 (table default.Table_7c04460336ea48e79db571d95cdb1bcc [id=8ea9f7fff43846dfa8310549e8e57f2f]) successfully deleted
I20240502 08:16:54.570812 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 913) took 2785 ms. Trace:
I20240502 08:16:54.573035 24180 rpcz_store.cc:276] 0502 08:16:51.785447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.883300 (+ 97853us) service_pool.cc:225] Handling call
0502 08:16:54.570697 (+2687397us) ts_tablet_manager.cc:1120] Deleting tablet fe5e08ad6af2442bb0a8fcc80a99b532
0502 08:16:54.570706 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.570799 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2687098}
I20240502 08:16:54.573376 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 914) took 2785 ms. Trace:
I20240502 08:16:54.573462 24180 rpcz_store.cc:276] 0502 08:16:51.787541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.883619 (+ 96078us) service_pool.cc:225] Handling call
0502 08:16:54.573255 (+2689636us) ts_tablet_manager.cc:1120] Deleting tablet 29f6c3cea4b14364b6d073a92171119d
0502 08:16:54.573262 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.573361 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2689523}
W20240502 08:16:54.573518 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet fe5e08ad6af2442bb0a8fcc80a99b532 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe5e08ad6af2442bb0a8fcc80a99b532 already in progress: creating tablet
I20240502 08:16:54.573769 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 916) took 2775 ms. Trace:
I20240502 08:16:54.573824 24180 rpcz_store.cc:276] 0502 08:16:51.797981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.884351 (+ 86370us) service_pool.cc:225] Handling call
0502 08:16:54.573688 (+2689337us) ts_tablet_manager.cc:1120] Deleting tablet 303a0a3748ee422faab856e1c4bff66d
0502 08:16:54.573695 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.573757 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2689183}
I20240502 08:16:54.574106 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 917) took 2776 ms. Trace:
I20240502 08:16:54.574162 24180 rpcz_store.cc:276] 0502 08:16:51.798090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.884522 (+ 86432us) service_pool.cc:225] Handling call
0502 08:16:54.574031 (+2689509us) ts_tablet_manager.cc:1120] Deleting tablet f685cbe705b149e69f9536e270a74a42
0502 08:16:54.574038 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.574095 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2689390}
W20240502 08:16:54.574306 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 29f6c3cea4b14364b6d073a92171119d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29f6c3cea4b14364b6d073a92171119d already in progress: creating tablet
I20240502 08:16:54.574399 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 899) took 3844 ms. Trace:
W20240502 08:16:54.574478 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 303a0a3748ee422faab856e1c4bff66d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 303a0a3748ee422faab856e1c4bff66d already in progress: creating tablet
I20240502 08:16:54.574453 24180 rpcz_store.cc:276] 0502 08:16:50.729471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.836361 (+1106890us) service_pool.cc:225] Handling call
0502 08:16:54.574317 (+2737956us) ts_tablet_manager.cc:1120] Deleting tablet c48cea94856f43b6861c3ba360414888
0502 08:16:54.574324 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.574376 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2680826}
I20240502 08:16:54.574654 24180 tablet_replica.cc:330] T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:54.574819 24180 raft_consensus.cc:2229] T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:54.574955 24180 raft_consensus.cc:2258] T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:54.575014 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet f685cbe705b149e69f9536e270a74a42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f685cbe705b149e69f9536e270a74a42 already in progress: creating tablet
W20240502 08:16:54.575189 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet c48cea94856f43b6861c3ba360414888 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c48cea94856f43b6861c3ba360414888 already in progress: creating tablet
I20240502 08:16:54.575933 23605 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 001e3f4f72f64dc69dbd521724912f07 (table default.Table_d7cff78803d44e5aba3b50cd8ff11e92 [id=5c21117955d946418c656330984ae747]). Expected version 3 got 0
I20240502 08:16:54.575963 25012 ts_tablet_manager.cc:1875] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.576853 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7ff62874389f42d0ba37aba1307ea27b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:54.577383 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ff62874389f42d0ba37aba1307ea27b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.577435 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 78ddfc7b74644a0bbf8222d5f90e6658 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:54.581465 23607 catalog_manager.cc:5484] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67). New cstate: current_term: 1 leader_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:54.582098 25455 tablet.cc:1722] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:54.587800 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2223dafc11344718b775bac0d51fa2b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:54.590252 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2223dafc11344718b775bac0d51fa2b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.588672 24180 ts_tablet_manager.cc:1875] T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.595671 24057 raft_consensus.cc:2454] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:54.596643 23723 leader_election.cc:302] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:54.597436 25418 raft_consensus.cc:2781] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:54.597909 25418 raft_consensus.cc:686] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:54.598783 25418 consensus_queue.cc:237] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.600715 24180 ts_tablet_manager.cc:1888] T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:54.600818 24180 log.cc:1196] T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/f797d3c6bdc4493398954fbbf9943971
I20240502 08:16:54.601126 24180 ts_tablet_manager.cc:1909] T f797d3c6bdc4493398954fbbf9943971 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:54.609160 25012 ts_tablet_manager.cc:1888] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:54.610240 25012 log.cc:1196] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/319fabf91458438ba5e40ab68b323055
I20240502 08:16:54.610780 25012 ts_tablet_manager.cc:1909] T 319fabf91458438ba5e40ab68b323055 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:54.613541 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 737) took 6439 ms. Trace:
I20240502 08:16:54.617206 24173 tablet_bootstrap.cc:654] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.617888 25012 rpcz_store.cc:276] 0502 08:16:48.174134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.281978 (+107844us) service_pool.cc:225] Handling call
0502 08:16:54.557681 (+6275703us) ts_tablet_manager.cc:1120] Deleting tablet 319fabf91458438ba5e40ab68b323055
0502 08:16:54.557694 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.596265 (+ 38571us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.609065 (+ 12800us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.613515 (+ 4450us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6275452}
I20240502 08:16:54.618629 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 319fabf91458438ba5e40ab68b323055 (table default.Table_becb097785284ffba69cdbe40e6480ab [id=4454421c70094eed8a8f3e42dcc8a524]) successfully deleted
I20240502 08:16:54.618713 25012 tablet_replica.cc:330] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:54.618891 25012 raft_consensus.cc:2229] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:54.619621 25012 pending_rounds.cc:62] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b: Trying to abort 1 pending ops.
I20240502 08:16:54.620007 25012 pending_rounds.cc:68] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156462015254528 op_type: NO_OP noop_request { }
I20240502 08:16:54.620285 25012 raft_consensus.cc:2866] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:54.620492 25012 raft_consensus.cc:2258] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.620939 24175 tablet_bootstrap.cc:492] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:54.621062 25012 ts_tablet_manager.cc:1875] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.621089 24175 ts_tablet_manager.cc:1367] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.009s
I20240502 08:16:54.621888 25158 raft_consensus.cc:482] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:54.622361 24175 raft_consensus.cc:348] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.622550 24175 raft_consensus.cc:374] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.622603 24175 raft_consensus.cc:729] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.622265 25158 raft_consensus.cc:504] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.622781 24175 consensus_queue.cc:260] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.624258 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b8d81f071d74b218fcfd4f4e8c20bab" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:54.624758 23723 leader_election.cc:341] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:54.627408 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b8d81f071d74b218fcfd4f4e8c20bab" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:54.627648 24057 raft_consensus.cc:2454] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:54.628430 24175 ts_tablet_manager.cc:1398] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 08:16:54.630774 24175 tablet_bootstrap.cc:492] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:54.631073 23723 leader_election.cc:302] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:54.631296 25296 raft_consensus.cc:2781] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:54.631383 25158 leader_election.cc:288] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:54.631399 25296 raft_consensus.cc:482] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:54.632683 25296 raft_consensus.cc:3037] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.633981 24175 tablet_bootstrap.cc:654] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.635232 25296 raft_consensus.cc:504] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.635890 25296 leader_election.cc:288] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:54.636719 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b8d81f071d74b218fcfd4f4e8c20bab" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:54.636813 24173 tablet_bootstrap.cc:492] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:54.636886 24057 raft_consensus.cc:3037] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.637063 24173 ts_tablet_manager.cc:1367] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.078s user 0.000s sys 0.004s
I20240502 08:16:54.637748 24173 raft_consensus.cc:348] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.639990 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b8d81f071d74b218fcfd4f4e8c20bab" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:54.640533 23723 leader_election.cc:341] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:54.642697 24174 tablet_bootstrap.cc:492] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:54.642989 24174 ts_tablet_manager.cc:1367] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.005s
I20240502 08:16:54.644546 24174 raft_consensus.cc:348] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.644966 24174 raft_consensus.cc:374] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.645056 24174 raft_consensus.cc:729] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.645625 24174 consensus_queue.cc:260] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.646888 24174 ts_tablet_manager.cc:1398] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:54.644300 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7c9232b45ef64cb58387478ac1b14b45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:54.648178 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c9232b45ef64cb58387478ac1b14b45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.650171 24175 tablet_bootstrap.cc:492] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:54.650326 24175 ts_tablet_manager.cc:1367] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240502 08:16:54.651206 24175 raft_consensus.cc:348] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.651437 24175 raft_consensus.cc:374] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.651494 24175 raft_consensus.cc:729] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.651712 24175 consensus_queue.cc:260] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.651958 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b61b626a7be44b199bf2359795269a93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:54.652470 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b61b626a7be44b199bf2359795269a93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.657471 24175 ts_tablet_manager.cc:1398] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:16:54.661463 24175 tablet_bootstrap.cc:492] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:54.661820 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 928) took 2499 ms. Trace:
I20240502 08:16:54.661931 24180 rpcz_store.cc:276] 0502 08:16:52.161999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.162184 (+ 185us) service_pool.cc:225] Handling call
0502 08:16:54.574618 (+2412434us) ts_tablet_manager.cc:1120] Deleting tablet f797d3c6bdc4493398954fbbf9943971
0502 08:16:54.574625 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.598072 (+ 23447us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.600645 (+ 2573us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.661780 (+ 61135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2412218}
I20240502 08:16:54.662283 24180 tablet_replica.cc:330] T 7a97427e819b4826b1ed89e6d021c8d4 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:54.662487 24180 raft_consensus.cc:2229] T 7a97427e819b4826b1ed89e6d021c8d4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:54.662637 24180 raft_consensus.cc:2258] T 7a97427e819b4826b1ed89e6d021c8d4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.662896 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet f797d3c6bdc4493398954fbbf9943971 (table default.Table_7f44f7f1aed6460cb1e73392da0987b8 [id=07d8592503e843e18faedfcce845a974]) successfully deleted
I20240502 08:16:54.662988 24175 tablet_bootstrap.cc:654] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.663193 24180 ts_tablet_manager.cc:1875] T 7a97427e819b4826b1ed89e6d021c8d4 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.674611 24180 ts_tablet_manager.cc:1888] T 7a97427e819b4826b1ed89e6d021c8d4 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:54.674762 24180 log.cc:1196] T 7a97427e819b4826b1ed89e6d021c8d4 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/7a97427e819b4826b1ed89e6d021c8d4
I20240502 08:16:54.675138 24180 ts_tablet_manager.cc:1909] T 7a97427e819b4826b1ed89e6d021c8d4 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:54.677531 25012 ts_tablet_manager.cc:1888] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:54.677670 23759 tablet_service.cc:1467] Processing CreateTablet for tablet c4f9e61686dc417bb5a88f96b4231f2e (DEFAULT_TABLE table=default.Table_0ca7df7eee144c08a8aa1c5cc0f008c4 [id=e541782bf7bb4a069324027aa554e2ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.679009 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 26ee4a0c34c347178fb4b82782bc49b5 (DEFAULT_TABLE table=default.Table_fae2341935c6408082d8654e1d8a3f85 [id=5342e70efc234c4cad807a095f6dc5b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.680481 25012 log.cc:1196] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/b8b1fa17dad143f0a7f1751e651eb81a
I20240502 08:16:54.678920 23894 tablet_service.cc:1467] Processing CreateTablet for tablet c4f9e61686dc417bb5a88f96b4231f2e (DEFAULT_TABLE table=default.Table_0ca7df7eee144c08a8aa1c5cc0f008c4 [id=e541782bf7bb4a069324027aa554e2ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.680927 25012 ts_tablet_manager.cc:1909] T b8b1fa17dad143f0a7f1751e651eb81a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:54.681073 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f9e61686dc417bb5a88f96b4231f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.681066 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 33928d65045c42f2b8c12a1f52dcf206 (DEFAULT_TABLE table=default.Table_08cf295230eb40d7a7807bbd049f127b [id=da35e70749ea4ee3b0cdee1a0a2bf77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.681824 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 738) took 6399 ms. Trace:
I20240502 08:16:54.681352 24174 tablet_bootstrap.cc:492] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:54.681897 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 26ee4a0c34c347178fb4b82782bc49b5 (DEFAULT_TABLE table=default.Table_fae2341935c6408082d8654e1d8a3f85 [id=5342e70efc234c4cad807a095f6dc5b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.681928 25012 rpcz_store.cc:276] 0502 08:16:48.282502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.282551 (+ 49us) service_pool.cc:225] Handling call
0502 08:16:54.618662 (+6336111us) ts_tablet_manager.cc:1120] Deleting tablet b8b1fa17dad143f0a7f1751e651eb81a
0502 08:16:54.618672 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.643529 (+ 24857us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.677396 (+ 33867us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.681804 (+ 4408us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6335916}
I20240502 08:16:54.682235 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ee4a0c34c347178fb4b82782bc49b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.682281 25012 tablet_replica.cc:330] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:54.682492 25012 raft_consensus.cc:2229] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:54.682677 25012 raft_consensus.cc:2258] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.683212 25012 ts_tablet_manager.cc:1875] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.685845 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet b8b1fa17dad143f0a7f1751e651eb81a (table default.Table_d4bd37f8c6e54ecf82cd247846aa06d0 [id=57722a7341c541b19efa213bcf48eb53]) successfully deleted
I20240502 08:16:54.686431 24175 tablet_bootstrap.cc:492] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:54.686560 24175 ts_tablet_manager.cc:1367] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240502 08:16:54.687355 24175 raft_consensus.cc:348] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.639237 24057 raft_consensus.cc:2454] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:54.687584 24175 raft_consensus.cc:374] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.687635 24175 raft_consensus.cc:729] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.687606 24173 raft_consensus.cc:374] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.677670 24028 tablet_service.cc:1467] Processing CreateTablet for tablet c4f9e61686dc417bb5a88f96b4231f2e (DEFAULT_TABLE table=default.Table_0ca7df7eee144c08a8aa1c5cc0f008c4 [id=e541782bf7bb4a069324027aa554e2ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.679852 24022 tablet_service.cc:1467] Processing CreateTablet for tablet 26ee4a0c34c347178fb4b82782bc49b5 (DEFAULT_TABLE table=default.Table_fae2341935c6408082d8654e1d8a3f85 [id=5342e70efc234c4cad807a095f6dc5b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.687826 24175 consensus_queue.cc:260] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.688066 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ee4a0c34c347178fb4b82782bc49b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.688154 23723 leader_election.cc:302] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:54.688676 25158 raft_consensus.cc:2781] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:54.688799 25158 raft_consensus.cc:686] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:54.689033 25158 consensus_queue.cc:237] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.692929 24174 tablet_bootstrap.cc:654] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.693372 24175 ts_tablet_manager.cc:1398] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:16:54.694314 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f9e61686dc417bb5a88f96b4231f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.694530 25408 raft_consensus.cc:482] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:54.694643 25408 raft_consensus.cc:504] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.695284 25408 leader_election.cc:288] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:54.695852 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577b149f52e44d5d9a9f6b9d85a6864d" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:54.696225 24175 tablet_bootstrap.cc:492] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:54.697837 24175 tablet_bootstrap.cc:654] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.700898 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577b149f52e44d5d9a9f6b9d85a6864d" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:54.701169 23788 raft_consensus.cc:2379] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in this term.
I20240502 08:16:54.701786 23858 leader_election.cc:302] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:54.702059 25357 raft_consensus.cc:3037] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.702827 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 930) took 2539 ms. Trace:
I20240502 08:16:54.702926 24180 rpcz_store.cc:276] 0502 08:16:52.163778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.163941 (+ 163us) service_pool.cc:225] Handling call
0502 08:16:54.662236 (+2498295us) ts_tablet_manager.cc:1120] Deleting tablet 7a97427e819b4826b1ed89e6d021c8d4
0502 08:16:54.662246 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.669504 (+ 7258us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.674491 (+ 4987us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.702804 (+ 28313us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2494477}
I20240502 08:16:54.703464 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ee4a0c34c347178fb4b82782bc49b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.703622 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33928d65045c42f2b8c12a1f52dcf206. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.704526 24180 tablet_replica.cc:330] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:54.704643 25357 raft_consensus.cc:2726] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:54.705006 23972 heartbeater.cc:347] Connected to a master server at 127.22.225.125:36189
I20240502 08:16:54.704732 24180 raft_consensus.cc:2229] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:54.705276 24180 raft_consensus.cc:2258] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.706463 24180 ts_tablet_manager.cc:1875] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.706740 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 7a97427e819b4826b1ed89e6d021c8d4 (table default.Table_e66f5d1a72b347f7a7b39bddbf74b263 [id=9b0006a9ac9b437fa608c587d73b09f5]) successfully deleted
I20240502 08:16:54.707031 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f9e61686dc417bb5a88f96b4231f2e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:54.709846 23993 consensus_peers.cc:485] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f797d3c6bdc4493398954fbbf9943971. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:54.687737 24173 raft_consensus.cc:729] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.715945 24173 consensus_queue.cc:260] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.716550 24173 ts_tablet_manager.cc:1398] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.079s user 0.001s sys 0.000s
I20240502 08:16:54.716889 24173 tablet_bootstrap.cc:492] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:54.726668 25012 ts_tablet_manager.cc:1888] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:54.727454 25012 log.cc:1196] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/98a96292b28a4b0aad88fe0a96730a68
I20240502 08:16:54.728075 25012 ts_tablet_manager.cc:1909] T 98a96292b28a4b0aad88fe0a96730a68 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:54.729398 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 739) took 6446 ms. Trace:
I20240502 08:16:54.730420 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet bb9b75d6be924d0291f94fc4a439dea6
I20240502 08:16:54.731519 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 2c28c47b4f814573971227b2208d1f83
I20240502 08:16:54.731536 25012 rpcz_store.cc:276] 0502 08:16:48.282868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.282913 (+ 45us) service_pool.cc:225] Handling call
0502 08:16:54.682233 (+6399320us) ts_tablet_manager.cc:1120] Deleting tablet 98a96292b28a4b0aad88fe0a96730a68
0502 08:16:54.682241 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.713037 (+ 30796us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.726524 (+ 13487us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.729373 (+ 2849us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6399174}
I20240502 08:16:54.731935 25012 tablet_replica.cc:330] T aacdf506221044ffada0fec0c7d42d0b P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:54.733537 24175 tablet_bootstrap.cc:492] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:54.733709 24175 ts_tablet_manager.cc:1367] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.000s
I20240502 08:16:54.733809 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 98a96292b28a4b0aad88fe0a96730a68 (table default.Table_2849e243f1a94b139d3234f0c8e3c755 [id=120f59a131ed47bb8f56dbc3a4ebec3b]) successfully deleted
I20240502 08:16:54.734025 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet b59c2f93d6514c9cbf358ab96f1a06b5
I20240502 08:16:54.734100 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet bb9b75d6be924d0291f94fc4a439dea6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:54.734304 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c28c47b4f814573971227b2208d1f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:54.734417 23609 catalog_manager.cc:6554] ReplaceTablet: tablet b59c2f93d6514c9cbf358ab96f1a06b5 deleted and replaced by tablet 7f7ea2d3bc8a494ebdefa6de8ca389e0
I20240502 08:16:54.734609 24175 raft_consensus.cc:348] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.734750 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet b59c2f93d6514c9cbf358ab96f1a06b5 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f7ea2d3bc8a494ebdefa6de8ca389e0) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:54.734812 24175 raft_consensus.cc:374] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.734772 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b3315254654d4234a879012f47feeb19" } new_table_name: "default.Table_1a803b0403c34760a4c27715e58b757c" modify_external_catalogs: true
I20240502 08:16:54.734865 24175 raft_consensus.cc:729] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.735145 24175 consensus_queue.cc:260] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.736129 24180 ts_tablet_manager.cc:1888] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:54.736295 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 31775513f3a446b08c0aac25ae0cab20
I20240502 08:16:54.737182 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6140ac5b9ad042a08e94f15c3c2efd16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:54.737313 23602 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet a2da8b9d3f374d8a812336057c0ce7d2 (table default.Table_3edd27da6927478281e76361718425a3 [id=76b58ce209a24b9a95b405c1c50de3ca]). Expected version 3 got 0
I20240502 08:16:54.737529 23602 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 7ed919909289410e867881baab841f53 (table default.Table_2edad641920b44ce9a4d5b7503f0279a [id=ee3912407a934444ad2c62f7b83d93ac]). Expected version 2 got 0
W20240502 08:16:54.737764 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6140ac5b9ad042a08e94f15c3c2efd16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.737936 23602 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet edd7244b9f5741a8aa1c6835fe544676 (table default.Table_4c6989f7e6bb4e99b447a3c1188f5b92 [id=9775ed3c642a4d7e81d5abc48acca163]). Expected version 2 got 0
I20240502 08:16:54.739065 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 37b499184f31454c96f66cfdf9143d14
I20240502 08:16:54.739738 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_36102dd8079049bcb0c8f32890c742fb" } new_table_name: "default.Table_6103c4da9979479ba3fde1ea61b4d7d7" modify_external_catalogs: true
I20240502 08:16:54.741381 24175 ts_tablet_manager.cc:1398] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:16:54.741416 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f17a7a3f2d4547799bdbea376e3771b2" } new_table_name: "default.Table_3d8235c36ecc44c6bae973b3cbafa696" modify_external_catalogs: true
I20240502 08:16:54.744355 24180 log.cc:1196] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/c5c377e037894728ac084b6500ad20de
I20240502 08:16:54.744765 24180 ts_tablet_manager.cc:1909] T c5c377e037894728ac084b6500ad20de P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:54.745121 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 927) took 2583 ms. Trace:
I20240502 08:16:54.745201 24180 rpcz_store.cc:276] 0502 08:16:52.161891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.162486 (+ 595us) service_pool.cc:225] Handling call
0502 08:16:54.703199 (+2540713us) ts_tablet_manager.cc:1120] Deleting tablet c5c377e037894728ac084b6500ad20de
0502 08:16:54.704474 (+ 1275us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.719245 (+ 14771us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.736007 (+ 16762us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.745096 (+ 9089us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2535269}
I20240502 08:16:54.745558 24180 tablet_replica.cc:330] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:54.745784 24180 raft_consensus.cc:2229] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:54.745970 24180 raft_consensus.cc:2258] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.746235 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet 31775513f3a446b08c0aac25ae0cab20 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:54.746546 23752 tablet_service.cc:1514] Processing DeleteTablet for tablet 78ddfc7b74644a0bbf8222d5f90e6658 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:54.746809 23752 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f02efa659e457ab57bc32ed3f66bb9 with delete_type TABLET_DATA_DELETED (replaced by tablet df06e0edb57f45e5ac71b568c14769cd) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:54.746861 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b8d81f071d74b218fcfd4f4e8c20bab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:54.746806 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet ee022e8e86f8477a8fa1e58d52644b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:54.747133 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e7432f1df94431a83cf68356e3e78a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:54.747511 24175 tablet_bootstrap.cc:492] T 751808dfc1964b539f6f836d786156aa P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:54.748432 25012 raft_consensus.cc:2229] T aacdf506221044ffada0fec0c7d42d0b P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:54.749058 24180 ts_tablet_manager.cc:1875] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.749115 25012 raft_consensus.cc:2258] T aacdf506221044ffada0fec0c7d42d0b P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.749506 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b499184f31454c96f66cfdf9143d14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:54.749702 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d04312bd81f46ed8c49c42b1af1aec9 with delete_type TABLET_DATA_DELETED (replaced by tablet b45f3233dae94be1a1d0d8fb426d7f2c) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:54.750335 24175 tablet_bootstrap.cc:654] T 751808dfc1964b539f6f836d786156aa P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.750952 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_37c06b36010d4694830aa2c23281c13c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:54.751457 25012 ts_tablet_manager.cc:1875] T aacdf506221044ffada0fec0c7d42d0b P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:54.751489 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37c06b36010d4694830aa2c23281c13c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.751600 24173 tablet_bootstrap.cc:654] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.753732 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_536141575d2c49fe807d39c1f38e3bb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:54.754206 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_536141575d2c49fe807d39c1f38e3bb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.733814 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 870edf582ab14ad1b57c4c4469b19954
I20240502 08:16:54.759868 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_bd032329c0574a53a1d3b7d977c4783b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:54.760470 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd032329c0574a53a1d3b7d977c4783b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.761843 24022 tablet_service.cc:1514] Processing DeleteTablet for tablet bb9b75d6be924d0291f94fc4a439dea6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:54.762039 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet b59c2f93d6514c9cbf358ab96f1a06b5 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f7ea2d3bc8a494ebdefa6de8ca389e0) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:54.762043 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c28c47b4f814573971227b2208d1f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:54.763809 24173 tablet_bootstrap.cc:492] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:54.763957 24173 ts_tablet_manager.cc:1367] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20240502 08:16:54.765233 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet c5c377e037894728ac084b6500ad20de (table default.Table_f8db538a85994d0dbee1b1cbbabf5e16 [id=57a76411ce5b4a9c8b973ce5d2da53ab]) successfully deleted
I20240502 08:16:54.765326 25012 ts_tablet_manager.cc:1888] T aacdf506221044ffada0fec0c7d42d0b P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:54.765476 25012 log.cc:1196] T aacdf506221044ffada0fec0c7d42d0b P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/aacdf506221044ffada0fec0c7d42d0b
I20240502 08:16:54.766785 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 33928d65045c42f2b8c12a1f52dcf206 (DEFAULT_TABLE table=default.Table_08cf295230eb40d7a7807bbd049f127b [id=da35e70749ea4ee3b0cdee1a0a2bf77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.767251 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33928d65045c42f2b8c12a1f52dcf206. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.767310 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet 31775513f3a446b08c0aac25ae0cab20 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:54.767375 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet 31775513f3a446b08c0aac25ae0cab20 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:54.767508 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b499184f31454c96f66cfdf9143d14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:54.767848 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 5f083ade55de4b6487a663f8daf526e8 (DEFAULT_TABLE table=default.Table_cc09d7ab929c4f4ba017a922d1b1cad9 [id=30b20b2d45764c21baa726be8a5c0107]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.767925 24037 tablet_service.cc:1467] Processing CreateTablet for tablet fe42e6c8c85648a0be51a6aa6211d3d6 (DEFAULT_TABLE table=default.Table_0c12355d49954405ae0e2b245ccdbc6a [id=5fc2baf87f464b2aa81c0239900326af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.768070 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b499184f31454c96f66cfdf9143d14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:54.768198 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f083ade55de4b6487a663f8daf526e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.768303 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe42e6c8c85648a0be51a6aa6211d3d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.768396 24036 tablet_service.cc:1467] Processing CreateTablet for tablet ed97fc6238be4d139587d6261d8460e6 (DEFAULT_TABLE table=default.Table_066d340c016f4dfea60385ea0b28b7a0 [id=8ab591c5296e49fdab9c7e8f3ae123cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.768656 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed97fc6238be4d139587d6261d8460e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.769816 23895 tablet_service.cc:1467] Processing CreateTablet for tablet ed97fc6238be4d139587d6261d8460e6 (DEFAULT_TABLE table=default.Table_066d340c016f4dfea60385ea0b28b7a0 [id=8ab591c5296e49fdab9c7e8f3ae123cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.770154 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed97fc6238be4d139587d6261d8460e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.771294 24029 tablet_service.cc:1467] Processing CreateTablet for tablet d14f52d78aaf4230889493b3a0be4f2a (DEFAULT_TABLE table=default.Table_90c5920a0df74de598d87d8ce85c91ed [id=22ec77d716aa4e5ab64b1168c1899d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.771575 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d14f52d78aaf4230889493b3a0be4f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.773226 23893 tablet_service.cc:1467] Processing CreateTablet for tablet fe42e6c8c85648a0be51a6aa6211d3d6 (DEFAULT_TABLE table=default.Table_0c12355d49954405ae0e2b245ccdbc6a [id=5fc2baf87f464b2aa81c0239900326af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.773345 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 870edf582ab14ad1b57c4c4469b19954 (DEFAULT_TABLE table=default.Table_7c85926073de4625bef40b9aa50bb805 [id=a1cded474ff5409b8851203364f65fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.773536 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe42e6c8c85648a0be51a6aa6211d3d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.773617 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870edf582ab14ad1b57c4c4469b19954. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.773763 23900 tablet_service.cc:1467] Processing CreateTablet for tablet d14f52d78aaf4230889493b3a0be4f2a (DEFAULT_TABLE table=default.Table_90c5920a0df74de598d87d8ce85c91ed [id=22ec77d716aa4e5ab64b1168c1899d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.774040 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d14f52d78aaf4230889493b3a0be4f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.777660 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet 870edf582ab14ad1b57c4c4469b19954 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:54.777935 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 870edf582ab14ad1b57c4c4469b19954 (DEFAULT_TABLE table=default.Table_7c85926073de4625bef40b9aa50bb805 [id=a1cded474ff5409b8851203364f65fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.778290 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870edf582ab14ad1b57c4c4469b19954. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.778360 23768 tablet_service.cc:1467] Processing CreateTablet for tablet ed97fc6238be4d139587d6261d8460e6 (DEFAULT_TABLE table=default.Table_066d340c016f4dfea60385ea0b28b7a0 [id=8ab591c5296e49fdab9c7e8f3ae123cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.778580 23752 tablet_service.cc:1467] Processing CreateTablet for tablet e8bb9a499b254ae69ba752b949f409c8 (DEFAULT_TABLE table=default.Table_d66d58dae2d54c2c8befedd495a51606 [id=a589c3abe0c74ce8aa0d44c42dbcb53a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.778729 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed97fc6238be4d139587d6261d8460e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.778756 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 870edf582ab14ad1b57c4c4469b19954 (DEFAULT_TABLE table=default.Table_7c85926073de4625bef40b9aa50bb805 [id=a1cded474ff5409b8851203364f65fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.778899 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8bb9a499b254ae69ba752b949f409c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.779083 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 870edf582ab14ad1b57c4c4469b19954 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:54.778601 23751 tablet_service.cc:1467] Processing CreateTablet for tablet d14f52d78aaf4230889493b3a0be4f2a (DEFAULT_TABLE table=default.Table_90c5920a0df74de598d87d8ce85c91ed [id=22ec77d716aa4e5ab64b1168c1899d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.779479 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d14f52d78aaf4230889493b3a0be4f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.781713 23886 tablet_service.cc:1514] Processing DeleteTablet for tablet 870edf582ab14ad1b57c4c4469b19954 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:54.781855 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 5f083ade55de4b6487a663f8daf526e8 (DEFAULT_TABLE table=default.Table_cc09d7ab929c4f4ba017a922d1b1cad9 [id=30b20b2d45764c21baa726be8a5c0107]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.782150 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f083ade55de4b6487a663f8daf526e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.782119 25158 raft_consensus.cc:482] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:54.782238 25158 raft_consensus.cc:504] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.782856 25158 leader_election.cc:288] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:54.783632 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ddfc7b74644a0bbf8222d5f90e6658" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:54.783843 23914 raft_consensus.cc:2454] T 78ddfc7b74644a0bbf8222d5f90e6658 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:54.784287 23723 leader_election.cc:302] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:54.784528 25158 raft_consensus.cc:2781] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:54.784587 25158 raft_consensus.cc:482] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:54.784639 25158 raft_consensus.cc:3037] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.785151 23749 tablet_service.cc:1467] Processing CreateTablet for tablet fe42e6c8c85648a0be51a6aa6211d3d6 (DEFAULT_TABLE table=default.Table_0c12355d49954405ae0e2b245ccdbc6a [id=5fc2baf87f464b2aa81c0239900326af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.785460 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe42e6c8c85648a0be51a6aa6211d3d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.785868 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 5f083ade55de4b6487a663f8daf526e8 (DEFAULT_TABLE table=default.Table_cc09d7ab929c4f4ba017a922d1b1cad9 [id=30b20b2d45764c21baa726be8a5c0107]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.786283 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f083ade55de4b6487a663f8daf526e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.787525 25158 raft_consensus.cc:504] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.787840 24022 tablet_service.cc:1467] Processing CreateTablet for tablet 33928d65045c42f2b8c12a1f52dcf206 (DEFAULT_TABLE table=default.Table_08cf295230eb40d7a7807bbd049f127b [id=da35e70749ea4ee3b0cdee1a0a2bf77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.788170 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33928d65045c42f2b8c12a1f52dcf206. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.788518 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ddfc7b74644a0bbf8222d5f90e6658" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:54.788679 23914 raft_consensus.cc:3037] T 78ddfc7b74644a0bbf8222d5f90e6658 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.789597 25158 leader_election.cc:288] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:54.789778 23596 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet c45aa0dd26524e0fba27e4eaf17d704e (table default.Table_cbe61670c678416995bf27144ab702d9 [id=589deb067c2b4cd086d0277ecfed5126], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet c45aa0dd26524e0fba27e4eaf17d704e consensus state has no leader
I20240502 08:16:54.789964 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870edf582ab14ad1b57c4c4469b19954. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.794871 23914 raft_consensus.cc:2454] T 78ddfc7b74644a0bbf8222d5f90e6658 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:54.795423 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ddfc7b74644a0bbf8222d5f90e6658" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:54.795624 24053 raft_consensus.cc:3037] T 78ddfc7b74644a0bbf8222d5f90e6658 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.795594 23723 leader_election.cc:302] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:54.795661 24175 tablet_bootstrap.cc:492] T 751808dfc1964b539f6f836d786156aa P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:54.795796 24175 ts_tablet_manager.cc:1367] T 751808dfc1964b539f6f836d786156aa P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20240502 08:16:54.796030 25158 raft_consensus.cc:2781] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:54.796155 25158 raft_consensus.cc:686] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:54.796401 25158 consensus_queue.cc:237] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.796614 24175 raft_consensus.cc:348] T 751808dfc1964b539f6f836d786156aa P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.796784 24175 raft_consensus.cc:374] T 751808dfc1964b539f6f836d786156aa P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.796838 24175 raft_consensus.cc:729] T 751808dfc1964b539f6f836d786156aa P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.797035 24175 consensus_queue.cc:260] T 751808dfc1964b539f6f836d786156aa P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.797611 24175 ts_tablet_manager.cc:1398] T 751808dfc1964b539f6f836d786156aa P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:54.797717 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ddfc7b74644a0bbf8222d5f90e6658" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:54.805023 24180 ts_tablet_manager.cc:1888] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:54.807034 24173 raft_consensus.cc:348] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.807273 24173 raft_consensus.cc:374] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.807340 24173 raft_consensus.cc:729] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.807747 24180 log.cc:1196] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/cb7b37f776df47da9aa6e7fba82a8999
I20240502 08:16:54.807842 24053 raft_consensus.cc:2454] T 78ddfc7b74644a0bbf8222d5f90e6658 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:54.808223 24180 ts_tablet_manager.cc:1909] T cb7b37f776df47da9aa6e7fba82a8999 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:54.813511 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 933) took 2648 ms. Trace:
I20240502 08:16:54.813705 25012 ts_tablet_manager.cc:1909] T aacdf506221044ffada0fec0c7d42d0b P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:54.813647 24180 rpcz_store.cc:276] 0502 08:16:52.164647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.164905 (+ 258us) service_pool.cc:225] Handling call
0502 08:16:54.745507 (+2580602us) ts_tablet_manager.cc:1120] Deleting tablet cb7b37f776df47da9aa6e7fba82a8999
0502 08:16:54.745517 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.752021 (+ 6504us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.804911 (+ 52890us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.813477 (+ 8566us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2577431}
I20240502 08:16:54.814270 24175 tablet_bootstrap.cc:492] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:54.816474 24173 consensus_queue.cc:260] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.816780 25317 raft_consensus.cc:482] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:54.816895 25317 raft_consensus.cc:504] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.817467 25317 leader_election.cc:288] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:54.817960 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 1325102bddf347e6a694cacd984804bf (DEFAULT_TABLE table=default.Table_fffdb860d1084781ac9c8ab11a5194b3 [id=e486116396f243178bdf4de6972ef057]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.818199 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d04312bd81f46ed8c49c42b1af1aec9" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:54.818331 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1325102bddf347e6a694cacd984804bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.818390 23788 raft_consensus.cc:2454] T 5d04312bd81f46ed8c49c42b1af1aec9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:54.818783 23993 leader_election.cc:302] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:54.818955 24180 tablet_replica.cc:330] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:54.819017 25317 raft_consensus.cc:2781] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:54.819072 25317 raft_consensus.cc:482] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:54.819123 25317 raft_consensus.cc:3037] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.819125 24180 raft_consensus.cc:2229] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:54.819417 24180 raft_consensus.cc:2258] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.819934 24180 ts_tablet_manager.cc:1875] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.821406 25317 raft_consensus.cc:504] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.822719 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d04312bd81f46ed8c49c42b1af1aec9" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:54.822922 23788 raft_consensus.cc:3037] T 5d04312bd81f46ed8c49c42b1af1aec9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.823181 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d04312bd81f46ed8c49c42b1af1aec9" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:54.823773 23996 leader_election.cc:341] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:54.823915 25317 leader_election.cc:288] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:54.824544 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 1325102bddf347e6a694cacd984804bf (DEFAULT_TABLE table=default.Table_fffdb860d1084781ac9c8ab11a5194b3 [id=e486116396f243178bdf4de6972ef057]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.824839 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet cb7b37f776df47da9aa6e7fba82a8999 (table default.Table_3def35abf8ad4ff49c5ae77acb410e01 [id=5550f8374136404e84d93cc14b9cf834]) successfully deleted
I20240502 08:16:54.824901 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1325102bddf347e6a694cacd984804bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.825100 23788 raft_consensus.cc:2454] T 5d04312bd81f46ed8c49c42b1af1aec9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:54.825490 23993 leader_election.cc:302] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:54.826284 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7b37f776df47da9aa6e7fba82a8999 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:54.826720 23905 tablet_service.cc:1467] Processing CreateTablet for tablet e8bb9a499b254ae69ba752b949f409c8 (DEFAULT_TABLE table=default.Table_d66d58dae2d54c2c8befedd495a51606 [id=a589c3abe0c74ce8aa0d44c42dbcb53a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.818718 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d04312bd81f46ed8c49c42b1af1aec9" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:54.832098 23996 leader_election.cc:341] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:54.836520 25317 raft_consensus.cc:2781] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:54.836742 25317 raft_consensus.cc:686] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:54.837005 25317 consensus_queue.cc:237] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.838085 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_825b08eccbb0430395675758faf7b2f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:54.838675 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_825b08eccbb0430395675758faf7b2f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.839388 24180 ts_tablet_manager.cc:1888] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:54.839509 24180 log.cc:1196] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/f5ccd43e78574f3288b199fa14bcc0e8
I20240502 08:16:54.839893 24180 ts_tablet_manager.cc:1909] T f5ccd43e78574f3288b199fa14bcc0e8 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:54.840219 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 936) took 2674 ms. Trace:
I20240502 08:16:54.843652 24028 tablet_service.cc:1467] Processing CreateTablet for tablet d5941c2ed5014934b4639dc277eacc6f (DEFAULT_TABLE table=default.Table_96756fabe5bb4bb0b8f9b6b105bf9501 [id=a50dbaddb1d6499b9eb6a7beaf019239]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:54.846755 23993 consensus_peers.cc:485] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5c377e037894728ac084b6500ad20de. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:54.840305 24180 rpcz_store.cc:276] 0502 08:16:52.165336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.165389 (+ 53us) service_pool.cc:225] Handling call
0502 08:16:54.818907 (+2653518us) ts_tablet_manager.cc:1120] Deleting tablet f5ccd43e78574f3288b199fa14bcc0e8
0502 08:16:54.818917 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.830845 (+ 11928us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.839276 (+ 8431us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.840201 (+ 925us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2649854}
I20240502 08:16:54.848009 24180 tablet_replica.cc:330] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:54.848203 24180 raft_consensus.cc:2229] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:54.848348 24180 raft_consensus.cc:2258] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.847831 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5941c2ed5014934b4639dc277eacc6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.848809 24180 ts_tablet_manager.cc:1875] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:54.851424 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1218fd56b8a4e88827f275f952ad782 consensus state has no leader
I20240502 08:16:54.851753 23602 master_service.cc:946] ReplaceTablet: received request to replace tablet a4d151770ee44130adfd822518b9b017 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:54.852397 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_18f36b7d14194553a25aad1cc668845a" } new_table_name: "default.Table_c52f7bb042fb42dab6ef2adf2125a8a9" modify_external_catalogs: true
W20240502 08:16:54.853335 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90d44be09f284f3d9042038c92f3b28d consensus state has no leader
W20240502 08:16:54.853869 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f4fb53d5cc9465888dce5ebb7a79379 consensus state has no leader
I20240502 08:16:54.854271 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1b391fb1270447ed9134fcc3d288a9e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:54.854856 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b391fb1270447ed9134fcc3d288a9e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.857614 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_47fe484d653742f582516ba50ff30131" } new_table_name: "default.Table_55a49eab5d8049af849691e864150bd7" modify_external_catalogs: true
I20240502 08:16:54.861083 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5f11f4a4e19542c3bb6267e4e858d5e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:54.861552 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f11f4a4e19542c3bb6267e4e858d5e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:54.864169 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet f5ccd43e78574f3288b199fa14bcc0e8 (table default.Table_95df2e76edc746a08164b4ff25809909 [id=d5a04f8005714d15a3710236c85b1317]) successfully deleted
I20240502 08:16:54.866209 24174 tablet_bootstrap.cc:492] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:54.866349 24174 ts_tablet_manager.cc:1367] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.185s user 0.006s sys 0.002s
I20240502 08:16:54.867024 24180 ts_tablet_manager.cc:1888] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:54.867132 24180 log.cc:1196] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/128c88af54d24ffd895e9de19cdb4c8a
I20240502 08:16:54.867451 24180 ts_tablet_manager.cc:1909] T 128c88af54d24ffd895e9de19cdb4c8a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:54.867565 24173 ts_tablet_manager.cc:1398] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.104s user 0.002s sys 0.000s
I20240502 08:16:54.867735 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 932) took 2703 ms. Trace:
I20240502 08:16:54.867892 24173 tablet_bootstrap.cc:492] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:54.867796 24180 rpcz_store.cc:276] 0502 08:16:52.164558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.164949 (+ 391us) service_pool.cc:225] Handling call
0502 08:16:54.847963 (+2683014us) ts_tablet_manager.cc:1120] Deleting tablet 128c88af54d24ffd895e9de19cdb4c8a
0502 08:16:54.847971 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.851524 (+ 3553us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.866933 (+ 15409us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.867719 (+ 786us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2679620}
I20240502 08:16:54.868080 24180 tablet_replica.cc:330] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:54.868240 24180 raft_consensus.cc:2229] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:54.868357 24180 raft_consensus.cc:2258] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.868826 24180 ts_tablet_manager.cc:1875] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.869751 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 128c88af54d24ffd895e9de19cdb4c8a (table default.Table_d67be3894f494223a45d8c5a2c3e6113 [id=f7982b1d892942648cf3fca657c2668c]) successfully deleted
I20240502 08:16:54.871074 23920 raft_consensus.cc:3037] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.871071 23893 tablet_service.cc:1467] Processing CreateTablet for tablet d5941c2ed5014934b4639dc277eacc6f (DEFAULT_TABLE table=default.Table_96756fabe5bb4bb0b8f9b6b105bf9501 [id=a50dbaddb1d6499b9eb6a7beaf019239]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.871480 25357 raft_consensus.cc:482] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:54.871805 23917 raft_consensus.cc:3037] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.872313 25423 consensus_queue.cc:1035] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:54.874418 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet b59c2f93d6514c9cbf358ab96f1a06b5 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f7ea2d3bc8a494ebdefa6de8ca389e0) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:54.871583 25357 raft_consensus.cc:504] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.875159 23603 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 7ed919909289410e867881baab841f53 (table default.Table_2edad641920b44ce9a4d5b7503f0279a [id=ee3912407a934444ad2c62f7b83d93ac]). Expected version 2 got 0
I20240502 08:16:54.875195 25357 leader_election.cc:288] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:54.875613 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 1325102bddf347e6a694cacd984804bf (DEFAULT_TABLE table=default.Table_fffdb860d1084781ac9c8ab11a5194b3 [id=e486116396f243178bdf4de6972ef057]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.867148 24174 raft_consensus.cc:348] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.877526 24174 raft_consensus.cc:374] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.877622 24174 raft_consensus.cc:729] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.877828 24174 consensus_queue.cc:260] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.880093 24175 tablet_bootstrap.cc:654] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.880172 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed919909289410e867881baab841f53" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:54.880218 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed919909289410e867881baab841f53" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:54.880344 23788 raft_consensus.cc:2454] T 7ed919909289410e867881baab841f53 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:54.880460 24053 raft_consensus.cc:2454] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:54.880849 23861 leader_election.cc:302] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:54.881284 25357 raft_consensus.cc:2781] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:54.881346 25357 raft_consensus.cc:482] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:54.881407 25357 raft_consensus.cc:3037] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.881865 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8bb9a499b254ae69ba752b949f409c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.881939 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 740) took 6598 ms. Trace:
I20240502 08:16:54.881995 25012 rpcz_store.cc:276] 0502 08:16:48.283198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.283254 (+ 56us) service_pool.cc:225] Handling call
0502 08:16:54.731877 (+6448623us) ts_tablet_manager.cc:1120] Deleting tablet aacdf506221044ffada0fec0c7d42d0b
0502 08:16:54.731890 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.762427 (+ 30537us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.765222 (+ 2795us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.881924 (+116702us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6448467}
I20240502 08:16:54.882272 25012 tablet_replica.cc:330] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:54.882457 25012 raft_consensus.cc:2229] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [term 2 LEADER]: Raft consensus shutting down.
I20240502 08:16:54.884069 25012 raft_consensus.cc:2258] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.884892 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5941c2ed5014934b4639dc277eacc6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.886507 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1325102bddf347e6a694cacd984804bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.886534 25012 ts_tablet_manager.cc:1875] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.888077 24173 tablet_bootstrap.cc:654] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.889551 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet aacdf506221044ffada0fec0c7d42d0b (table default.Table_b2e1c08bfc404c7ebb2720631ad10495 [id=9480d59a76fc401e959a1e41caafc845]) successfully deleted
I20240502 08:16:54.889642 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 751808dfc1964b539f6f836d786156aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:54.889822 24022 tablet_service.cc:1514] Processing DeleteTablet for tablet 955e7bd2216646c6aef5acce8d5b0d77 with delete_type TABLET_DATA_DELETED (replaced by tablet 4645143a00d440a7b6d56c5e30d249f1) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:54.890122 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 78ddfc7b74644a0bbf8222d5f90e6658 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:54.890600 24036 tablet_service.cc:1467] Processing CreateTablet for tablet e8bb9a499b254ae69ba752b949f409c8 (DEFAULT_TABLE table=default.Table_d66d58dae2d54c2c8befedd495a51606 [id=a589c3abe0c74ce8aa0d44c42dbcb53a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.890784 24174 ts_tablet_manager.cc:1398] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.024s user 0.003s sys 0.000s
I20240502 08:16:54.891330 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8bb9a499b254ae69ba752b949f409c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.893791 24174 tablet_bootstrap.cc:492] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:54.895018 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet e47459b5f5bb4ecb96f76240fefa1f5d with delete_type TABLET_DATA_DELETED (replaced by tablet 4917c1f0decf43c2aca74c67d8b45341) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:54.895766 23751 tablet_service.cc:1467] Processing CreateTablet for tablet d5941c2ed5014934b4639dc277eacc6f (DEFAULT_TABLE table=default.Table_96756fabe5bb4bb0b8f9b6b105bf9501 [id=a50dbaddb1d6499b9eb6a7beaf019239]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.896200 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5941c2ed5014934b4639dc277eacc6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.899395 25357 raft_consensus.cc:504] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.899542 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet bb9b75d6be924d0291f94fc4a439dea6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:54.899884 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c28c47b4f814573971227b2208d1f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:54.900019 25357 leader_election.cc:288] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:54.902833 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed919909289410e867881baab841f53" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:54.903079 24053 raft_consensus.cc:3037] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.903120 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed919909289410e867881baab841f53" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:54.903337 23788 raft_consensus.cc:3037] T 7ed919909289410e867881baab841f53 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.905937 24053 raft_consensus.cc:2454] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:54.906577 23788 raft_consensus.cc:2454] T 7ed919909289410e867881baab841f53 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:54.906888 23861 leader_election.cc:302] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:54.907459 25357 raft_consensus.cc:2781] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:54.907621 25357 raft_consensus.cc:686] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:16:54.910532 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet e9941681b26d4c97b06a91155043b040 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:54.911738 24173 tablet_bootstrap.cc:492] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:54.911890 24173 ts_tablet_manager.cc:1367] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.001s
I20240502 08:16:54.913868 25357 consensus_queue.cc:237] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.917009 24173 raft_consensus.cc:348] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.917893 24174 tablet_bootstrap.cc:654] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.921293 24173 raft_consensus.cc:374] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.921442 24173 raft_consensus.cc:729] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.921499 25012 ts_tablet_manager.cc:1888] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20240502 08:16:54.921880 24173 consensus_queue.cc:260] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.926509 25012 log.cc:1196] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/5bcc7169796e4ee0910f5e7084849012
I20240502 08:16:54.927357 25012 ts_tablet_manager.cc:1909] T 5bcc7169796e4ee0910f5e7084849012 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:54.930699 24174 tablet_bootstrap.cc:492] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:54.930850 24174 ts_tablet_manager.cc:1367] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.001s
I20240502 08:16:54.927583 24181 ts_tablet_manager.cc:1888] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:54.931865 24181 log.cc:1196] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/7a97427e819b4826b1ed89e6d021c8d4
I20240502 08:16:54.931988 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 744) took 6355 ms. Trace:
I20240502 08:16:54.932091 24173 ts_tablet_manager.cc:1398] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 08:16:54.932595 24173 tablet_bootstrap.cc:492] T 6e7432f1df94431a83cf68356e3e78a3 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:54.933393 25012 rpcz_store.cc:276] 0502 08:16:48.576568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.576608 (+ 40us) service_pool.cc:225] Handling call
0502 08:16:54.882238 (+6305630us) ts_tablet_manager.cc:1120] Deleting tablet 5bcc7169796e4ee0910f5e7084849012
0502 08:16:54.882244 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.908378 (+ 26134us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.921359 (+ 12981us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.931965 (+ 10606us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6305460}
I20240502 08:16:54.934433 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 5bcc7169796e4ee0910f5e7084849012 (table default.Table_bc9b016da1ce42b6a0509fbb664c8548 [id=8c0e32da69b34dd0b2a3c68e4ba7a166]) successfully deleted
I20240502 08:16:54.934660 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 745) took 6357 ms. Trace:
I20240502 08:16:54.935786 25012 rpcz_store.cc:276] 0502 08:16:48.576948 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.576985 (+ 37us) service_pool.cc:225] Handling call
0502 08:16:54.934529 (+6357544us) ts_tablet_manager.cc:1120] Deleting tablet 8e2c611d7af6428db289a7e2fbd586d4
0502 08:16:54.934538 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.934645 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6357333}
I20240502 08:16:54.936826 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 746) took 6359 ms. Trace:
W20240502 08:16:54.937834 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 8e2c611d7af6428db289a7e2fbd586d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e2c611d7af6428db289a7e2fbd586d4 already in progress: creating tablet
I20240502 08:16:54.937880 25012 rpcz_store.cc:276] 0502 08:16:48.577349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.577395 (+ 46us) service_pool.cc:225] Handling call
0502 08:16:54.936717 (+6359322us) ts_tablet_manager.cc:1120] Deleting tablet e608dd4f99004bb795de5e05ea700e08
0502 08:16:54.936724 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.936810 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6359142}
W20240502 08:16:54.938789 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e608dd4f99004bb795de5e05ea700e08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e608dd4f99004bb795de5e05ea700e08 already in progress: creating tablet
I20240502 08:16:54.938920 25012 tablet_replica.cc:330] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:54.940691 25012 raft_consensus.cc:2229] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:54.941679 25012 raft_consensus.cc:2258] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.944249 24174 raft_consensus.cc:348] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:54.944460 24174 raft_consensus.cc:374] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.944523 24174 raft_consensus.cc:729] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.945044 25012 ts_tablet_manager.cc:1875] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.941035 24175 tablet_bootstrap.cc:492] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:54.946004 24175 ts_tablet_manager.cc:1367] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.132s user 0.012s sys 0.000s
I20240502 08:16:54.946213 25227 consensus_queue.cc:1035] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:54.948935 24174 consensus_queue.cc:260] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:54.951323 24173 tablet_bootstrap.cc:654] T 6e7432f1df94431a83cf68356e3e78a3 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.952512 24174 ts_tablet_manager.cc:1398] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20240502 08:16:54.952808 24174 tablet_bootstrap.cc:492] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:54.953109 24175 raft_consensus.cc:348] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.953846 24175 raft_consensus.cc:374] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.953963 24175 raft_consensus.cc:729] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.954332 24175 consensus_queue.cc:260] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:54.954937 24175 ts_tablet_manager.cc:1398] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 08:16:54.955400 24175 tablet_bootstrap.cc:492] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:54.957430 24175 tablet_bootstrap.cc:654] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.962829 24174 tablet_bootstrap.cc:654] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.963806 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_417bf79bff5e4e01ab05528227bf9081" } modify_external_catalogs: true
I20240502 08:16:54.964094 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_417bf79bff5e4e01ab05528227bf9081" } modify_external_catalogs: true
I20240502 08:16:54.965859 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7952191ae36044e9aef62593bbe0bb27" } modify_external_catalogs: true
I20240502 08:16:54.966104 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7952191ae36044e9aef62593bbe0bb27" } modify_external_catalogs: true
I20240502 08:16:54.968652 23604 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet a2da8b9d3f374d8a812336057c0ce7d2 (table default.Table_3edd27da6927478281e76361718425a3 [id=76b58ce209a24b9a95b405c1c50de3ca]). Expected version 3 got 0
I20240502 08:16:54.979566 24174 tablet_bootstrap.cc:492] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:54.979722 24174 ts_tablet_manager.cc:1367] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20240502 08:16:54.980432 24174 raft_consensus.cc:348] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:54.980618 24174 raft_consensus.cc:374] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.980664 24174 raft_consensus.cc:729] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.981510 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fc9f3ca03af3472d9628fdfe64a89450" } modify_external_catalogs: true
I20240502 08:16:54.981842 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fc9f3ca03af3472d9628fdfe64a89450" } modify_external_catalogs: true
I20240502 08:16:54.986531 24180 ts_tablet_manager.cc:1888] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:54.986656 24180 log.cc:1196] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/1da01e2c2f054ccfb53383e70e16da84
I20240502 08:16:54.986935 24180 ts_tablet_manager.cc:1909] T 1da01e2c2f054ccfb53383e70e16da84 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:54.987282 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 931) took 2823 ms. Trace:
I20240502 08:16:54.987367 24180 rpcz_store.cc:276] 0502 08:16:52.163891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.165550 (+ 1659us) service_pool.cc:225] Handling call
0502 08:16:54.868041 (+2702491us) ts_tablet_manager.cc:1120] Deleting tablet 1da01e2c2f054ccfb53383e70e16da84
0502 08:16:54.868049 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.972839 (+104790us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.986420 (+ 13581us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.987258 (+ 838us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2699118}
I20240502 08:16:54.987697 24180 tablet_replica.cc:330] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:54.987874 24180 raft_consensus.cc:2229] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:54.988013 24180 raft_consensus.cc:2258] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:54.988152 24174 consensus_queue.cc:260] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:54.988548 24180 ts_tablet_manager.cc:1875] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:54.988590 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 1da01e2c2f054ccfb53383e70e16da84 (table default.Table_3ef466d7a87644e99070934d65e5c851 [id=f596bcb8d6f04ac680be38e763a33750]) successfully deleted
I20240502 08:16:54.989734 24174 ts_tablet_manager.cc:1398] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 08:16:54.989995 24174 tablet_bootstrap.cc:492] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:54.991322 24174 tablet_bootstrap.cc:654] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.999395 24175 tablet_bootstrap.cc:492] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:54.999620 24175 ts_tablet_manager.cc:1367] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.044s user 0.012s sys 0.006s
I20240502 08:16:55.000386 24175 raft_consensus.cc:348] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.000628 24175 raft_consensus.cc:374] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.000713 24175 raft_consensus.cc:729] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.004084 25012 ts_tablet_manager.cc:1888] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:55.005226 24174 tablet_bootstrap.cc:492] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:55.005369 24174 ts_tablet_manager.cc:1367] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240502 08:16:55.005542 25012 log.cc:1196] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/df9402cf8abc4b4d893d42f9a3a99016
I20240502 08:16:55.006134 25012 ts_tablet_manager.cc:1909] T df9402cf8abc4b4d893d42f9a3a99016 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:55.006160 24174 raft_consensus.cc:348] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.006366 24174 raft_consensus.cc:374] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.006440 24174 raft_consensus.cc:729] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.007099 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 751) took 6396 ms. Trace:
I20240502 08:16:55.009120 25012 rpcz_store.cc:276] 0502 08:16:48.611014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.611075 (+ 61us) service_pool.cc:225] Handling call
0502 08:16:54.938870 (+6327795us) ts_tablet_manager.cc:1120] Deleting tablet df9402cf8abc4b4d893d42f9a3a99016
0502 08:16:54.938880 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.989785 (+ 50905us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.993513 (+ 3728us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.007076 (+ 13563us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6238404}
W20240502 08:16:55.009276 23993 consensus_peers.cc:485] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:55.010830 24174 consensus_queue.cc:260] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.012334 24174 ts_tablet_manager.cc:1398] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:16:55.012622 24174 tablet_bootstrap.cc:492] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:55.000960 24175 consensus_queue.cc:260] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.013952 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet df9402cf8abc4b4d893d42f9a3a99016 (table default.Table_4b967ef2b2fc4e2caa4219afcae841a6 [id=c32fc96889554c32bdf1d4550633e91f]) successfully deleted
I20240502 08:16:55.014117 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 752) took 6402 ms. Trace:
I20240502 08:16:55.014187 25012 rpcz_store.cc:276] 0502 08:16:48.611371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.611432 (+ 61us) service_pool.cc:225] Handling call
0502 08:16:55.013997 (+6402565us) ts_tablet_manager.cc:1120] Deleting tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58
0502 08:16:55.014007 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.014099 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6313417}
W20240502 08:16:55.015040 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58 already in progress: creating tablet
I20240502 08:16:55.019218 24175 ts_tablet_manager.cc:1398] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20240502 08:16:55.019732 24174 tablet_bootstrap.cc:654] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.020577 25012 tablet_replica.cc:330] T 7a97427e819b4826b1ed89e6d021c8d4 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:55.020741 25012 raft_consensus.cc:2229] T 7a97427e819b4826b1ed89e6d021c8d4 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:55.020895 25012 raft_consensus.cc:2258] T 7a97427e819b4826b1ed89e6d021c8d4 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.023607 24175 tablet_bootstrap.cc:492] T 001e3f4f72f64dc69dbd521724912f07 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:55.023860 25012 ts_tablet_manager.cc:1875] T 7a97427e819b4826b1ed89e6d021c8d4 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.026983 24175 tablet_bootstrap.cc:654] T 001e3f4f72f64dc69dbd521724912f07 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.028010 25357 consensus_queue.cc:1035] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:55.037600 24174 tablet_bootstrap.cc:492] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:55.037746 24174 ts_tablet_manager.cc:1367] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20240502 08:16:55.040266 25296 raft_consensus.cc:482] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.040433 25296 raft_consensus.cc:504] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.041083 25296 leader_election.cc:288] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.041745 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07344b78894c42809391d0fe2a9cd8f7" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:55.041774 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07344b78894c42809391d0fe2a9cd8f7" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:55.042294 23723 leader_election.cc:341] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:55.042704 23723 leader_election.cc:341] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.042788 23723 leader_election.cc:302] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:55.042990 25296 raft_consensus.cc:2726] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:55.043526 24174 raft_consensus.cc:348] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.043690 24174 raft_consensus.cc:374] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.043743 24174 raft_consensus.cc:729] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.044956 24174 consensus_queue.cc:260] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.046883 24174 ts_tablet_manager.cc:1398] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 08:16:55.047268 24174 tablet_bootstrap.cc:492] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:55.048687 24181 ts_tablet_manager.cc:1909] T 7a97427e819b4826b1ed89e6d021c8d4 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:55.049165 24174 tablet_bootstrap.cc:654] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.049768 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 751) took 6913 ms. Trace:
I20240502 08:16:55.058593 25408 consensus_queue.cc:1035] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:55.059089 25317 raft_consensus.cc:482] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.059230 25317 raft_consensus.cc:504] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.059890 25317 leader_election.cc:288] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.060662 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2da8b9d3f374d8a812336057c0ce7d2" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:55.061250 23996 leader_election.cc:341] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.065845 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2da8b9d3f374d8a812336057c0ce7d2" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:55.066167 23788 raft_consensus.cc:2454] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:55.066594 23993 leader_election.cc:302] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:55.067265 25317 raft_consensus.cc:2781] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:55.067363 25317 raft_consensus.cc:482] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:55.067427 25317 raft_consensus.cc:3037] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.071254 24174 tablet_bootstrap.cc:492] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:55.071410 24174 ts_tablet_manager.cc:1367] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240502 08:16:55.074385 24180 ts_tablet_manager.cc:1888] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:55.074535 24180 log.cc:1196] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/654ce33d1e3d472eb6ae6a726eb803bb
I20240502 08:16:55.074981 24180 ts_tablet_manager.cc:1909] T 654ce33d1e3d472eb6ae6a726eb803bb P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:55.076776 24175 tablet_bootstrap.cc:492] T 001e3f4f72f64dc69dbd521724912f07 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:55.076978 24175 ts_tablet_manager.cc:1367] T 001e3f4f72f64dc69dbd521724912f07 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.008s
I20240502 08:16:55.077793 24175 raft_consensus.cc:348] T 001e3f4f72f64dc69dbd521724912f07 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.078774 24175 raft_consensus.cc:374] T 001e3f4f72f64dc69dbd521724912f07 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.078899 24175 raft_consensus.cc:729] T 001e3f4f72f64dc69dbd521724912f07 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.050024 24181 rpcz_store.cc:276] 0502 08:16:48.136433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.137247 (+ 814us) service_pool.cc:225] Handling call
0502 08:16:54.500832 (+6363585us) ts_tablet_manager.cc:1120] Deleting tablet 7a97427e819b4826b1ed89e6d021c8d4
0502 08:16:54.500843 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:54.623386 (+122543us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:54.925126 (+301740us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.049710 (+124584us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6363436}
I20240502 08:16:55.079154 24175 consensus_queue.cc:260] T 001e3f4f72f64dc69dbd521724912f07 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.082825 24181 tablet_replica.cc:330] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:55.083014 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 7a97427e819b4826b1ed89e6d021c8d4 (table default.Table_e66f5d1a72b347f7a7b39bddbf74b263 [id=9b0006a9ac9b437fa608c587d73b09f5]) successfully deleted
I20240502 08:16:55.083307 24175 ts_tablet_manager.cc:1398] T 001e3f4f72f64dc69dbd521724912f07 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:16:55.083263 24181 raft_consensus.cc:2229] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:55.083626 24181 raft_consensus.cc:2258] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.086541 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 934) took 2921 ms. Trace:
I20240502 08:16:55.086688 24180 rpcz_store.cc:276] 0502 08:16:52.164745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.164819 (+ 74us) service_pool.cc:225] Handling call
0502 08:16:54.987654 (+2822835us) ts_tablet_manager.cc:1120] Deleting tablet 654ce33d1e3d472eb6ae6a726eb803bb
0502 08:16:54.987663 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.012034 (+ 24371us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.074285 (+ 62251us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.086485 (+ 12200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":768,"tablet-delete.queue_time_us":2818560}
I20240502 08:16:55.087314 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 654ce33d1e3d472eb6ae6a726eb803bb (table default.Table_6f4d25ef5b90455ca233af7b70b9073a [id=2fe4c270673b4204aaccc289d3b9d929]) successfully deleted
I20240502 08:16:55.089850 24180 tablet_replica.cc:330] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
W20240502 08:16:55.089893 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a059351d148e4376adfbe7128c88c351 consensus state has no leader
I20240502 08:16:55.093827 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b1d809964c99404891db2d83cb16887f" } modify_external_catalogs: true
I20240502 08:16:55.094036 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b1d809964c99404891db2d83cb16887f" } modify_external_catalogs: true
I20240502 08:16:55.094934 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_10dce9ce73a64cc9a0bf79972f69aed9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:55.095459 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10dce9ce73a64cc9a0bf79972f69aed9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:55.096758 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e9a35b9486da4aac8f3d3febc93b7816 consensus state has no leader
I20240502 08:16:55.097054 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f8db538a85994d0dbee1b1cbbabf5e16" } modify_external_catalogs: true
I20240502 08:16:55.097222 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f8db538a85994d0dbee1b1cbbabf5e16" } modify_external_catalogs: true
I20240502 08:16:55.097631 24180 raft_consensus.cc:2229] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:55.097985 24180 raft_consensus.cc:2258] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.098454 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_cc0830cedb8b4372bd149e16c0548861"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:55.098835 25012 ts_tablet_manager.cc:1888] T 7a97427e819b4826b1ed89e6d021c8d4 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:55.098954 25012 log.cc:1196] T 7a97427e819b4826b1ed89e6d021c8d4 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/7a97427e819b4826b1ed89e6d021c8d4
W20240502 08:16:55.099017 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc0830cedb8b4372bd149e16c0548861 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.099293 25012 ts_tablet_manager.cc:1909] T 7a97427e819b4826b1ed89e6d021c8d4 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:55.081679 24174 raft_consensus.cc:348] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.100662 24180 ts_tablet_manager.cc:1875] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.100698 24174 raft_consensus.cc:374] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.100796 24174 raft_consensus.cc:729] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.101704 25296 raft_consensus.cc:482] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.101847 25296 raft_consensus.cc:504] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.102463 25296 leader_election.cc:288] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.103207 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd7244b9f5741a8aa1c6835fe544676" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:55.103250 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd7244b9f5741a8aa1c6835fe544676" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:55.101017 24174 consensus_queue.cc:260] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.103935 25413 raft_consensus.cc:482] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.104084 25413 raft_consensus.cc:504] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.104635 25413 leader_election.cc:288] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.105017 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955e7bd2216646c6aef5acce8d5b0d77" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:55.105243 23788 raft_consensus.cc:2454] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
W20240502 08:16:55.105743 23723 leader_election.cc:341] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.105732 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955e7bd2216646c6aef5acce8d5b0d77" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:55.105899 23920 raft_consensus.cc:2454] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
W20240502 08:16:55.105952 23723 leader_election.cc:341] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.106040 23723 leader_election.cc:302] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:55.106709 23996 leader_election.cc:302] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:55.107053 25413 raft_consensus.cc:2781] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:55.107156 25413 raft_consensus.cc:482] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:55.107220 25413 raft_consensus.cc:3037] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.107529 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e9941681b26d4c97b06a91155043b040
I20240502 08:16:55.108458 23601 catalog_manager.cc:6554] ReplaceTablet: tablet e9941681b26d4c97b06a91155043b040 deleted and replaced by tablet eaac3bd5aa674422b7feb85ed9db1f70
I20240502 08:16:55.108892 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet e9941681b26d4c97b06a91155043b040 with delete_type TABLET_DATA_DELETED (replaced by tablet eaac3bd5aa674422b7feb85ed9db1f70) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:55.109048 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet e9941681b26d4c97b06a91155043b040 with delete_type TABLET_DATA_DELETED (replaced by tablet eaac3bd5aa674422b7feb85ed9db1f70) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:55.109750 25296 raft_consensus.cc:2726] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:55.111852 23601 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet e0b0d4871dd24607854100a4ef762e25 (table default.Table_5301ffccbac743618a78289c2c1415e1 [id=910c7cae36264e39bbfca0be919538e0]). Expected version 1 got 0
I20240502 08:16:55.116129 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f57e3a39d54b7ab372d4b145b7a696 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:55.117249 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet c00f29fb76b44d6ea4f76963d2d0751c with delete_type TABLET_DATA_DELETED (replaced by tablet a17fb4f2f02e43dc9f7cfdb7c73f7774) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:55.118351 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet 18d284f71937459db0c0676ce4568175 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:55.119908 25408 raft_consensus.cc:482] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.120085 25408 raft_consensus.cc:504] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.120724 25408 leader_election.cc:288] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:55.121717 25296 raft_consensus.cc:482] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.121853 25296 raft_consensus.cc:504] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.122056 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f83bcf640f465bb95d87a991cc5ff6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:55.122402 25296 leader_election.cc:288] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:55.122838 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f83bcf640f465bb95d87a991cc5ff6" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:55.122995 23861 leader_election.cc:341] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.123055 23920 raft_consensus.cc:2454] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:55.121466 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f83bcf640f465bb95d87a991cc5ff6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:55.123287 23788 raft_consensus.cc:2454] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:55.123975 23723 leader_election.cc:302] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:55.124260 25296 raft_consensus.cc:2781] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:55.124293 23858 leader_election.cc:302] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:55.124373 25296 raft_consensus.cc:482] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:55.124460 25296 raft_consensus.cc:3037] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.124759 25408 raft_consensus.cc:2781] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:55.124852 25408 raft_consensus.cc:482] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:55.124920 25408 raft_consensus.cc:3037] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.127174 25296 raft_consensus.cc:504] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.128191 25296 leader_election.cc:288] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:55.128291 25408 raft_consensus.cc:504] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.129838 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f83bcf640f465bb95d87a991cc5ff6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:55.130077 23788 raft_consensus.cc:2379] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:16:55.130571 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f83bcf640f465bb95d87a991cc5ff6" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:55.130762 23920 raft_consensus.cc:2379] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a297a6c7c2504ca68e87b24f6691ffb2 in current term 1: Already voted for candidate 5b89a1f0485a44b7ac1710e2369a809b in this term.
I20240502 08:16:55.131320 25408 leader_election.cc:288] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:55.131980 25408 raft_consensus.cc:482] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.132217 25408 raft_consensus.cc:504] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.133275 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1d946bd6de48f1a8d8fe5f179d22b6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:55.133502 23788 raft_consensus.cc:2454] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:55.133968 23858 leader_election.cc:302] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:55.134248 25357 raft_consensus.cc:2781] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:55.134337 25357 raft_consensus.cc:482] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:55.134416 25357 raft_consensus.cc:3037] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.134387 25296 raft_consensus.cc:482] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.134534 25408 leader_election.cc:288] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:55.134521 25296 raft_consensus.cc:504] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.135111 25296 leader_election.cc:288] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.135531 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee022e8e86f8477a8fa1e58d52644b03" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:55.135624 24174 ts_tablet_manager.cc:1398] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.064s user 0.002s sys 0.000s
W20240502 08:16:55.136055 23723 leader_election.cc:341] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.137226 25357 raft_consensus.cc:504] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.138075 24174 tablet_bootstrap.cc:492] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:55.140094 25357 leader_election.cc:288] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:55.140287 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1d946bd6de48f1a8d8fe5f179d22b6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:55.140470 23788 raft_consensus.cc:3037] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.141073 23835 heartbeater.cc:347] Connected to a master server at 127.22.225.125:36189
I20240502 08:16:55.142890 23788 raft_consensus.cc:2454] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:55.143599 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet e9941681b26d4c97b06a91155043b040 with delete_type TABLET_DATA_DELETED (replaced by tablet eaac3bd5aa674422b7feb85ed9db1f70) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:55.143915 24174 tablet_bootstrap.cc:654] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.144006 23858 leader_election.cc:302] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:55.144290 25357 raft_consensus.cc:2781] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:55.144425 25357 raft_consensus.cc:686] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:16:55.144443 25317 raft_consensus.cc:504] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.144662 25357 consensus_queue.cc:237] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.146001 25317 leader_election.cc:288] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.146579 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2da8b9d3f374d8a812336057c0ce7d2" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:55.146986 24181 ts_tablet_manager.cc:1875] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.147032 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1d946bd6de48f1a8d8fe5f179d22b6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:55.147509 23996 leader_election.cc:341] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.147945 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f83bcf640f465bb95d87a991cc5ff6" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:55.148722 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2da8b9d3f374d8a812336057c0ce7d2" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:55.148869 23788 raft_consensus.cc:3037] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:55.150514 23861 leader_election.cc:341] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.150844 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1d946bd6de48f1a8d8fe5f179d22b6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:55.152702 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f83bcf640f465bb95d87a991cc5ff6" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:16:55.153035 23861 leader_election.cc:341] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.157325 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 89fefde55e514e5fabae3f4e21f44ca3
I20240502 08:16:55.157678 23788 raft_consensus.cc:2454] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:55.158156 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2ec1dfc748994bfe8841412d61167608"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:55.158756 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ec1dfc748994bfe8841412d61167608 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.163475 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet 89fefde55e514e5fabae3f4e21f44ca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:55.175526 24174 tablet_bootstrap.cc:492] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:55.175700 24174 ts_tablet_manager.cc:1367] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20240502 08:16:55.180408 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet 89fefde55e514e5fabae3f4e21f44ca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:55.180670 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet 89fefde55e514e5fabae3f4e21f44ca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:55.181022 23723 leader_election.cc:341] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:55.181502 23723 leader_election.cc:341] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.181617 23723 leader_election.cc:302] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:55.182010 25418 raft_consensus.cc:2726] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:16:55.181959 25296 consensus_queue.cc:1035] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:55.183617 24174 raft_consensus.cc:348] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.184005 24174 raft_consensus.cc:374] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.184120 24174 raft_consensus.cc:729] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.184227 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee022e8e86f8477a8fa1e58d52644b03" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:55.184486 24174 consensus_queue.cc:260] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.186000 24174 ts_tablet_manager.cc:1398] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240502 08:16:55.186573 24174 tablet_bootstrap.cc:492] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:55.163368 23993 leader_election.cc:302] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:55.192790 25317 raft_consensus.cc:2781] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:55.192911 25317 raft_consensus.cc:686] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:55.152312 25413 raft_consensus.cc:504] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:55.195058 23723 leader_election.cc:341] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.195158 24175 tablet_bootstrap.cc:492] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:55.195178 23723 leader_election.cc:302] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:55.193127 25317 consensus_queue.cc:237] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.195813 25158 raft_consensus.cc:2726] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:55.196767 25408 raft_consensus.cc:482] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.196928 25408 raft_consensus.cc:504] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.197515 25408 leader_election.cc:288] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:55.199442 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f02efa659e457ab57bc32ed3f66bb9" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:55.199571 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 753) took 6390 ms. Trace:
I20240502 08:16:55.199640 25012 rpcz_store.cc:276] 0502 08:16:48.808981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.809040 (+ 59us) service_pool.cc:225] Handling call
0502 08:16:55.020530 (+6211490us) ts_tablet_manager.cc:1120] Deleting tablet 7a97427e819b4826b1ed89e6d021c8d4
0502 08:16:55.020539 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.063764 (+ 43225us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.098711 (+ 34947us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.199552 (+100841us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6211255}
I20240502 08:16:55.200127 25012 tablet_replica.cc:330] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:55.200091 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 7a97427e819b4826b1ed89e6d021c8d4 (table default.Table_e66f5d1a72b347f7a7b39bddbf74b263 [id=9b0006a9ac9b437fa608c587d73b09f5]) successfully deleted
I20240502 08:16:55.200321 25012 raft_consensus.cc:2229] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:55.200706 25012 raft_consensus.cc:2258] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:55.201987 23858 consensus_peers.cc:485] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1da01e2c2f054ccfb53383e70e16da84. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:55.198673 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f02efa659e457ab57bc32ed3f66bb9" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:55.202840 25413 leader_election.cc:288] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.203485 23861 leader_election.cc:302] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:55.204051 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955e7bd2216646c6aef5acce8d5b0d77" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:55.204206 23920 raft_consensus.cc:3037] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.204857 25012 ts_tablet_manager.cc:1875] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.205978 25408 raft_consensus.cc:2726] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:55.146899 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f83bcf640f465bb95d87a991cc5ff6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:55.207407 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955e7bd2216646c6aef5acce8d5b0d77" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
W20240502 08:16:55.207502 23861 leader_election.cc:341] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.207599 23783 raft_consensus.cc:3037] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.207633 23861 leader_election.cc:302] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:55.208159 25408 raft_consensus.cc:2726] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:16:55.210186 25413 consensus_queue.cc:1035] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:55.210880 23783 raft_consensus.cc:2454] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:55.211192 23993 leader_election.cc:302] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:55.216629 25413 raft_consensus.cc:2781] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:55.216779 25413 raft_consensus.cc:686] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:55.216806 24175 tablet_bootstrap.cc:654] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.217008 25413 consensus_queue.cc:237] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.222518 23920 raft_consensus.cc:2454] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:55.229501 23920 raft_consensus.cc:3037] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.235138 25408 consensus_queue.cc:1035] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:55.235797 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1014ad7e82a245ea801b1866b929b1c5" } new_table_name: "default.Table_aa64c20542734193a82f4e071dcb0000" modify_external_catalogs: true
I20240502 08:16:55.237588 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fc86e7c02217467c9a9e596e55aeee08" } new_table_name: "default.Table_cea911d15c5b442b8c1f412b3a03539a" modify_external_catalogs: true
W20240502 08:16:55.243347 24051 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.018s sys 0.035s
I20240502 08:16:55.243532 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_206aa937a259485a90fa3dd13e7bd838"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:55.243924 25012 ts_tablet_manager.cc:1888] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:55.244027 25012 log.cc:1196] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/bd4a31e5aa834adeb9f59267247c6980
I20240502 08:16:55.244377 25012 ts_tablet_manager.cc:1909] T bd4a31e5aa834adeb9f59267247c6980 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
W20240502 08:16:55.245124 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_206aa937a259485a90fa3dd13e7bd838 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:55.250263 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b1a2bcc6937c4a95b71103dbf8738702 consensus state has no leader
I20240502 08:16:55.250656 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 754) took 6441 ms. Trace:
I20240502 08:16:55.250698 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_33c693cfb89641a2921c7d39872b5bbc" } new_table_name: "default.Table_7a5da2bba6f64c658d98d56c9465e80f" modify_external_catalogs: true
I20240502 08:16:55.250763 25012 rpcz_store.cc:276] 0502 08:16:48.809089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.809337 (+ 248us) service_pool.cc:225] Handling call
0502 08:16:55.200078 (+6390741us) ts_tablet_manager.cc:1120] Deleting tablet bd4a31e5aa834adeb9f59267247c6980
0502 08:16:55.200089 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.220364 (+ 20275us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.243812 (+ 23448us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.250634 (+ 6822us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6390585}
I20240502 08:16:55.251231 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet bd4a31e5aa834adeb9f59267247c6980 (table default.Table_4e40fa1cf0ab4dedb2c6aadb2418901b [id=fd781805eb4c45f09fe6460144472816]) successfully deleted
I20240502 08:16:55.251888 25158 consensus_queue.cc:1035] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:55.252393 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c991463cf8ad4d3b8c0727e45c2317b4 consensus state has no leader
I20240502 08:16:55.252790 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ae2ecda79c0245b59ef76809b5e3cdcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:55.253275 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae2ecda79c0245b59ef76809b5e3cdcc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:55.259373 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f5496fa258de4988b9502136f0baf779 consensus state has no leader
I20240502 08:16:55.259768 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9cb7a43194434b49ad817e7876631133"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:55.260298 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cb7a43194434b49ad817e7876631133 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.257951 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d6b312d9972e4d8fa7dfb6fcad7314b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:55.262555 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet df08f0d3af4c4c7e8abce656714b1b7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:55.264022 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 755) took 6444 ms. Trace:
I20240502 08:16:55.264163 25012 rpcz_store.cc:276] 0502 08:16:48.819348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.819419 (+ 71us) service_pool.cc:225] Handling call
0502 08:16:55.251451 (+6432032us) ts_tablet_manager.cc:1120] Deleting tablet 7d07efb303b64ce19d993c4c76a47dfc
0502 08:16:55.251462 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.264000 (+ 12538us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6431820}
I20240502 08:16:55.264796 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 756) took 6445 ms. Trace:
W20240502 08:16:55.264681 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 7d07efb303b64ce19d993c4c76a47dfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d07efb303b64ce19d993c4c76a47dfc already in progress: creating tablet
I20240502 08:16:55.264971 25012 rpcz_store.cc:276] 0502 08:16:48.819488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.819549 (+ 61us) service_pool.cc:225] Handling call
0502 08:16:55.264667 (+6445118us) ts_tablet_manager.cc:1120] Deleting tablet 18625a1d971c43549b9f80e6046efc9c
0502 08:16:55.264679 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.264778 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6444973}
W20240502 08:16:55.264506 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6b312d9972e4d8fa7dfb6fcad7314b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.265570 25012 tablet_replica.cc:330] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:55.266321 25012 raft_consensus.cc:2229] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:55.244736 25463 tablet.cc:1722] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:16:55.266680 25012 raft_consensus.cc:2258] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.267246 25012 ts_tablet_manager.cc:1875] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:55.266667 24051 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.596s user 0.018s sys 0.035s
I20240502 08:16:55.279414 24175 tablet_bootstrap.cc:492] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:55.279551 24175 ts_tablet_manager.cc:1367] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.084s user 0.009s sys 0.002s
I20240502 08:16:55.280218 24175 raft_consensus.cc:348] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.280444 24175 raft_consensus.cc:374] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.280512 24175 raft_consensus.cc:729] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.280810 24173 tablet_bootstrap.cc:492] T 6e7432f1df94431a83cf68356e3e78a3 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:55.280983 24173 ts_tablet_manager.cc:1367] T 6e7432f1df94431a83cf68356e3e78a3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.348s user 0.017s sys 0.021s
W20240502 08:16:55.281204 23996 consensus_peers.cc:485] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:55.280726 24175 consensus_queue.cc:260] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.281759 24173 raft_consensus.cc:348] T 6e7432f1df94431a83cf68356e3e78a3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:55.265565 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 18625a1d971c43549b9f80e6046efc9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18625a1d971c43549b9f80e6046efc9c already in progress: creating tablet
I20240502 08:16:55.281942 24173 raft_consensus.cc:374] T 6e7432f1df94431a83cf68356e3e78a3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.281994 24173 raft_consensus.cc:729] T 6e7432f1df94431a83cf68356e3e78a3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.281914 24175 ts_tablet_manager.cc:1398] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:55.282418 24175 tablet_bootstrap.cc:492] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:55.284233 24175 tablet_bootstrap.cc:654] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.287607 24181 ts_tablet_manager.cc:1888] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:55.287710 24181 log.cc:1196] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/bd4a31e5aa834adeb9f59267247c6980
I20240502 08:16:55.288094 24181 ts_tablet_manager.cc:1909] T bd4a31e5aa834adeb9f59267247c6980 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:55.288448 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 753) took 7151 ms. Trace:
I20240502 08:16:55.288533 24181 rpcz_store.cc:276] 0502 08:16:48.136672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.137420 (+ 748us) service_pool.cc:225] Handling call
0502 08:16:55.082772 (+6945352us) ts_tablet_manager.cc:1120] Deleting tablet bd4a31e5aa834adeb9f59267247c6980
0502 08:16:55.082783 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.224394 (+141611us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.287503 (+ 63109us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.288425 (+ 922us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6945201}
I20240502 08:16:55.288872 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 754) took 7142 ms. Trace:
I20240502 08:16:55.288938 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet bd4a31e5aa834adeb9f59267247c6980 (table default.Table_4e40fa1cf0ab4dedb2c6aadb2418901b [id=fd781805eb4c45f09fe6460144472816]) successfully deleted
I20240502 08:16:55.288944 24181 rpcz_store.cc:276] 0502 08:16:48.146366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.146553 (+ 187us) service_pool.cc:225] Handling call
0502 08:16:55.288775 (+7142222us) ts_tablet_manager.cc:1120] Deleting tablet 8e2c611d7af6428db289a7e2fbd586d4
0502 08:16:55.288782 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.288858 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7142061}
I20240502 08:16:55.289228 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 755) took 7108 ms. Trace:
W20240502 08:16:55.289310 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 8e2c611d7af6428db289a7e2fbd586d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e2c611d7af6428db289a7e2fbd586d4 already in progress: creating tablet
I20240502 08:16:55.289347 24181 rpcz_store.cc:276] 0502 08:16:48.180868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.181341 (+ 473us) service_pool.cc:225] Handling call
0502 08:16:55.289130 (+7107789us) ts_tablet_manager.cc:1120] Deleting tablet e608dd4f99004bb795de5e05ea700e08
0502 08:16:55.289137 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.289212 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7107590}
I20240502 08:16:55.289666 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 760) took 7020 ms. Trace:
W20240502 08:16:55.289764 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e608dd4f99004bb795de5e05ea700e08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e608dd4f99004bb795de5e05ea700e08 already in progress: creating tablet
I20240502 08:16:55.289774 24181 rpcz_store.cc:276] 0502 08:16:48.269290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.269394 (+ 104us) service_pool.cc:225] Handling call
0502 08:16:55.289541 (+7020147us) ts_tablet_manager.cc:1120] Deleting tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58
0502 08:16:55.289549 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.289648 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7019942}
I20240502 08:16:55.290380 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 761) took 6981 ms. Trace:
W20240502 08:16:55.290639 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58 already in progress: creating tablet
I20240502 08:16:55.294075 24181 rpcz_store.cc:276] 0502 08:16:48.309000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.309072 (+ 72us) service_pool.cc:225] Handling call
0502 08:16:55.290272 (+6981200us) ts_tablet_manager.cc:1120] Deleting tablet 7d07efb303b64ce19d993c4c76a47dfc
0502 08:16:55.290283 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.290364 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6980980}
I20240502 08:16:55.294514 24181 tablet_replica.cc:330] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:55.293808 24173 consensus_queue.cc:260] T 6e7432f1df94431a83cf68356e3e78a3 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:55.294625 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 7d07efb303b64ce19d993c4c76a47dfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d07efb303b64ce19d993c4c76a47dfc already in progress: creating tablet
I20240502 08:16:55.296094 24173 ts_tablet_manager.cc:1398] T 6e7432f1df94431a83cf68356e3e78a3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 08:16:55.296384 24173 tablet_bootstrap.cc:492] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:55.297372 24051 raft_consensus.cc:3037] T 6e7432f1df94431a83cf68356e3e78a3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.298072 24173 tablet_bootstrap.cc:654] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.300091 24174 tablet_bootstrap.cc:654] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.304724 25494 consensus_queue.cc:1035] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:55.312340 25012 ts_tablet_manager.cc:1888] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:16:55.312501 25012 log.cc:1196] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/e134e90efb4a48bb82b53a80bda708b3
I20240502 08:16:55.312922 25012 ts_tablet_manager.cc:1909] T e134e90efb4a48bb82b53a80bda708b3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:55.313237 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 759) took 6492 ms. Trace:
I20240502 08:16:55.313513 25012 rpcz_store.cc:276] 0502 08:16:48.820772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.820813 (+ 41us) service_pool.cc:225] Handling call
0502 08:16:55.265522 (+6444709us) ts_tablet_manager.cc:1120] Deleting tablet e134e90efb4a48bb82b53a80bda708b3
0502 08:16:55.265532 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.271655 (+ 6123us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.312241 (+ 40586us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.313220 (+ 979us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6443983}
I20240502 08:16:55.317982 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 762) took 6491 ms. Trace:
I20240502 08:16:55.318130 25012 rpcz_store.cc:276] 0502 08:16:48.826788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.826843 (+ 55us) service_pool.cc:225] Handling call
0502 08:16:55.317822 (+6490979us) ts_tablet_manager.cc:1120] Deleting tablet 571886fdbc2c43e19c790b2b7830ec27
0502 08:16:55.317833 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.317954 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6490823}
I20240502 08:16:55.318445 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet e134e90efb4a48bb82b53a80bda708b3 (table default.Table_af464bf1ebe54d7281005831aa79b8bf [id=47682c26a85842c7b8991bf3828a9336]) successfully deleted
I20240502 08:16:55.318615 25502 raft_consensus.cc:482] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:16:55.318761 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 571886fdbc2c43e19c790b2b7830ec27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 571886fdbc2c43e19c790b2b7830ec27 already in progress: creating tablet
I20240502 08:16:55.318727 25502 raft_consensus.cc:504] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:55.320008 23858 leader_election.cc:341] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: a0224317e65747e1add533d33472f7f3
I20240502 08:16:55.320454 25502 leader_election.cc:288] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:55.320668 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 763) took 6493 ms. Trace:
I20240502 08:16:55.320731 25012 rpcz_store.cc:276] 0502 08:16:48.826973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.827020 (+ 47us) service_pool.cc:225] Handling call
0502 08:16:55.320559 (+6493539us) ts_tablet_manager.cc:1120] Deleting tablet e37d58fbcf6b476fb8c4558b3763c94f
0502 08:16:55.320570 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.320654 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6491334}
I20240502 08:16:55.321063 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0224317e65747e1add533d33472f7f3" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:55.321219 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 770) took 6422 ms. Trace:
I20240502 08:16:55.321281 24054 raft_consensus.cc:2379] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in this term.
I20240502 08:16:55.321280 25012 rpcz_store.cc:276] 0502 08:16:48.898804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.898874 (+ 70us) service_pool.cc:225] Handling call
0502 08:16:55.321131 (+6422257us) ts_tablet_manager.cc:1120] Deleting tablet 0e559ce43d2c440fa50715f25cc2c7e9
0502 08:16:55.321139 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.321205 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6422063}
I20240502 08:16:55.321791 24181 raft_consensus.cc:2229] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:55.321794 23861 leader_election.cc:302] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:55.321987 24181 raft_consensus.cc:2258] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.322639 24181 ts_tablet_manager.cc:1875] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:55.322635 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e37d58fbcf6b476fb8c4558b3763c94f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e37d58fbcf6b476fb8c4558b3763c94f already in progress: creating tablet
W20240502 08:16:55.322885 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 0e559ce43d2c440fa50715f25cc2c7e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e559ce43d2c440fa50715f25cc2c7e9 already in progress: creating tablet
I20240502 08:16:55.323707 24175 tablet_bootstrap.cc:492] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:55.323873 24175 ts_tablet_manager.cc:1367] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.003s
I20240502 08:16:55.324548 24175 raft_consensus.cc:348] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.324776 24175 raft_consensus.cc:374] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.324859 24175 raft_consensus.cc:729] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.325109 24175 consensus_queue.cc:260] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.325419 25408 raft_consensus.cc:3037] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.328773 25408 raft_consensus.cc:2726] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:55.328908 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet bb57cb00e884420bacc48f16739eb29c from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:55.329142 25296 raft_consensus.cc:482] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.329505 24175 ts_tablet_manager.cc:1398] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240502 08:16:55.329669 25296 raft_consensus.cc:504] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.329864 24175 tablet_bootstrap.cc:492] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:55.330366 25296 leader_election.cc:288] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:55.330843 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47459b5f5bb4ecb96f76240fefa1f5d" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:55.331768 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47459b5f5bb4ecb96f76240fefa1f5d" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:55.332361 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 775) took 6432 ms. Trace:
I20240502 08:16:55.332450 25012 rpcz_store.cc:276] 0502 08:16:48.899562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.899911 (+ 349us) service_pool.cc:225] Handling call
0502 08:16:55.332235 (+6432324us) ts_tablet_manager.cc:1120] Deleting tablet 982333cbce18439da3b35693b3f6ca0a
0502 08:16:55.332247 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.332343 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6432166}
I20240502 08:16:55.333636 24175 tablet_bootstrap.cc:654] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.334383 25012 tablet_replica.cc:330] T e549cad1b3be4b28abb0bf5af720eb52 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
W20240502 08:16:55.334367 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 982333cbce18439da3b35693b3f6ca0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 982333cbce18439da3b35693b3f6ca0a already in progress: creating tablet
W20240502 08:16:55.331350 23723 leader_election.cc:341] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:55.334815 23723 leader_election.cc:341] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.334908 23723 leader_election.cc:302] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:55.332406 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet 25c089f0c1b14e419f250d03d4eaad12 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e56af7b42cb481a8363d60f49ee3f61) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:55.335520 25012 raft_consensus.cc:2229] T e549cad1b3be4b28abb0bf5af720eb52 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:55.335754 25012 raft_consensus.cc:2258] T e549cad1b3be4b28abb0bf5af720eb52 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.338064 25296 raft_consensus.cc:2726] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:55.338476 25012 ts_tablet_manager.cc:1875] T e549cad1b3be4b28abb0bf5af720eb52 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.338827 23610 master_service.cc:946] ReplaceTablet: received request to replace tablet 6e7432f1df94431a83cf68356e3e78a3 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:55.339388 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_136c710cf7d94381a2afea174adb4e56" } new_table_name: "default.Table_4dd883502bdd4291ac0d3f15a4548db5" modify_external_catalogs: true
I20240502 08:16:55.346362 24173 tablet_bootstrap.cc:492] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:55.346591 24173 ts_tablet_manager.cc:1367] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.002s
W20240502 08:16:55.346874 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b9f0b0afc3c47759e0e94ca44cf9711 consensus state has no leader
I20240502 08:16:55.347394 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_cf7ced0046f647eb82829af1fb7c985b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:55.347441 24173 raft_consensus.cc:348] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.347610 24173 raft_consensus.cc:374] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.347651 24173 raft_consensus.cc:729] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.347905 24173 consensus_queue.cc:260] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.348621 24173 ts_tablet_manager.cc:1398] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:55.348996 24173 tablet_bootstrap.cc:492] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:55.352308 24175 tablet_bootstrap.cc:492] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:55.352469 24175 ts_tablet_manager.cc:1367] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.002s
I20240502 08:16:55.353221 24175 raft_consensus.cc:348] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.353618 24175 raft_consensus.cc:374] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.353713 24175 raft_consensus.cc:729] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.353982 24175 consensus_queue.cc:260] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.354136 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_38d10d7d3ecf4f89accb1c4dbfebdbea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:55.354562 24175 ts_tablet_manager.cc:1398] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:16:55.354827 24175 tablet_bootstrap.cc:492] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:55.356544 24175 tablet_bootstrap.cc:654] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.360481 24180 ts_tablet_manager.cc:1888] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:55.360634 24180 log.cc:1196] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/1ba1baa2279045b3adc58d187a12cbf3
I20240502 08:16:55.361008 24180 ts_tablet_manager.cc:1909] T 1ba1baa2279045b3adc58d187a12cbf3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:55.361724 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5520c58206f4499bac98e73fb4122056" } new_table_name: "default.Table_182e745428984cea82aebec013cccc1b" modify_external_catalogs: true
I20240502 08:16:55.363541 24173 tablet_bootstrap.cc:654] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.362975 25012 ts_tablet_manager.cc:1888] T e549cad1b3be4b28abb0bf5af720eb52 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:55.370985 25012 log.cc:1196] T e549cad1b3be4b28abb0bf5af720eb52 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/e549cad1b3be4b28abb0bf5af720eb52
I20240502 08:16:55.371412 25012 ts_tablet_manager.cc:1909] T e549cad1b3be4b28abb0bf5af720eb52 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
W20240502 08:16:55.372284 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf7ced0046f647eb82829af1fb7c985b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.373459 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 780) took 6473 ms. Trace:
I20240502 08:16:55.373641 25012 rpcz_store.cc:276] 0502 08:16:48.899989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.900077 (+ 88us) service_pool.cc:225] Handling call
0502 08:16:55.334339 (+6434262us) ts_tablet_manager.cc:1120] Deleting tablet e549cad1b3be4b28abb0bf5af720eb52
0502 08:16:55.334347 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.348029 (+ 13682us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.362878 (+ 14849us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.373437 (+ 10559us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6434159}
I20240502 08:16:55.364666 24181 ts_tablet_manager.cc:1888] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:55.374141 24181 log.cc:1196] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/e549cad1b3be4b28abb0bf5af720eb52
I20240502 08:16:55.374555 24181 ts_tablet_manager.cc:1909] T e549cad1b3be4b28abb0bf5af720eb52 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:55.374634 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet e549cad1b3be4b28abb0bf5af720eb52 (table default.Table_b461b560f3094407a1766253cfc2ece7 [id=4b9348e7a75242788274eee8f4c94838]) successfully deleted
I20240502 08:16:55.374783 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 776) took 6475 ms. Trace:
I20240502 08:16:55.374889 23972 heartbeater.cc:347] Connected to a master server at 127.22.225.125:36189
W20240502 08:16:55.377085 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e03c13b1cb584f5994dff77692b0f442 consensus state has no leader
I20240502 08:16:55.377189 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 935) took 3212 ms. Trace:
I20240502 08:16:55.377271 24180 rpcz_store.cc:276] 0502 08:16:52.164859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.164897 (+ 38us) service_pool.cc:225] Handling call
0502 08:16:55.089771 (+2924874us) ts_tablet_manager.cc:1120] Deleting tablet 1ba1baa2279045b3adc58d187a12cbf3
0502 08:16:55.089782 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.317015 (+227233us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.360370 (+ 43355us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.377165 (+ 16795us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2920548}
I20240502 08:16:55.377606 24180 tablet_replica.cc:330] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:55.377753 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 1ba1baa2279045b3adc58d187a12cbf3 (table default.Table_1704983c4496461d9fa8ea54781f5cb1 [id=1dbff884223d4ad9843fe2512d1f295c]) successfully deleted
I20240502 08:16:55.377795 24180 raft_consensus.cc:2229] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:55.377934 24180 raft_consensus.cc:2258] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.379559 24180 ts_tablet_manager.cc:1875] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.380426 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 762) took 6975 ms. Trace:
I20240502 08:16:55.380558 24181 rpcz_store.cc:276] 0502 08:16:48.405370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.405434 (+ 64us) service_pool.cc:225] Handling call
0502 08:16:55.294462 (+6889028us) ts_tablet_manager.cc:1120] Deleting tablet e549cad1b3be4b28abb0bf5af720eb52
0502 08:16:55.294473 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.349658 (+ 55185us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.364577 (+ 14919us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.380406 (+ 15829us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3908,"tablet-delete.queue_time_us":6888711}
I20240502 08:16:55.381129 25012 rpcz_store.cc:276] 0502 08:16:48.899647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.900203 (+ 556us) service_pool.cc:225] Handling call
0502 08:16:55.374658 (+6474455us) ts_tablet_manager.cc:1120] Deleting tablet 725982e4712e442d8ff1fd3f796e18f5
0502 08:16:55.374670 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.374765 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6474308}
I20240502 08:16:55.381561 24181 tablet_replica.cc:330] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:55.381736 25012 tablet_replica.cc:330] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:55.381806 24181 raft_consensus.cc:2229] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:55.382093 25012 raft_consensus.cc:2229] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:55.382161 24181 raft_consensus.cc:2258] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.382455 25012 pending_rounds.cc:62] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b: Trying to abort 1 pending ops.
I20240502 08:16:55.382599 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet e549cad1b3be4b28abb0bf5af720eb52 (table default.Table_b461b560f3094407a1766253cfc2ece7 [id=4b9348e7a75242788274eee8f4c94838]) successfully deleted
I20240502 08:16:55.382652 25012 pending_rounds.cc:68] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156473876070400 op_type: NO_OP noop_request { }
I20240502 08:16:55.382752 25012 raft_consensus.cc:2866] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:55.382833 25012 raft_consensus.cc:2258] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.382833 24181 ts_tablet_manager.cc:1875] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:55.382848 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 725982e4712e442d8ff1fd3f796e18f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 725982e4712e442d8ff1fd3f796e18f5 already in progress: creating tablet
I20240502 08:16:55.382978 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_2d4cb7ffff364fe6becc190a0dadfb25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:55.383294 25012 ts_tablet_manager.cc:1875] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:55.383482 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d4cb7ffff364fe6becc190a0dadfb25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.390122 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3e0123590a92473dabaacc9c158859e7" } modify_external_catalogs: true
I20240502 08:16:55.390481 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3e0123590a92473dabaacc9c158859e7" } modify_external_catalogs: true
I20240502 08:16:55.392483 24173 tablet_bootstrap.cc:492] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:55.392699 24173 ts_tablet_manager.cc:1367] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.003s
I20240502 08:16:55.393400 24173 raft_consensus.cc:348] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.393638 24173 raft_consensus.cc:374] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.393740 24173 raft_consensus.cc:729] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.393983 24173 consensus_queue.cc:260] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.394490 24173 ts_tablet_manager.cc:1398] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:55.398809 25502 consensus_queue.cc:1035] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:55.407451 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38d10d7d3ecf4f89accb1c4dbfebdbea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.411149 24173 tablet_bootstrap.cc:492] T 128c88af54d24ffd895e9de19cdb4c8a P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:55.416694 24181 ts_tablet_manager.cc:1888] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:55.418730 25012 ts_tablet_manager.cc:1888] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:55.420732 24180 ts_tablet_manager.cc:1888] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:55.420847 24180 log.cc:1196] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/24a4104f4cb64889ad66810788ad6b29
I20240502 08:16:55.421221 24180 ts_tablet_manager.cc:1909] T 24a4104f4cb64889ad66810788ad6b29 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:55.424644 25012 log.cc:1196] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/c1e746013fde4de4862f213792e235ed
I20240502 08:16:55.426441 25012 ts_tablet_manager.cc:1909] T c1e746013fde4de4862f213792e235ed P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:55.428434 23920 raft_consensus.cc:3037] T 751808dfc1964b539f6f836d786156aa P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.430078 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 777) took 6530 ms. Trace:
I20240502 08:16:55.430153 25012 rpcz_store.cc:276] 0502 08:16:48.899722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.900379 (+ 657us) service_pool.cc:225] Handling call
0502 08:16:55.381699 (+6481320us) ts_tablet_manager.cc:1120] Deleting tablet c1e746013fde4de4862f213792e235ed
0502 08:16:55.381706 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.395205 (+ 13499us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.418645 (+ 23440us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.430058 (+ 11413us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6481153}
I20240502 08:16:55.431300 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 778) took 6531 ms. Trace:
I20240502 08:16:55.431372 25012 rpcz_store.cc:276] 0502 08:16:48.899796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.900576 (+ 780us) service_pool.cc:225] Handling call
0502 08:16:55.431171 (+6530595us) ts_tablet_manager.cc:1120] Deleting tablet 9a2cbe751dc4450eaa3eb60268ad510d
0502 08:16:55.431180 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.431281 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6530442}
I20240502 08:16:55.435212 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet c1e746013fde4de4862f213792e235ed (table default.Table_42d6187c77b347e58022b68ed044f3fe [id=ecfe360ec5554bdcac562a1c614e1672]) successfully deleted
W20240502 08:16:55.435564 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 9a2cbe751dc4450eaa3eb60268ad510d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a2cbe751dc4450eaa3eb60268ad510d already in progress: creating tablet
I20240502 08:16:55.437220 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 779) took 6537 ms. Trace:
I20240502 08:16:55.437673 24181 log.cc:1196] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/f797d3c6bdc4493398954fbbf9943971
I20240502 08:16:55.437656 25012 rpcz_store.cc:276] 0502 08:16:48.899888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.900174 (+ 286us) service_pool.cc:225] Handling call
0502 08:16:55.437094 (+6536920us) ts_tablet_manager.cc:1120] Deleting tablet b123bb81d0b44d4ea6443536168d085c
0502 08:16:55.437103 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.437201 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":894,"tablet-delete.queue_time_us":6536742}
I20240502 08:16:55.438150 24181 ts_tablet_manager.cc:1909] T f797d3c6bdc4493398954fbbf9943971 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:55.438769 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 774) took 6539 ms. Trace:
I20240502 08:16:55.438994 25012 rpcz_store.cc:276] 0502 08:16:48.899485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.899813 (+ 328us) service_pool.cc:225] Handling call
0502 08:16:55.438425 (+6538612us) ts_tablet_manager.cc:1120] Deleting tablet e9133e69e1724a26bc3650ea658ca349
0502 08:16:55.438435 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.438751 (+ 316us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6534806}
I20240502 08:16:55.439142 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 763) took 7016 ms. Trace:
W20240502 08:16:55.439272 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet b123bb81d0b44d4ea6443536168d085c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b123bb81d0b44d4ea6443536168d085c already in progress: creating tablet
W20240502 08:16:55.439585 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e9133e69e1724a26bc3650ea658ca349 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9133e69e1724a26bc3650ea658ca349 already in progress: creating tablet
I20240502 08:16:55.439833 25012 tablet_replica.cc:330] T 5a1e208733844829b13222e00a432439 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:55.440230 25012 raft_consensus.cc:2229] T 5a1e208733844829b13222e00a432439 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:55.441090 25012 raft_consensus.cc:2258] T 5a1e208733844829b13222e00a432439 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.442054 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet dfd897ba97204b6788fd2d99cf2a0303 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:55.444096 24174 tablet_bootstrap.cc:492] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:55.443979 24181 rpcz_store.cc:276] 0502 08:16:48.422738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.426445 (+ 3707us) service_pool.cc:225] Handling call
0502 08:16:55.381502 (+6955057us) ts_tablet_manager.cc:1120] Deleting tablet f797d3c6bdc4493398954fbbf9943971
0502 08:16:55.381510 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.385500 (+ 3990us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.416581 (+ 31081us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.439120 (+ 22539us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6954860}
I20240502 08:16:55.444394 24174 ts_tablet_manager.cc:1367] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.258s user 0.004s sys 0.000s
I20240502 08:16:55.426367 24173 tablet_bootstrap.cc:654] T 128c88af54d24ffd895e9de19cdb4c8a P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.444774 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet f797d3c6bdc4493398954fbbf9943971 (table default.Table_7f44f7f1aed6460cb1e73392da0987b8 [id=07d8592503e843e18faedfcce845a974]) successfully deleted
I20240502 08:16:55.445327 24174 raft_consensus.cc:348] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.445605 24174 raft_consensus.cc:374] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.445679 24174 raft_consensus.cc:729] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.445899 24174 consensus_queue.cc:260] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.446162 25012 ts_tablet_manager.cc:1875] T 5a1e208733844829b13222e00a432439 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.449772 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 764) took 7019 ms. Trace:
I20240502 08:16:55.449911 24181 rpcz_store.cc:276] 0502 08:16:48.430168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.430383 (+ 215us) service_pool.cc:225] Handling call
0502 08:16:55.444441 (+7014058us) ts_tablet_manager.cc:1120] Deleting tablet 18625a1d971c43549b9f80e6046efc9c
0502 08:16:55.444452 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.449752 (+ 5300us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7013833}
I20240502 08:16:55.450321 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 937) took 3284 ms. Trace:
I20240502 08:16:55.450596 24181 tablet_replica.cc:330] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:55.450814 24181 raft_consensus.cc:2229] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:55.450937 24180 rpcz_store.cc:276] 0502 08:16:52.165468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.165549 (+ 81us) service_pool.cc:225] Handling call
0502 08:16:55.377548 (+3211999us) ts_tablet_manager.cc:1120] Deleting tablet 24a4104f4cb64889ad66810788ad6b29
0502 08:16:55.377557 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.417802 (+ 40245us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.420633 (+ 2831us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.450301 (+ 29668us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3208237}
W20240502 08:16:55.451176 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 18625a1d971c43549b9f80e6046efc9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18625a1d971c43549b9f80e6046efc9c already in progress: creating tablet
I20240502 08:16:55.451517 24181 raft_consensus.cc:2258] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.452373 24181 ts_tablet_manager.cc:1875] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.452646 24180 tablet_replica.cc:330] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:55.452863 24180 raft_consensus.cc:2229] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:55.453049 24180 raft_consensus.cc:2258] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.452528 24173 tablet_bootstrap.cc:492] T 128c88af54d24ffd895e9de19cdb4c8a P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:55.453769 24173 ts_tablet_manager.cc:1367] T 128c88af54d24ffd895e9de19cdb4c8a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20240502 08:16:55.454591 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b04d67567cc04ca49aeb83ae20849d4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:55.454684 24173 raft_consensus.cc:348] T 128c88af54d24ffd895e9de19cdb4c8a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.454943 24173 raft_consensus.cc:374] T 128c88af54d24ffd895e9de19cdb4c8a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.455003 24173 raft_consensus.cc:729] T 128c88af54d24ffd895e9de19cdb4c8a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.455335 24173 consensus_queue.cc:260] T 128c88af54d24ffd895e9de19cdb4c8a P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.456018 24173 ts_tablet_manager.cc:1398] T 128c88af54d24ffd895e9de19cdb4c8a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:55.456256 24174 ts_tablet_manager.cc:1398] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 08:16:55.456367 24173 tablet_bootstrap.cc:492] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:55.457144 24174 tablet_bootstrap.cc:492] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:55.458339 24173 tablet_bootstrap.cc:654] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:55.460289 23723 consensus_peers.cc:485] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:55.461668 24180 ts_tablet_manager.cc:1875] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.468291 25012 ts_tablet_manager.cc:1888] T 5a1e208733844829b13222e00a432439 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:55.468528 25012 log.cc:1196] T 5a1e208733844829b13222e00a432439 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/5a1e208733844829b13222e00a432439
I20240502 08:16:55.468969 25012 ts_tablet_manager.cc:1909] T 5a1e208733844829b13222e00a432439 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:55.469343 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 781) took 4648 ms. Trace:
I20240502 08:16:55.473616 25012 rpcz_store.cc:276] 0502 08:16:50.820615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:50.820668 (+ 53us) service_pool.cc:225] Handling call
0502 08:16:55.439601 (+4618933us) ts_tablet_manager.cc:1120] Deleting tablet 5a1e208733844829b13222e00a432439
0502 08:16:55.439787 (+ 186us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.451139 (+ 11352us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.468170 (+ 17031us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.469324 (+ 1154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4451928}
I20240502 08:16:55.474195 25012 tablet_replica.cc:330] T f5ccd43e78574f3288b199fa14bcc0e8 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:55.474576 25012 raft_consensus.cc:2229] T f5ccd43e78574f3288b199fa14bcc0e8 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:55.474822 25012 raft_consensus.cc:2258] T f5ccd43e78574f3288b199fa14bcc0e8 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.475028 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 5a1e208733844829b13222e00a432439 (table default.Table_7c04460336ea48e79db571d95cdb1bcc [id=8ea9f7fff43846dfa8310549e8e57f2f]) successfully deleted
W20240502 08:16:55.476442 23993 consensus_peers.cc:485] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24a4104f4cb64889ad66810788ad6b29. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:55.476900 25012 ts_tablet_manager.cc:1875] T f5ccd43e78574f3288b199fa14bcc0e8 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.480238 24181 ts_tablet_manager.cc:1888] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:16:55.482882 24180 ts_tablet_manager.cc:1888] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:55.483337 24180 log.cc:1196] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/751808dfc1964b539f6f836d786156aa
I20240502 08:16:55.487687 24175 tablet_bootstrap.cc:492] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:55.487839 24175 ts_tablet_manager.cc:1367] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.133s user 0.002s sys 0.011s
I20240502 08:16:55.488516 24175 raft_consensus.cc:348] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.488705 24175 raft_consensus.cc:374] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.488762 24175 raft_consensus.cc:729] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.489006 24175 consensus_queue.cc:260] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.489542 24175 ts_tablet_manager.cc:1398] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:55.489912 24175 tablet_bootstrap.cc:492] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:55.491585 24175 tablet_bootstrap.cc:654] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.484189 24173 tablet_bootstrap.cc:492] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:55.493065 24173 ts_tablet_manager.cc:1367] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20240502 08:16:55.494136 24173 raft_consensus.cc:348] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.494416 24173 raft_consensus.cc:374] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.494547 24173 raft_consensus.cc:729] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.494866 24173 consensus_queue.cc:260] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.496119 24173 ts_tablet_manager.cc:1398] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:55.496471 24173 tablet_bootstrap.cc:492] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:55.498183 24173 tablet_bootstrap.cc:654] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.451934 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 24a4104f4cb64889ad66810788ad6b29 (table default.Table_307041e730f248a5bd4d265f08aad5e4 [id=f2316137abe74f5c8e79faf0c4638540]) successfully deleted
W20240502 08:16:55.498541 23594 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 46601.2us
W20240502 08:16:55.499357 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b04d67567cc04ca49aeb83ae20849d4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.505282 24181 log.cc:1196] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/e134e90efb4a48bb82b53a80bda708b3
I20240502 08:16:55.505774 24181 ts_tablet_manager.cc:1909] T e134e90efb4a48bb82b53a80bda708b3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:55.506690 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 767) took 7047 ms. Trace:
I20240502 08:16:55.506824 24181 rpcz_store.cc:276] 0502 08:16:48.459357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.459412 (+ 55us) service_pool.cc:225] Handling call
0502 08:16:55.450553 (+6991141us) ts_tablet_manager.cc:1120] Deleting tablet e134e90efb4a48bb82b53a80bda708b3
0502 08:16:55.450562 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.461391 (+ 10829us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.480113 (+ 18722us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.506671 (+ 26558us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6990943}
I20240502 08:16:55.507444 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 770) took 7006 ms. Trace:
I20240502 08:16:55.507570 24181 rpcz_store.cc:276] 0502 08:16:48.500644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.500693 (+ 49us) service_pool.cc:225] Handling call
0502 08:16:55.507317 (+7006624us) ts_tablet_manager.cc:1120] Deleting tablet 571886fdbc2c43e19c790b2b7830ec27
0502 08:16:55.507327 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.507420 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7006448}
I20240502 08:16:55.507731 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet e134e90efb4a48bb82b53a80bda708b3 (table default.Table_af464bf1ebe54d7281005831aa79b8bf [id=47682c26a85842c7b8991bf3828a9336]) successfully deleted
I20240502 08:16:55.508056 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 771) took 7006 ms. Trace:
W20240502 08:16:55.508086 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 571886fdbc2c43e19c790b2b7830ec27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 571886fdbc2c43e19c790b2b7830ec27 already in progress: creating tablet
I20240502 08:16:55.508124 24181 rpcz_store.cc:276] 0502 08:16:48.501295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.501353 (+ 58us) service_pool.cc:225] Handling call
0502 08:16:55.507945 (+7006592us) ts_tablet_manager.cc:1120] Deleting tablet e37d58fbcf6b476fb8c4558b3763c94f
0502 08:16:55.507955 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.508039 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7006434}
W20240502 08:16:55.508527 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e37d58fbcf6b476fb8c4558b3763c94f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e37d58fbcf6b476fb8c4558b3763c94f already in progress: creating tablet
I20240502 08:16:55.508870 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 778) took 6966 ms. Trace:
I20240502 08:16:55.508934 24181 rpcz_store.cc:276] 0502 08:16:48.542604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.542657 (+ 53us) service_pool.cc:225] Handling call
0502 08:16:55.508755 (+6966098us) ts_tablet_manager.cc:1120] Deleting tablet 0e559ce43d2c440fa50715f25cc2c7e9
0502 08:16:55.508763 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.508856 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6965889}
W20240502 08:16:55.509305 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 0e559ce43d2c440fa50715f25cc2c7e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e559ce43d2c440fa50715f25cc2c7e9 already in progress: creating tablet
I20240502 08:16:55.511528 25012 ts_tablet_manager.cc:1888] T f5ccd43e78574f3288b199fa14bcc0e8 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:55.511683 25012 log.cc:1196] T f5ccd43e78574f3288b199fa14bcc0e8 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/f5ccd43e78574f3288b199fa14bcc0e8
I20240502 08:16:55.512065 25012 ts_tablet_manager.cc:1909] T f5ccd43e78574f3288b199fa14bcc0e8 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:55.512387 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 850) took 4410 ms. Trace:
I20240502 08:16:55.512507 25012 rpcz_store.cc:276] 0502 08:16:51.101979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.781770 (+1679791us) service_pool.cc:225] Handling call
0502 08:16:55.474143 (+2692373us) ts_tablet_manager.cc:1120] Deleting tablet f5ccd43e78574f3288b199fa14bcc0e8
0502 08:16:55.474153 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.508224 (+ 34071us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.511416 (+ 3192us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.512367 (+ 951us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2692200}
I20240502 08:16:55.512959 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet f5ccd43e78574f3288b199fa14bcc0e8 (table default.Table_95df2e76edc746a08164b4ff25809909 [id=d5a04f8005714d15a3710236c85b1317]) successfully deleted
I20240502 08:16:55.513047 25012 tablet_replica.cc:330] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:55.513231 25012 raft_consensus.cc:2229] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:55.515897 25012 raft_consensus.cc:2258] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:55.516389 23993 consensus_peers.cc:485] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb7b37f776df47da9aa6e7fba82a8999. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:55.516793 25012 ts_tablet_manager.cc:1875] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.521641 24181 tablet_replica.cc:330] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:55.522006 24181 raft_consensus.cc:2229] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:55.522421 24181 raft_consensus.cc:2258] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.522719 24174 tablet_bootstrap.cc:654] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.523330 25405 raft_consensus.cc:482] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.523847 24180 ts_tablet_manager.cc:1909] T 751808dfc1964b539f6f836d786156aa P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:55.523792 24173 tablet_bootstrap.cc:492] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:55.523977 25405 raft_consensus.cc:504] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.524175 24175 tablet_bootstrap.cc:492] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:55.524176 24173 ts_tablet_manager.cc:1367] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240502 08:16:55.524289 24175 ts_tablet_manager.cc:1367] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.005s
I20240502 08:16:55.524641 25405 leader_election.cc:288] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.524718 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 929) took 3362 ms. Trace:
I20240502 08:16:55.524803 24180 rpcz_store.cc:276] 0502 08:16:52.162130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.165684 (+ 3554us) service_pool.cc:225] Handling call
0502 08:16:55.451773 (+3286089us) ts_tablet_manager.cc:1120] Deleting tablet 751808dfc1964b539f6f836d786156aa
0502 08:16:55.452605 (+ 832us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.464634 (+ 12029us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.482779 (+ 18145us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.524695 (+ 41916us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3282313}
I20240502 08:16:55.525014 24175 raft_consensus.cc:348] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.525187 24175 raft_consensus.cc:374] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.525246 24175 raft_consensus.cc:729] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.525292 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 751808dfc1964b539f6f836d786156aa (table default.Table_ad81523678c3450c880efd1ecfc5f95e [id=911d4d5a86e54a5696d37a2bd9f69733]) successfully deleted
I20240502 08:16:55.525506 24175 consensus_queue.cc:260] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.525653 24173 raft_consensus.cc:348] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.525774 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 938) took 3188 ms. Trace:
I20240502 08:16:55.525756 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001e3f4f72f64dc69dbd521724912f07" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:55.525847 24180 rpcz_store.cc:276] 0502 08:16:52.337776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.338454 (+ 678us) service_pool.cc:225] Handling call
0502 08:16:55.525652 (+3187198us) ts_tablet_manager.cc:1120] Deleting tablet 87a3df4fbb224dd0b4ec6141584d374b
0502 08:16:55.525662 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.525757 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3186973}
I20240502 08:16:55.525885 24173 raft_consensus.cc:374] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.525962 23920 raft_consensus.cc:2454] T 001e3f4f72f64dc69dbd521724912f07 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:55.526104 24173 raft_consensus.cc:729] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.526304 24175 ts_tablet_manager.cc:1398] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240502 08:16:55.526301 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 87a3df4fbb224dd0b4ec6141584d374b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87a3df4fbb224dd0b4ec6141584d374b already in progress: creating tablet
I20240502 08:16:55.526348 23996 leader_election.cc:302] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:55.526384 24173 consensus_queue.cc:260] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.526681 25317 raft_consensus.cc:2781] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:55.526760 25317 raft_consensus.cc:482] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:55.526865 25317 raft_consensus.cc:3037] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:55.530602 23993 consensus_peers.cc:485] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 751808dfc1964b539f6f836d786156aa. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:55.529511 25317 raft_consensus.cc:504] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.530189 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001e3f4f72f64dc69dbd521724912f07" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:55.531349 25317 leader_election.cc:288] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.531980 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001e3f4f72f64dc69dbd521724912f07" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:55.532034 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001e3f4f72f64dc69dbd521724912f07" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:55.532219 23920 raft_consensus.cc:3037] T 001e3f4f72f64dc69dbd521724912f07 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.533739 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 939) took 3195 ms. Trace:
I20240502 08:16:55.533826 24180 rpcz_store.cc:276] 0502 08:16:52.337919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.349001 (+ 11082us) service_pool.cc:225] Handling call
0502 08:16:55.533630 (+3184629us) ts_tablet_manager.cc:1120] Deleting tablet 9f3c00e5393c47df924bcb5dcf2a0f19
0502 08:16:55.533640 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.533722 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3184398}
W20240502 08:16:55.534250 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 9f3c00e5393c47df924bcb5dcf2a0f19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f3c00e5393c47df924bcb5dcf2a0f19 already in progress: creating tablet
I20240502 08:16:55.534351 24173 ts_tablet_manager.cc:1398] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240502 08:16:55.534644 24173 tablet_bootstrap.cc:492] T 592ab8523b8d479d9caead321d5888c3 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:55.534754 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 940) took 3196 ms. Trace:
I20240502 08:16:55.534862 24180 rpcz_store.cc:276] 0502 08:16:52.337998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.349269 (+ 11271us) service_pool.cc:225] Handling call
0502 08:16:55.534561 (+3185292us) ts_tablet_manager.cc:1120] Deleting tablet d2bb4dba23ac4b4eb53b3f2d75d0442e
0502 08:16:55.534570 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.534737 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3185179}
I20240502 08:16:55.535174 23920 raft_consensus.cc:2454] T 001e3f4f72f64dc69dbd521724912f07 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
W20240502 08:16:55.535375 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet d2bb4dba23ac4b4eb53b3f2d75d0442e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2bb4dba23ac4b4eb53b3f2d75d0442e already in progress: creating tablet
I20240502 08:16:55.535560 23996 leader_election.cc:302] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:55.535773 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 941) took 3197 ms. Trace:
I20240502 08:16:55.535848 25012 ts_tablet_manager.cc:1888] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:16:55.535840 24180 rpcz_store.cc:276] 0502 08:16:52.338070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.349401 (+ 11331us) service_pool.cc:225] Handling call
0502 08:16:55.535673 (+3186272us) ts_tablet_manager.cc:1120] Deleting tablet 2085ced52fff46ebaa98324cbb2d8c11
0502 08:16:55.535682 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.535757 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3186182}
I20240502 08:16:55.535961 25012 log.cc:1196] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/642744194e22461f8316a78689fd99ff
I20240502 08:16:55.536185 24173 tablet_bootstrap.cc:654] T 592ab8523b8d479d9caead321d5888c3 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:55.536242 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 2085ced52fff46ebaa98324cbb2d8c11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2085ced52fff46ebaa98324cbb2d8c11 already in progress: creating tablet
I20240502 08:16:55.536326 25012 ts_tablet_manager.cc:1909] T 642744194e22461f8316a78689fd99ff P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:55.536564 24180 tablet_replica.cc:330] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:55.536469 23783 raft_consensus.cc:2454] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:55.536810 24180 raft_consensus.cc:2229] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:55.537147 24180 raft_consensus.cc:2258] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.537756 24180 ts_tablet_manager.cc:1875] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.544519 25357 raft_consensus.cc:482] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.544673 25357 raft_consensus.cc:504] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.545432 25357 leader_election.cc:288] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:55.546568 23858 leader_election.cc:341] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 128c88af54d24ffd895e9de19cdb4c8a
I20240502 08:16:55.547015 24175 tablet_bootstrap.cc:492] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:55.549257 24175 tablet_bootstrap.cc:654] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.552073 25418 raft_consensus.cc:482] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.552217 25418 raft_consensus.cc:504] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.552839 25418 leader_election.cc:288] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:55.536233 25317 raft_consensus.cc:2781] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:55.543857 24181 ts_tablet_manager.cc:1875] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.546430 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128c88af54d24ffd895e9de19cdb4c8a" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:55.555450 24174 tablet_bootstrap.cc:492] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:55.555640 24174 ts_tablet_manager.cc:1367] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.099s user 0.004s sys 0.000s
I20240502 08:16:55.556195 23602 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 001e3f4f72f64dc69dbd521724912f07 (table default.Table_d7cff78803d44e5aba3b50cd8ff11e92 [id=5c21117955d946418c656330984ae747]). Expected version 3 got 0
I20240502 08:16:55.557298 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ac7f78ed7ac94104a09a418eed600e2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:55.557897 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac7f78ed7ac94104a09a418eed600e2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.558009 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00f29fb76b44d6ea4f76963d2d0751c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:55.558784 24051 raft_consensus.cc:2454] T 128c88af54d24ffd895e9de19cdb4c8a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:55.558809 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00f29fb76b44d6ea4f76963d2d0751c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:55.559072 24054 raft_consensus.cc:2454] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:55.559404 25317 raft_consensus.cc:686] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:55.559451 23861 leader_election.cc:302] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:55.559885 25357 raft_consensus.cc:2781] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:55.559970 25357 raft_consensus.cc:482] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:55.560045 25357 raft_consensus.cc:3037] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.560015 25317 consensus_queue.cc:237] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.564108 25357 raft_consensus.cc:504] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.560294 23723 leader_election.cc:302] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
W20240502 08:16:55.564853 23723 leader_election.cc:341] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.565243 25357 leader_election.cc:288] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:55.565618 25418 raft_consensus.cc:2781] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:55.565706 25418 raft_consensus.cc:482] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:55.565773 25418 raft_consensus.cc:3037] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.565752 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128c88af54d24ffd895e9de19cdb4c8a" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:55.565980 24054 raft_consensus.cc:3037] T 128c88af54d24ffd895e9de19cdb4c8a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.568459 25418 raft_consensus.cc:504] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.569110 25418 leader_election.cc:288] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:55.570624 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00f29fb76b44d6ea4f76963d2d0751c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:55.571847 23723 leader_election.cc:341] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.573168 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 851) took 4471 ms. Trace:
I20240502 08:16:55.573271 25012 rpcz_store.cc:276] 0502 08:16:51.102060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.781973 (+1679913us) service_pool.cc:225] Handling call
0502 08:16:55.512998 (+2731025us) ts_tablet_manager.cc:1120] Deleting tablet 642744194e22461f8316a78689fd99ff
0502 08:16:55.513007 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.531275 (+ 18268us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.535764 (+ 4489us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.573148 (+ 37384us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2730908}
I20240502 08:16:55.573673 25012 tablet_replica.cc:330] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:55.573889 25012 raft_consensus.cc:2229] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:55.574048 25012 raft_consensus.cc:2258] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.574280 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 642744194e22461f8316a78689fd99ff (table default.Table_8bed20f633e348679b0087687298cc42 [id=18ab2c0716464d82a0e33bd99bd0735d]) successfully deleted
I20240502 08:16:55.574388 25158 consensus_queue.cc:1035] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:55.574698 25012 ts_tablet_manager.cc:1875] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.576368 24175 tablet_bootstrap.cc:492] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:55.576493 24175 ts_tablet_manager.cc:1367] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.003s
I20240502 08:16:55.577154 24175 raft_consensus.cc:348] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.577323 24175 raft_consensus.cc:374] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.577371 24175 raft_consensus.cc:729] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.577550 24175 consensus_queue.cc:260] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.578132 24175 ts_tablet_manager.cc:1398] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:16:55.578554 24175 tablet_bootstrap.cc:492] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:55.579465 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_26889a40ce5a43bcbda607dcda6887bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:55.580825 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26889a40ce5a43bcbda607dcda6887bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.582170 25418 raft_consensus.cc:482] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.582496 25418 raft_consensus.cc:504] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.584002 25418 leader_election.cc:288] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.584730 24054 raft_consensus.cc:2454] T 128c88af54d24ffd895e9de19cdb4c8a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:55.584707 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00f29fb76b44d6ea4f76963d2d0751c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:55.584996 24181 ts_tablet_manager.cc:1888] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:55.585135 24181 log.cc:1196] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/c5c377e037894728ac084b6500ad20de
I20240502 08:16:55.584983 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090bec30f5664421b9bfe3f2810afd14" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:55.585209 23861 leader_election.cc:302] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:55.585556 25357 raft_consensus.cc:2781] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:55.585700 25357 raft_consensus.cc:686] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:16:55.584944 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090bec30f5664421b9bfe3f2810afd14" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:55.585954 25357 consensus_queue.cc:237] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.586755 24175 tablet_bootstrap.cc:654] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:55.586726 23723 leader_election.cc:341] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:55.588305 23723 leader_election.cc:341] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.588409 23723 leader_election.cc:302] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:55.589126 25418 raft_consensus.cc:2726] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:55.589740 23858 leader_election.cc:341] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 128c88af54d24ffd895e9de19cdb4c8a
I20240502 08:16:55.593029 25012 ts_tablet_manager.cc:1888] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:55.596575 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_769de68591d64bdb9b80f7bb337da552" } new_table_name: "default.Table_978b0604ce1c4638bcd51dc8389941d9" modify_external_catalogs: true
I20240502 08:16:55.597965 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet 090bec30f5664421b9bfe3f2810afd14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:55.609253 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e00f744a28c6496e98f875d47bdccf95" } new_table_name: "default.Table_964212922b2c496d98accde52e896b09" modify_external_catalogs: true
I20240502 08:16:55.597241 24175 tablet_bootstrap.cc:492] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:55.611147 24175 ts_tablet_manager.cc:1367] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.001s
W20240502 08:16:55.613294 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e20e126b1c9546e4a194bf9842f444a5 consensus state has no leader
I20240502 08:16:55.614156 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2ecca22d4d934ddbacb4eb1826759e48" } new_table_name: "default.Table_40d78b21141944aea5fe3b77b41a7534" modify_external_catalogs: true
I20240502 08:16:55.620121 23603 catalog_manager.cc:5484] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66). New cstate: current_term: 1 leader_uuid: "5b89a1f0485a44b7ac1710e2369a809b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:55.620523 23603 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 7ed919909289410e867881baab841f53 (table default.Table_2edad641920b44ce9a4d5b7503f0279a [id=ee3912407a934444ad2c62f7b83d93ac]). Expected version 2 got 0
I20240502 08:16:55.628167 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 18d82e75f2624e6782f55987cea223cb (DEFAULT_TABLE table=default.Table_536141575d2c49fe807d39c1f38e3bb1 [id=01eaf80972e949a6927aeaffa17a7714]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.629853 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18d82e75f2624e6782f55987cea223cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.632004 24174 raft_consensus.cc:348] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.632172 24174 raft_consensus.cc:374] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.632211 24174 raft_consensus.cc:729] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.634068 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 0281c8123eab4c48bddc377f5695c6e0
I20240502 08:16:55.635222 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fdc1e83837dc40ca99542547bc5fa957"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:55.635810 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdc1e83837dc40ca99542547bc5fa957 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.637245 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet 0281c8123eab4c48bddc377f5695c6e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:55.638192 23751 tablet_service.cc:1467] Processing CreateTablet for tablet da91c16f5cc3474ea21dbab9ea1dd5be (DEFAULT_TABLE table=default.Table_37c06b36010d4694830aa2c23281c13c [id=2631ced2961f4d5895a160a157a865a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.638578 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da91c16f5cc3474ea21dbab9ea1dd5be. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:55.647970 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 14075c65ed7a4aa78166943aa32f904f consensus state has no leader
I20240502 08:16:55.649165 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 558a3a4a8383439bb64af1d3eaa64572
I20240502 08:16:55.650025 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6
I20240502 08:16:55.651072 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 39cc4ef1174a4358bad777b0471c5f76
I20240502 08:16:55.653857 23759 tablet_service.cc:1467] Processing CreateTablet for tablet b3b1e3e2e9514f289725f4afc9adb116 (DEFAULT_TABLE table=default.Table_b61b626a7be44b199bf2359795269a93 [id=2781d5032e474ba4b9aab3ade8f53695]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.654224 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b1e3e2e9514f289725f4afc9adb116. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.657657 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:55.657838 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 39cc4ef1174a4358bad777b0471c5f76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:55.664712 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet 558a3a4a8383439bb64af1d3eaa64572 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:55.667435 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 36ecf30203634e2792f39c4234cfde1b (DEFAULT_TABLE table=default.Table_2bcd9c267ebf4fd3a2763c589938275f [id=4dd444fe8355478eb81f90a9beeb0c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.667959 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36ecf30203634e2792f39c4234cfde1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.670360 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 5edadeb53a354c39a33ff68ffbde5ad1 (DEFAULT_TABLE table=default.Table_44816133bdb24a48a436fa62d832ed30 [id=0e6e6a4ac8464ede993f94cfef1f896b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.670679 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5edadeb53a354c39a33ff68ffbde5ad1. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:55.677635 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ecde716a0c643cdb0698a37cde1c906 consensus state has no leader
I20240502 08:16:55.584942 24051 raft_consensus.cc:3037] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.684489 24181 ts_tablet_manager.cc:1909] T c5c377e037894728ac084b6500ad20de P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:55.686616 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet 0281c8123eab4c48bddc377f5695c6e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:55.688292 25413 consensus_queue.cc:1035] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.072s
I20240502 08:16:55.689374 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 779) took 7059 ms. Trace:
I20240502 08:16:55.690054 24029 tablet_service.cc:1467] Processing CreateTablet for tablet da91c16f5cc3474ea21dbab9ea1dd5be (DEFAULT_TABLE table=default.Table_37c06b36010d4694830aa2c23281c13c [id=2631ced2961f4d5895a160a157a865a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.692551 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 18d82e75f2624e6782f55987cea223cb (DEFAULT_TABLE table=default.Table_536141575d2c49fe807d39c1f38e3bb1 [id=01eaf80972e949a6927aeaffa17a7714]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.693524 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:55.693733 24022 tablet_service.cc:1514] Processing DeleteTablet for tablet 39cc4ef1174a4358bad777b0471c5f76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:55.694330 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 36ecf30203634e2792f39c4234cfde1b (DEFAULT_TABLE table=default.Table_2bcd9c267ebf4fd3a2763c589938275f [id=4dd444fe8355478eb81f90a9beeb0c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.667526 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 54218d994ab84cc5a5c30161403f7006 (DEFAULT_TABLE table=default.Table_7ff62874389f42d0ba37aba1307ea27b [id=bb8d679cedb944fe845ddb0c64f51e20]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.695057 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54218d994ab84cc5a5c30161403f7006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.696512 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet 558a3a4a8383439bb64af1d3eaa64572 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:55.696556 24173 tablet_bootstrap.cc:492] T 592ab8523b8d479d9caead321d5888c3 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:55.696712 24051 raft_consensus.cc:2454] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:55.697072 24173 ts_tablet_manager.cc:1367] T 592ab8523b8d479d9caead321d5888c3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.162s user 0.019s sys 0.016s
I20240502 08:16:55.697082 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da91c16f5cc3474ea21dbab9ea1dd5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.697360 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 54218d994ab84cc5a5c30161403f7006 (DEFAULT_TABLE table=default.Table_7ff62874389f42d0ba37aba1307ea27b [id=bb8d679cedb944fe845ddb0c64f51e20]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.697698 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 5edadeb53a354c39a33ff68ffbde5ad1 (DEFAULT_TABLE table=default.Table_44816133bdb24a48a436fa62d832ed30 [id=0e6e6a4ac8464ede993f94cfef1f896b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.697703 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18d82e75f2624e6782f55987cea223cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.697888 24173 raft_consensus.cc:348] T 592ab8523b8d479d9caead321d5888c3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.698071 24173 raft_consensus.cc:374] T 592ab8523b8d479d9caead321d5888c3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.698146 24173 raft_consensus.cc:729] T 592ab8523b8d479d9caead321d5888c3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.698383 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet bb57cb00e884420bacc48f16739eb29c
I20240502 08:16:55.698374 24173 consensus_queue.cc:260] T 592ab8523b8d479d9caead321d5888c3 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.698830 23609 catalog_manager.cc:6554] ReplaceTablet: tablet bb57cb00e884420bacc48f16739eb29c deleted and replaced by tablet 82082b214d834186a185e0c65e9c1b41
I20240502 08:16:55.698846 24173 ts_tablet_manager.cc:1398] T 592ab8523b8d479d9caead321d5888c3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:55.699117 24173 tablet_bootstrap.cc:492] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:55.700485 24173 tablet_bootstrap.cc:654] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.701330 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet bb57cb00e884420bacc48f16739eb29c with delete_type TABLET_DATA_DELETED (replaced by tablet 82082b214d834186a185e0c65e9c1b41) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:55.701462 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet bb57cb00e884420bacc48f16739eb29c with delete_type TABLET_DATA_DELETED (replaced by tablet 82082b214d834186a185e0c65e9c1b41) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:55.697167 24037 tablet_service.cc:1467] Processing CreateTablet for tablet b3b1e3e2e9514f289725f4afc9adb116 (DEFAULT_TABLE table=default.Table_b61b626a7be44b199bf2359795269a93 [id=2781d5032e474ba4b9aab3ade8f53695]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.702064 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b1e3e2e9514f289725f4afc9adb116. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.702044 25158 consensus_queue.cc:1035] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:55.702278 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 36dece738afd42b99f3f38d7fd11a0af
I20240502 08:16:55.701798 23723 leader_election.cc:302] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:55.703539 25158 raft_consensus.cc:2781] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:55.703660 25158 raft_consensus.cc:686] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:55.703725 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 36dece738afd42b99f3f38d7fd11a0af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:55.703877 25158 consensus_queue.cc:237] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.704742 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet dfd897ba97204b6788fd2d99cf2a0303
I20240502 08:16:55.705319 23601 catalog_manager.cc:6554] ReplaceTablet: tablet dfd897ba97204b6788fd2d99cf2a0303 deleted and replaced by tablet fd3f59a2654a459e8bfd3be29c203e07
I20240502 08:16:55.705698 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet e4ac2b5073e34cd98e0246620fc1515d from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:55.705803 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54218d994ab84cc5a5c30161403f7006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.706415 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet 36dece738afd42b99f3f38d7fd11a0af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:55.706696 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet dfd897ba97204b6788fd2d99cf2a0303 with delete_type TABLET_DATA_DELETED (replaced by tablet fd3f59a2654a459e8bfd3be29c203e07) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:55.706972 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet dfd897ba97204b6788fd2d99cf2a0303 with delete_type TABLET_DATA_DELETED (replaced by tablet fd3f59a2654a459e8bfd3be29c203e07) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:55.708721 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_07b885519a2f4dcab31df7a12ddd2154"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:55.708679 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b8657ee1f37f41e6a9ac770b3c4fdf73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:55.709259 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07b885519a2f4dcab31df7a12ddd2154 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:55.709357 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8657ee1f37f41e6a9ac770b3c4fdf73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.710095 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5edadeb53a354c39a33ff68ffbde5ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.710683 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_96b57ae1050b4b6e819e085ac1d3c79c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:55.711120 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96b57ae1050b4b6e819e085ac1d3c79c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.712213 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_77228df00cdf4938acabcf03c8b4525e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:55.712639 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77228df00cdf4938acabcf03c8b4525e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.713798 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36ecf30203634e2792f39c4234cfde1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.713831 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0a4a6e3b1dd044f1a0e88c6c03f8c8b6" } new_table_name: "default.Table_3c1db2ea8e3d49efa8439ddf3a30734c" modify_external_catalogs: true
I20240502 08:16:55.715090 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ca502551d3ba4c76868d9a79c06a8569"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:55.715471 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca502551d3ba4c76868d9a79c06a8569 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.716544 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_834d338053304767992670afbbddc67d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:55.717037 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_834d338053304767992670afbbddc67d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.697106 24181 rpcz_store.cc:276] 0502 08:16:48.625274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.625330 (+ 56us) service_pool.cc:225] Handling call
0502 08:16:55.509530 (+6884200us) ts_tablet_manager.cc:1120] Deleting tablet c5c377e037894728ac084b6500ad20de
0502 08:16:55.509538 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.563869 (+ 54331us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.567757 (+ 3888us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.684844 (+117087us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6883933}
I20240502 08:16:55.718178 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 783) took 7051 ms. Trace:
I20240502 08:16:55.718287 24181 rpcz_store.cc:276] 0502 08:16:48.666833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.666999 (+ 166us) service_pool.cc:225] Handling call
0502 08:16:55.717985 (+7050986us) ts_tablet_manager.cc:1120] Deleting tablet e9133e69e1724a26bc3650ea658ca349
0502 08:16:55.717996 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.718132 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7050736}
I20240502 08:16:55.718633 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 784) took 7051 ms. Trace:
W20240502 08:16:55.718690 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e9133e69e1724a26bc3650ea658ca349 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9133e69e1724a26bc3650ea658ca349 already in progress: creating tablet
I20240502 08:16:55.718731 24181 rpcz_store.cc:276] 0502 08:16:48.666942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.667281 (+ 339us) service_pool.cc:225] Handling call
0502 08:16:55.718495 (+7051214us) ts_tablet_manager.cc:1120] Deleting tablet 982333cbce18439da3b35693b3f6ca0a
0502 08:16:55.718504 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.718599 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7051055}
I20240502 08:16:55.719053 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 785) took 7043 ms. Trace:
W20240502 08:16:55.719137 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 982333cbce18439da3b35693b3f6ca0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 982333cbce18439da3b35693b3f6ca0a already in progress: creating tablet
I20240502 08:16:55.719142 24181 rpcz_store.cc:276] 0502 08:16:48.676024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.676080 (+ 56us) service_pool.cc:225] Handling call
0502 08:16:55.718936 (+7042856us) ts_tablet_manager.cc:1120] Deleting tablet 725982e4712e442d8ff1fd3f796e18f5
0502 08:16:55.718944 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.719016 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7042636}
I20240502 08:16:55.719470 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 786) took 6994 ms. Trace:
W20240502 08:16:55.719532 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 725982e4712e442d8ff1fd3f796e18f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 725982e4712e442d8ff1fd3f796e18f5 already in progress: creating tablet
I20240502 08:16:55.719563 24181 rpcz_store.cc:276] 0502 08:16:48.724959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.725040 (+ 81us) service_pool.cc:225] Handling call
0502 08:16:55.719346 (+6994306us) ts_tablet_manager.cc:1120] Deleting tablet 9a2cbe751dc4450eaa3eb60268ad510d
0502 08:16:55.719354 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.719433 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6994102}
I20240502 08:16:55.719911 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 787) took 6972 ms. Trace:
W20240502 08:16:55.719972 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 9a2cbe751dc4450eaa3eb60268ad510d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a2cbe751dc4450eaa3eb60268ad510d already in progress: creating tablet
I20240502 08:16:55.720006 24181 rpcz_store.cc:276] 0502 08:16:48.747061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:48.747122 (+ 61us) service_pool.cc:225] Handling call
0502 08:16:55.719773 (+6972651us) ts_tablet_manager.cc:1120] Deleting tablet b123bb81d0b44d4ea6443536168d085c
0502 08:16:55.719781 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.719884 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6971937}
I20240502 08:16:55.720315 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 856) took 5771 ms. Trace:
W20240502 08:16:55.720368 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet b123bb81d0b44d4ea6443536168d085c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b123bb81d0b44d4ea6443536168d085c already in progress: creating tablet
I20240502 08:16:55.720407 24181 rpcz_store.cc:276] 0502 08:16:49.949170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.208083 (+2258913us) service_pool.cc:225] Handling call
0502 08:16:55.720195 (+3512112us) ts_tablet_manager.cc:1120] Deleting tablet ee3d91d55ffd49ceb20d2c33408f5a5a
0502 08:16:55.720204 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.720279 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3511896}
I20240502 08:16:55.720695 24181 tablet_replica.cc:330] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
W20240502 08:16:55.720837 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet ee3d91d55ffd49ceb20d2c33408f5a5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee3d91d55ffd49ceb20d2c33408f5a5a already in progress: creating tablet
I20240502 08:16:55.720916 24181 raft_consensus.cc:2229] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:55.721077 24181 raft_consensus.cc:2258] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.724992 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a82f807957744746a06a4d90e90f0127"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:55.725700 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a82f807957744746a06a4d90e90f0127 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.735378 25296 raft_consensus.cc:482] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.735599 25296 raft_consensus.cc:504] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.736263 25296 leader_election.cc:288] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.736810 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18d284f71937459db0c0676ce4568175" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:55.737452 23723 leader_election.cc:341] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.742262 24181 ts_tablet_manager.cc:1875] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.747821 25357 consensus_queue.cc:1035] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:55.750883 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7fe583cbf4af493e82d3ec0508ea0f0b consensus state has no leader
I20240502 08:16:55.758697 24174 consensus_queue.cc:260] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.759449 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0287506f7f94418eb19163c272a30fb8" } new_table_name: "default.Table_851f246623304504aa9ddf2e6c5e9082" modify_external_catalogs: true
I20240502 08:16:55.761693 24174 ts_tablet_manager.cc:1398] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.147s user 0.002s sys 0.000s
I20240502 08:16:55.762033 24174 tablet_bootstrap.cc:492] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:55.764302 25296 consensus_queue.cc:1035] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:55.768041 25296 raft_consensus.cc:482] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.768168 25296 raft_consensus.cc:504] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.768803 25296 leader_election.cc:288] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.770203 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f57e3a39d54b7ab372d4b145b7a696" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:55.770836 23723 leader_election.cc:341] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.771929 24181 ts_tablet_manager.cc:1888] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:55.772049 24181 log.cc:1196] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/a0224317e65747e1add533d33472f7f3
I20240502 08:16:55.772387 24181 ts_tablet_manager.cc:1909] T a0224317e65747e1add533d33472f7f3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:55.772698 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 857) took 5823 ms. Trace:
I20240502 08:16:55.772784 24181 rpcz_store.cc:276] 0502 08:16:49.949256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.208327 (+2259071us) service_pool.cc:225] Handling call
0502 08:16:55.720633 (+3512306us) ts_tablet_manager.cc:1120] Deleting tablet a0224317e65747e1add533d33472f7f3
0502 08:16:55.720640 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.745370 (+ 24730us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.771811 (+ 26441us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.772681 (+ 870us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3512168}
I20240502 08:16:55.773094 24181 tablet_replica.cc:330] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:55.773289 24181 raft_consensus.cc:2229] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:55.773265 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet a0224317e65747e1add533d33472f7f3 (table default.Table_5782cc56973c43cfac93b2560c62c688 [id=9a3faa2c394841cfbfab3425282d84d8]) successfully deleted
I20240502 08:16:55.773448 24181 raft_consensus.cc:2258] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.773932 24181 ts_tablet_manager.cc:1875] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:55.780911 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6088f160a13c4ff09f72bed72f69eaf8 consensus state has no leader
I20240502 08:16:55.786708 24174 tablet_bootstrap.cc:654] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:55.791970 23595 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet e20e126b1c9546e4a194bf9842f444a5 (table default.Table_978b0604ce1c4638bcd51dc8389941d9 [id=f8b12b3bb4904c18be60b567cb05b739], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet e20e126b1c9546e4a194bf9842f444a5 consensus state has no leader
I20240502 08:16:55.793860 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9e89353273c3429b9facd6433d847938"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:55.794436 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e89353273c3429b9facd6433d847938 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.794641 24174 tablet_bootstrap.cc:492] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:55.794768 24174 ts_tablet_manager.cc:1367] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240502 08:16:55.795490 24174 raft_consensus.cc:348] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.795691 24174 raft_consensus.cc:374] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.795754 24174 raft_consensus.cc:729] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.795949 24174 consensus_queue.cc:260] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.796471 24174 ts_tablet_manager.cc:1398] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:55.798889 25158 raft_consensus.cc:482] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.799019 25158 raft_consensus.cc:504] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.799592 25158 leader_election.cc:288] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.800274 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25c089f0c1b14e419f250d03d4eaad12" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:55.802314 23723 leader_election.cc:341] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.804142 23607 catalog_manager.cc:5484] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67). New cstate: current_term: 1 leader_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:55.804551 23607 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 001e3f4f72f64dc69dbd521724912f07 (table default.Table_d7cff78803d44e5aba3b50cd8ff11e92 [id=5c21117955d946418c656330984ae747]). Expected version 3 got 0
I20240502 08:16:55.805104 23607 catalog_manager.cc:5484] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67). New cstate: current_term: 1 leader_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:55.805455 23607 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet a2da8b9d3f374d8a812336057c0ce7d2 (table default.Table_3edd27da6927478281e76361718425a3 [id=76b58ce209a24b9a95b405c1c50de3ca]). Expected version 3 got 0
I20240502 08:16:55.806471 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_45f3dcd544ab41b6ac4dffd8d9d3157e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:55.806627 25531 raft_consensus.cc:482] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.806756 25531 raft_consensus.cc:504] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:16:55.807029 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45f3dcd544ab41b6ac4dffd8d9d3157e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.807492 25531 leader_election.cc:288] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.807962 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b0d4871dd24607854100a4ef762e25" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:55.808458 23723 leader_election.cc:341] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.816161 24180 ts_tablet_manager.cc:1888] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 08:16:55.816074 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ee25478ce0e04c4091efb71e8277beff" } new_table_name: "default.Table_2cececca1c024327bba827ba5487e985" modify_external_catalogs: true
I20240502 08:16:55.816270 24180 log.cc:1196] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/050c16739efb4037a4adc78a2ffd3cba
I20240502 08:16:55.816691 24180 ts_tablet_manager.cc:1909] T 050c16739efb4037a4adc78a2ffd3cba P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:55.816680 23920 raft_consensus.cc:3037] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.821231 25012 log.cc:1196] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/a0224317e65747e1add533d33472f7f3
I20240502 08:16:55.821588 25012 ts_tablet_manager.cc:1909] T a0224317e65747e1add533d33472f7f3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:55.821906 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 853) took 4114 ms. Trace:
I20240502 08:16:55.821919 23886 tablet_service.cc:1514] Processing DeleteTablet for tablet 39cc4ef1174a4358bad777b0471c5f76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:55.821995 25012 rpcz_store.cc:276] 0502 08:16:51.707079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.782096 (+1075017us) service_pool.cc:225] Handling call
0502 08:16:55.573620 (+2791524us) ts_tablet_manager.cc:1120] Deleting tablet a0224317e65747e1add533d33472f7f3
0502 08:16:55.573630 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.590062 (+ 16432us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.592929 (+ 2867us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.821889 (+228960us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2791419}
I20240502 08:16:55.822291 25012 tablet_replica.cc:330] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:55.822331 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 5edadeb53a354c39a33ff68ffbde5ad1 (DEFAULT_TABLE table=default.Table_44816133bdb24a48a436fa62d832ed30 [id=0e6e6a4ac8464ede993f94cfef1f896b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.822744 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5edadeb53a354c39a33ff68ffbde5ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.823525 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0281c8123eab4c48bddc377f5695c6e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:55.823690 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 558a3a4a8383439bb64af1d3eaa64572 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:54 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:55.824030 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 54218d994ab84cc5a5c30161403f7006 (DEFAULT_TABLE table=default.Table_7ff62874389f42d0ba37aba1307ea27b [id=bb8d679cedb944fe845ddb0c64f51e20]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.824337 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54218d994ab84cc5a5c30161403f7006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.825142 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet bb57cb00e884420bacc48f16739eb29c with delete_type TABLET_DATA_DELETED (replaced by tablet 82082b214d834186a185e0c65e9c1b41) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:55.825333 23903 tablet_service.cc:1514] Processing DeleteTablet for tablet dfd897ba97204b6788fd2d99cf2a0303 with delete_type TABLET_DATA_DELETED (replaced by tablet fd3f59a2654a459e8bfd3be29c203e07) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:55.825798 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 18d82e75f2624e6782f55987cea223cb (DEFAULT_TABLE table=default.Table_536141575d2c49fe807d39c1f38e3bb1 [id=01eaf80972e949a6927aeaffa17a7714]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.826153 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18d82e75f2624e6782f55987cea223cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.826069 23919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b0d4871dd24607854100a4ef762e25" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:55.828343 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 36ecf30203634e2792f39c4234cfde1b (DEFAULT_TABLE table=default.Table_2bcd9c267ebf4fd3a2763c589938275f [id=4dd444fe8355478eb81f90a9beeb0c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.828672 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36ecf30203634e2792f39c4234cfde1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.829779 23891 tablet_service.cc:1467] Processing CreateTablet for tablet da91c16f5cc3474ea21dbab9ea1dd5be (DEFAULT_TABLE table=default.Table_37c06b36010d4694830aa2c23281c13c [id=2631ced2961f4d5895a160a157a865a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.830111 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da91c16f5cc3474ea21dbab9ea1dd5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.833618 23888 tablet_service.cc:1514] Processing DeleteTablet for tablet 36dece738afd42b99f3f38d7fd11a0af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:55.833774 23900 tablet_service.cc:1467] Processing CreateTablet for tablet b3b1e3e2e9514f289725f4afc9adb116 (DEFAULT_TABLE table=default.Table_b61b626a7be44b199bf2359795269a93 [id=2781d5032e474ba4b9aab3ade8f53695]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.833806 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f57e3a39d54b7ab372d4b145b7a696" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:55.834069 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b1e3e2e9514f289725f4afc9adb116. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.834199 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet a0224317e65747e1add533d33472f7f3 (table default.Table_5782cc56973c43cfac93b2560c62c688 [id=9a3faa2c394841cfbfab3425282d84d8]) successfully deleted
I20240502 08:16:55.837214 25423 consensus_queue.cc:1035] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:55.837661 23895 tablet_service.cc:1514] Processing DeleteTablet for tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:55.837813 25494 raft_consensus.cc:482] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.837908 25494 raft_consensus.cc:504] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.838476 25494 leader_election.cc:288] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:55.838809 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18d284f71937459db0c0676ce4568175" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:55.838874 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d04312bd81f46ed8c49c42b1af1aec9" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:55.839082 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25c089f0c1b14e419f250d03d4eaad12" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:55.840451 23723 leader_election.cc:341] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.840561 23723 leader_election.cc:302] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:55.840517 25531 consensus_queue.cc:1035] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:55.840747 25296 raft_consensus.cc:2726] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:55.840888 23723 leader_election.cc:341] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.840966 23723 leader_election.cc:302] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:55.841126 25296 raft_consensus.cc:2726] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:55.841256 23723 consensus_peers.cc:485] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:16:55.841359 23723 leader_election.cc:341] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.841437 23723 leader_election.cc:302] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:55.841638 25531 raft_consensus.cc:2726] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:55.841722 23723 leader_election.cc:341] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.841789 23723 leader_election.cc:302] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:55.841943 25531 raft_consensus.cc:2726] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:55.843416 24175 raft_consensus.cc:348] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.843588 24175 raft_consensus.cc:374] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.843641 24175 raft_consensus.cc:729] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.843837 24175 consensus_queue.cc:260] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.844224 24175 ts_tablet_manager.cc:1398] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.233s user 0.001s sys 0.001s
I20240502 08:16:55.844465 24175 tablet_bootstrap.cc:492] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:55.848873 25408 consensus_queue.cc:1035] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:55.852041 25012 raft_consensus.cc:2229] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:55.852207 25012 raft_consensus.cc:2258] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.852717 25012 ts_tablet_manager.cc:1875] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.854337 25158 raft_consensus.cc:482] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.854473 25158 raft_consensus.cc:504] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.855093 25158 leader_election.cc:288] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.855818 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a19bb8705c45c0bfc537d3a6bf8ae2" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:55.856352 23723 leader_election.cc:341] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.857345 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a19bb8705c45c0bfc537d3a6bf8ae2" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:55.857532 23916 raft_consensus.cc:2454] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:55.857910 24175 tablet_bootstrap.cc:654] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.858372 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fd05e15f7383485b9966bdf3ab0ab045" } modify_external_catalogs: true
I20240502 08:16:55.858559 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fd05e15f7383485b9966bdf3ab0ab045" } modify_external_catalogs: true
I20240502 08:16:55.859333 23723 leader_election.cc:302] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:55.859776 25158 raft_consensus.cc:2781] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:55.859858 25158 raft_consensus.cc:482] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:55.859923 25158 raft_consensus.cc:3037] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.860733 24181 ts_tablet_manager.cc:1888] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:16:55.860842 24181 log.cc:1196] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/642744194e22461f8316a78689fd99ff
I20240502 08:16:55.861207 24181 ts_tablet_manager.cc:1909] T 642744194e22461f8316a78689fd99ff P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:55.861991 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 855) took 5912 ms. Trace:
I20240502 08:16:55.862066 24181 rpcz_store.cc:276] 0502 08:16:49.949098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.095512 (+2146414us) service_pool.cc:225] Handling call
0502 08:16:55.773049 (+3677537us) ts_tablet_manager.cc:1120] Deleting tablet 642744194e22461f8316a78689fd99ff
0502 08:16:55.773059 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.845511 (+ 72452us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.860617 (+ 15106us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.861970 (+ 1353us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3561846}
I20240502 08:16:55.862502 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 862) took 5137 ms. Trace:
I20240502 08:16:55.862519 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 642744194e22461f8316a78689fd99ff (table default.Table_8bed20f633e348679b0087687298cc42 [id=18ab2c0716464d82a0e33bd99bd0735d]) successfully deleted
I20240502 08:16:55.862571 24181 rpcz_store.cc:276] 0502 08:16:50.725189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.223639 (+1498450us) service_pool.cc:225] Handling call
0502 08:16:55.862349 (+3638710us) ts_tablet_manager.cc:1120] Deleting tablet c48cea94856f43b6861c3ba360414888
0502 08:16:55.862358 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.862487 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3637104}
I20240502 08:16:55.862617 25158 raft_consensus.cc:504] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.862823 24181 tablet_replica.cc:330] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:55.863018 24181 raft_consensus.cc:2229] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240502 08:16:55.863008 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet c48cea94856f43b6861c3ba360414888 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c48cea94856f43b6861c3ba360414888 already in progress: creating tablet
I20240502 08:16:55.863160 24181 raft_consensus.cc:2258] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.863188 25158 leader_election.cc:288] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.863660 24181 ts_tablet_manager.cc:1875] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.864038 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a19bb8705c45c0bfc537d3a6bf8ae2" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:55.864755 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a19bb8705c45c0bfc537d3a6bf8ae2" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:55.864920 23916 raft_consensus.cc:3037] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.867348 23916 raft_consensus.cc:2454] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
W20240502 08:16:55.867740 23723 leader_election.cc:341] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.868543 25531 raft_consensus.cc:482] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.868680 25531 raft_consensus.cc:504] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.868580 23723 leader_election.cc:302] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:55.869279 25531 leader_election.cc:288] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.869540 25531 raft_consensus.cc:2781] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:55.869664 25531 raft_consensus.cc:686] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:55.869869 25531 consensus_queue.cc:237] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:55.870147 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af6efeebaebc495aba017a922e243c39" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:55.870787 23723 leader_election.cc:341] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.871189 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af6efeebaebc495aba017a922e243c39" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:55.872009 23723 leader_election.cc:341] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:55.872347 23723 leader_election.cc:302] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:55.872705 25296 raft_consensus.cc:2726] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:55.876603 25317 raft_consensus.cc:482] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.876801 25317 raft_consensus.cc:504] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.877705 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c8131846dc41678827b86e87be18d3" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:55.877904 23786 raft_consensus.cc:2454] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:55.878304 25012 ts_tablet_manager.cc:1888] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:55.878300 23993 leader_election.cc:302] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:55.878528 25423 raft_consensus.cc:2781] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:55.878610 25423 raft_consensus.cc:482] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:55.878676 25423 raft_consensus.cc:3037] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.878861 25012 log.cc:1196] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/654ce33d1e3d472eb6ae6a726eb803bb
I20240502 08:16:55.879369 25012 ts_tablet_manager.cc:1909] T 654ce33d1e3d472eb6ae6a726eb803bb P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:55.880792 25423 raft_consensus.cc:504] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.881202 24181 ts_tablet_manager.cc:1888] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:55.881316 24181 log.cc:1196] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/e2126bf2d4ab44628b52481dc2ec3a99
I20240502 08:16:55.881373 25423 leader_election.cc:288] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.881666 24181 ts_tablet_manager.cc:1909] T e2126bf2d4ab44628b52481dc2ec3a99 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:55.881660 25494 raft_consensus.cc:482] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.881775 25494 raft_consensus.cc:504] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.882038 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c8131846dc41678827b86e87be18d3" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:55.882193 23786 raft_consensus.cc:3037] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.882421 25494 leader_election.cc:288] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:55.882432 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 863) took 5157 ms. Trace:
I20240502 08:16:55.883052 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00f29fb76b44d6ea4f76963d2d0751c" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:55.883502 24181 rpcz_store.cc:276] 0502 08:16:50.725304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.225278 (+1499974us) service_pool.cc:225] Handling call
0502 08:16:55.862783 (+3637505us) ts_tablet_manager.cc:1120] Deleting tablet e2126bf2d4ab44628b52481dc2ec3a99
0502 08:16:55.862792 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.878205 (+ 15413us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.881129 (+ 2924us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.882414 (+ 1285us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3616940}
I20240502 08:16:55.883817 24181 tablet_replica.cc:330] T f5ccd43e78574f3288b199fa14bcc0e8 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:55.883967 24181 raft_consensus.cc:2229] T f5ccd43e78574f3288b199fa14bcc0e8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:55.884088 24181 raft_consensus.cc:2258] T f5ccd43e78574f3288b199fa14bcc0e8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.884110 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet e2126bf2d4ab44628b52481dc2ec3a99 (table default.Table_5b4e617d1c8949daa34aa358c1528f88 [id=92cbf164660d48bd974bf8a26b8dfb0d]) successfully deleted
I20240502 08:16:55.884395 23786 raft_consensus.cc:2454] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:55.884562 24181 ts_tablet_manager.cc:1875] T f5ccd43e78574f3288b199fa14bcc0e8 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.884793 24173 tablet_bootstrap.cc:492] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:55.885017 24173 ts_tablet_manager.cc:1367] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.186s user 0.022s sys 0.010s
I20240502 08:16:55.885129 23993 leader_election.cc:302] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:55.885452 25413 raft_consensus.cc:482] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.885614 25413 raft_consensus.cc:504] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.886096 25317 leader_election.cc:288] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.886229 25413 leader_election.cc:288] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.886219 25317 raft_consensus.cc:2781] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:55.886489 25317 raft_consensus.cc:686] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:55.886714 25317 consensus_queue.cc:237] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.886776 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00f29fb76b44d6ea4f76963d2d0751c" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:55.887020 24051 raft_consensus.cc:2379] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:16:55.887408 23861 leader_election.cc:302] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:55.887923 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f83bcf640f465bb95d87a991cc5ff6" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:55.888118 23786 raft_consensus.cc:2379] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 149ce744feab49b5aa5bcaa14bd6aba5 in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:16:55.888032 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f83bcf640f465bb95d87a991cc5ff6" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:55.888247 23922 raft_consensus.cc:2379] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 149ce744feab49b5aa5bcaa14bd6aba5 in current term 1: Already voted for candidate 5b89a1f0485a44b7ac1710e2369a809b in this term.
I20240502 08:16:55.888484 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c8131846dc41678827b86e87be18d3" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:55.888617 23916 raft_consensus.cc:2454] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:55.888814 25494 raft_consensus.cc:3037] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.888850 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c8131846dc41678827b86e87be18d3" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:55.889233 23920 raft_consensus.cc:3037] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.891284 25494 raft_consensus.cc:2726] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:55.891750 23996 leader_election.cc:302] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:55.892616 25423 raft_consensus.cc:3037] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.893031 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6103c4da9979479ba3fde1ea61b4d7d7" } new_table_name: "default.Table_907037a5ad7346a195559d77b1d691d1" modify_external_catalogs: true
I20240502 08:16:55.893245 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e46226aed42b4c30869f1eeb5016c9e3" } modify_external_catalogs: true
I20240502 08:16:55.893404 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e46226aed42b4c30869f1eeb5016c9e3" } modify_external_catalogs: true
I20240502 08:16:55.894783 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_45f3dcd544ab41b6ac4dffd8d9d3157e" } modify_external_catalogs: true
I20240502 08:16:55.894961 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_45f3dcd544ab41b6ac4dffd8d9d3157e" } modify_external_catalogs: true
I20240502 08:16:55.895496 25423 raft_consensus.cc:2726] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:55.896420 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2e5b16fa5b3b4be3af4d6843606d91f6" } modify_external_catalogs: true
I20240502 08:16:55.896595 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2e5b16fa5b3b4be3af4d6843606d91f6" } modify_external_catalogs: true
I20240502 08:16:55.897967 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_81ee0423dbbe4ff899c42249227cea34" } new_table_name: "default.Table_876663d4e8ac4320920b2d42fe658578" modify_external_catalogs: true
I20240502 08:16:55.898905 24173 raft_consensus.cc:348] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.899081 24173 raft_consensus.cc:374] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.899143 24173 raft_consensus.cc:729] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.900034 24173 consensus_queue.cc:260] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.900380 24181 ts_tablet_manager.cc:1888] T f5ccd43e78574f3288b199fa14bcc0e8 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:55.900487 24181 log.cc:1196] T f5ccd43e78574f3288b199fa14bcc0e8 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/f5ccd43e78574f3288b199fa14bcc0e8
I20240502 08:16:55.900590 24173 ts_tablet_manager.cc:1398] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 08:16:55.900804 24107 heartbeater.cc:347] Connected to a master server at 127.22.225.125:36189
I20240502 08:16:55.900864 24173 tablet_bootstrap.cc:492] T 9761df0bbdee44af90135bffb8b5828f P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:55.903992 24174 tablet_bootstrap.cc:492] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:55.905247 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 942) took 3373 ms. Trace:
I20240502 08:16:55.905355 24173 tablet_bootstrap.cc:654] T 9761df0bbdee44af90135bffb8b5828f P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.905335 24180 rpcz_store.cc:276] 0502 08:16:52.531349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.531410 (+ 61us) service_pool.cc:225] Handling call
0502 08:16:55.536519 (+3005109us) ts_tablet_manager.cc:1120] Deleting tablet 050c16739efb4037a4adc78a2ffd3cba
0502 08:16:55.536530 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.611401 (+ 74871us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.816048 (+204647us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.905224 (+ 89176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3004913}
I20240502 08:16:55.905510 24174 tablet_bootstrap.cc:654] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.905881 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 050c16739efb4037a4adc78a2ffd3cba (table default.Table_3a66619663c642f5be35fd8249184e1a [id=ba62a673079d424992daa0f6b0c5d6b9]) successfully deleted
I20240502 08:16:55.907294 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 951) took 2887 ms. Trace:
I20240502 08:16:55.907377 24180 rpcz_store.cc:276] 0502 08:16:53.019992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.020050 (+ 58us) service_pool.cc:225] Handling call
0502 08:16:55.907141 (+2887091us) ts_tablet_manager.cc:1120] Deleting tablet 8075b03643a14a01ab75921fc3d3dace
0502 08:16:55.907168 (+ 27us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.907275 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2886873}
I20240502 08:16:55.907637 24180 tablet_replica.cc:330] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:55.907822 24180 raft_consensus.cc:2229] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:55.908155 24180 raft_consensus.cc:2258] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.908712 24180 ts_tablet_manager.cc:1875] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:55.909902 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 8075b03643a14a01ab75921fc3d3dace because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8075b03643a14a01ab75921fc3d3dace already in progress: creating tablet
I20240502 08:16:55.917884 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 854) took 4210 ms. Trace:
I20240502 08:16:55.917989 25012 rpcz_store.cc:276] 0502 08:16:51.707201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.782205 (+1075004us) service_pool.cc:225] Handling call
0502 08:16:55.822246 (+3040041us) ts_tablet_manager.cc:1120] Deleting tablet 654ce33d1e3d472eb6ae6a726eb803bb
0502 08:16:55.822254 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.855477 (+ 33223us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.878186 (+ 22709us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.917853 (+ 39667us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29542,"tablet-delete.queue_time_us":3039942}
I20240502 08:16:55.918401 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 654ce33d1e3d472eb6ae6a726eb803bb (table default.Table_6f4d25ef5b90455ca233af7b70b9073a [id=2fe4c270673b4204aaccc289d3b9d929]) successfully deleted
I20240502 08:16:55.918502 25012 tablet_replica.cc:330] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:55.918673 25012 raft_consensus.cc:2229] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:55.918819 25012 raft_consensus.cc:2258] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.919589 25012 ts_tablet_manager.cc:1875] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.919881 24180 ts_tablet_manager.cc:1888] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:16:55.921151 24180 log.cc:1196] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/77c49da1e0f0435ba241a6a66526e790
I20240502 08:16:55.921654 24180 ts_tablet_manager.cc:1909] T 77c49da1e0f0435ba241a6a66526e790 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:55.922003 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 952) took 2901 ms. Trace:
I20240502 08:16:55.900818 24181 ts_tablet_manager.cc:1909] T f5ccd43e78574f3288b199fa14bcc0e8 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:55.922250 24180 rpcz_store.cc:276] 0502 08:16:53.020996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.021042 (+ 46us) service_pool.cc:225] Handling call
0502 08:16:55.907596 (+2886554us) ts_tablet_manager.cc:1120] Deleting tablet 77c49da1e0f0435ba241a6a66526e790
0502 08:16:55.907604 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.917076 (+ 9472us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.919801 (+ 2725us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.921984 (+ 2183us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2886398}
I20240502 08:16:55.923733 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 77c49da1e0f0435ba241a6a66526e790 (table default.Table_8a9cdf1f53b0494694035cd549988bf0 [id=0de18f9c7e194e698b4f37603bfd4780]) successfully deleted
I20240502 08:16:55.924199 24174 tablet_bootstrap.cc:492] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:55.924343 24174 ts_tablet_manager.cc:1367] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.005s
I20240502 08:16:55.925360 24174 raft_consensus.cc:348] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.925693 24174 raft_consensus.cc:374] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.925758 24174 raft_consensus.cc:729] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.926043 24174 consensus_queue.cc:260] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.927253 23920 raft_consensus.cc:2454] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:55.929136 24180 tablet_replica.cc:330] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:55.924183 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 866) took 4059 ms. Trace:
I20240502 08:16:55.934633 24180 raft_consensus.cc:2229] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:55.934813 24181 rpcz_store.cc:276] 0502 08:16:51.864809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.245886 (+381077us) service_pool.cc:225] Handling call
0502 08:16:55.883775 (+3637889us) ts_tablet_manager.cc:1120] Deleting tablet f5ccd43e78574f3288b199fa14bcc0e8
0502 08:16:55.883783 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.895464 (+ 11681us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.900182 (+ 4718us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.924161 (+ 23979us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3632035}
I20240502 08:16:55.935302 24181 tablet_replica.cc:330] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:55.935546 24180 raft_consensus.cc:2258] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.935818 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet f5ccd43e78574f3288b199fa14bcc0e8 (table default.Table_95df2e76edc746a08164b4ff25809909 [id=d5a04f8005714d15a3710236c85b1317]) successfully deleted
I20240502 08:16:55.936172 24180 ts_tablet_manager.cc:1875] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.929427 24174 ts_tablet_manager.cc:1398] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:16:55.939776 24174 tablet_bootstrap.cc:492] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:55.902971 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d04312bd81f46ed8c49c42b1af1aec9" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:55.940608 23858 leader_election.cc:302] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:55.941035 25494 raft_consensus.cc:2726] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:55.941392 24174 tablet_bootstrap.cc:654] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.942397 24181 raft_consensus.cc:2229] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:55.942734 24181 raft_consensus.cc:2258] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:55.943253 24181 ts_tablet_manager.cc:1875] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:55.945518 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 79a36b64095f445ba21a6304b887a112 (DEFAULT_TABLE table=default.Table_4e5d657dc235492dad016da586febee0 [id=78aaf5a0106b48bd8ad2b395f5c01fc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.946815 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 5590e9d53f5641d38032efccaea3e31d (DEFAULT_TABLE table=default.Table_2d4fafaec4f240edae01ff9a29566924 [id=3caea5f1b1324587a58a879e6ac6768f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.947721 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 5590e9d53f5641d38032efccaea3e31d (DEFAULT_TABLE table=default.Table_2d4fafaec4f240edae01ff9a29566924 [id=3caea5f1b1324587a58a879e6ac6768f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.947788 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 79a36b64095f445ba21a6304b887a112 (DEFAULT_TABLE table=default.Table_4e5d657dc235492dad016da586febee0 [id=78aaf5a0106b48bd8ad2b395f5c01fc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.948084 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5590e9d53f5641d38032efccaea3e31d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.948434 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79a36b64095f445ba21a6304b887a112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.948820 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 858c022bdb5449d980acc8b0eeec9ff0 (DEFAULT_TABLE table=default.Table_c26660ee28674f03916ea5aecabf0a21 [id=d5a4cd7237b645c18da74b82f95a7b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.952378 24051 raft_consensus.cc:3037] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.954766 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 16a1c5276f014410b740c87b9971fb48 (DEFAULT_TABLE table=default.Table_e46226aed42b4c30869f1eeb5016c9e3 [id=adb1d5f1f35342c1b5fd4545844d80b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.955550 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 5590e9d53f5641d38032efccaea3e31d (DEFAULT_TABLE table=default.Table_2d4fafaec4f240edae01ff9a29566924 [id=3caea5f1b1324587a58a879e6ac6768f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.956050 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5590e9d53f5641d38032efccaea3e31d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.957545 25012 ts_tablet_manager.cc:1888] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:55.957820 25012 log.cc:1196] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/f797d3c6bdc4493398954fbbf9943971
I20240502 08:16:55.958624 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 79a36b64095f445ba21a6304b887a112 (DEFAULT_TABLE table=default.Table_4e5d657dc235492dad016da586febee0 [id=78aaf5a0106b48bd8ad2b395f5c01fc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.959056 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79a36b64095f445ba21a6304b887a112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.959139 25012 ts_tablet_manager.cc:1909] T f797d3c6bdc4493398954fbbf9943971 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:55.966769 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 858c022bdb5449d980acc8b0eeec9ff0 (DEFAULT_TABLE table=default.Table_c26660ee28674f03916ea5aecabf0a21 [id=d5a4cd7237b645c18da74b82f95a7b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.967291 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858c022bdb5449d980acc8b0eeec9ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.972154 25494 consensus_queue.cc:1035] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:55.974469 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 16a1c5276f014410b740c87b9971fb48 (DEFAULT_TABLE table=default.Table_e46226aed42b4c30869f1eeb5016c9e3 [id=adb1d5f1f35342c1b5fd4545844d80b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.974826 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16a1c5276f014410b740c87b9971fb48. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:55.986109 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90d44be09f284f3d9042038c92f3b28d consensus state has no leader
I20240502 08:16:55.986601 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3c1db2ea8e3d49efa8439ddf3a30734c" } modify_external_catalogs: true
I20240502 08:16:55.986814 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3c1db2ea8e3d49efa8439ddf3a30734c" } modify_external_catalogs: true
W20240502 08:16:55.987370 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6f15b4b87a04707a3b4f37538e3b64a consensus state has no leader
I20240502 08:16:55.987740 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_eb6b182c8aff4600af282bf9dd035542"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:55.988260 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb6b182c8aff4600af282bf9dd035542 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.989408 23609 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 350ab6bedf0f4f9fb8d72da1beb80891
I20240502 08:16:55.989797 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_191144ffbcfe41bfaef1158d4a866ec6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:55.990271 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_191144ffbcfe41bfaef1158d4a866ec6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.991300 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 506644a1ead54a1faeaa8583ef96b449
I20240502 08:16:55.992014 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1345a4fd2d3e46deaecb136b548f5b3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:55.992408 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1345a4fd2d3e46deaecb136b548f5b3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.992362 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 506644a1ead54a1faeaa8583ef96b449 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:55.993263 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 858c022bdb5449d980acc8b0eeec9ff0 (DEFAULT_TABLE table=default.Table_c26660ee28674f03916ea5aecabf0a21 [id=d5a4cd7237b645c18da74b82f95a7b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.993757 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858c022bdb5449d980acc8b0eeec9ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.994016 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet fc87f91093b94b209caa6f89691bcde3
I20240502 08:16:55.994674 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet fc87f91093b94b209caa6f89691bcde3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:55.994629 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d2ed4128fb3845cf9d0372ba1bb5bd61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:55.995043 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2ed4128fb3845cf9d0372ba1bb5bd61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:55.997229 25423 raft_consensus.cc:482] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:55.997356 25423 raft_consensus.cc:504] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:55.998018 25423 leader_election.cc:288] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:55.998685 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbef606ce7ac499d9bd84b49d804621c" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:55.998878 23788 raft_consensus.cc:2454] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:55.999037 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbef606ce7ac499d9bd84b49d804621c" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:55.999434 23993 leader_election.cc:302] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:55.999814 25405 raft_consensus.cc:2781] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:55.999889 25405 raft_consensus.cc:482] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:55.999948 25405 raft_consensus.cc:3037] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:56.001202 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79a36b64095f445ba21a6304b887a112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.002144 25405 raft_consensus.cc:504] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.002625 25405 leader_election.cc:288] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:56.003741 23996 leader_election.cc:341] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.004415 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbef606ce7ac499d9bd84b49d804621c" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:56.004545 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbef606ce7ac499d9bd84b49d804621c" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:56.004702 23788 raft_consensus.cc:3037] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:56.004948 23996 leader_election.cc:341] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.006234 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet fc87f91093b94b209caa6f89691bcde3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:56.006843 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16a1c5276f014410b740c87b9971fb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.007025 23788 raft_consensus.cc:2454] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:56.007752 23895 tablet_service.cc:1514] Processing DeleteTablet for tablet fc87f91093b94b209caa6f89691bcde3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:56.008217 23993 leader_election.cc:302] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:56.008283 24175 tablet_bootstrap.cc:492] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:56.008414 24175 ts_tablet_manager.cc:1367] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.164s user 0.001s sys 0.011s
I20240502 08:16:56.008587 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet 506644a1ead54a1faeaa8583ef96b449 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:56.008950 23604 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet af6efeebaebc495aba017a922e243c39 (table default.Table_82f82cd6e702420283c288c10f905d90 [id=f293b22818344aea9eb6232141432183]). Expected version 1 got 0
I20240502 08:16:56.010167 24175 raft_consensus.cc:348] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.010571 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_cccac0aeb15d4f568676f29b4d73a49f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:56.011083 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cccac0aeb15d4f568676f29b4d73a49f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:56.011880 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858c022bdb5449d980acc8b0eeec9ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.013546 23605 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 7ed919909289410e867881baab841f53 (table default.Table_2edad641920b44ce9a4d5b7503f0279a [id=ee3912407a934444ad2c62f7b83d93ac]). Expected version 2 got 0
I20240502 08:16:56.013969 23605 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 001e3f4f72f64dc69dbd521724912f07 (table default.Table_d7cff78803d44e5aba3b50cd8ff11e92 [id=5c21117955d946418c656330984ae747]). Expected version 3 got 0
I20240502 08:16:56.014192 23605 catalog_manager.cc:5484] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66). New cstate: current_term: 1 leader_uuid: "5b89a1f0485a44b7ac1710e2369a809b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:56.015189 25423 raft_consensus.cc:2781] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:56.015309 25423 raft_consensus.cc:686] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:56.015525 25423 consensus_queue.cc:237] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.017336 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:56.017585 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5590e9d53f5641d38032efccaea3e31d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:56.018044 25184 log.cc:924] Time spent T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1: Append to log took a long time: real 0.100s user 0.000s sys 0.003s
I20240502 08:16:56.020874 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c8131846dc41678827b86e87be18d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:56.021090 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e7432f1df94431a83cf68356e3e78a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:56.021095 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet fbef606ce7ac499d9bd84b49d804621c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:56.021327 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet c00f29fb76b44d6ea4f76963d2d0751c with delete_type TABLET_DATA_DELETED (replaced by tablet a17fb4f2f02e43dc9f7cfdb7c73f7774) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:56.021116 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 128c88af54d24ffd895e9de19cdb4c8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:56.023319 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet 506644a1ead54a1faeaa8583ef96b449 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:56.025360 25531 raft_consensus.cc:482] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.025616 25531 raft_consensus.cc:504] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.027029 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df08f0d3af4c4c7e8abce656714b1b7f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:56.027707 23723 leader_election.cc:341] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.030800 24180 ts_tablet_manager.cc:1888] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:56.031035 24180 log.cc:1196] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/07344b78894c42809391d0fe2a9cd8f7
I20240502 08:16:56.033345 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df08f0d3af4c4c7e8abce656714b1b7f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:56.035714 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 16a1c5276f014410b740c87b9971fb48 (DEFAULT_TABLE table=default.Table_e46226aed42b4c30869f1eeb5016c9e3 [id=adb1d5f1f35342c1b5fd4545844d80b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.036221 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16a1c5276f014410b740c87b9971fb48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.043931 24173 tablet_bootstrap.cc:492] T 9761df0bbdee44af90135bffb8b5828f P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:56.044067 24173 ts_tablet_manager.cc:1367] T 9761df0bbdee44af90135bffb8b5828f P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.143s user 0.025s sys 0.061s
I20240502 08:16:56.044695 24173 raft_consensus.cc:348] T 9761df0bbdee44af90135bffb8b5828f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.044883 24173 raft_consensus.cc:374] T 9761df0bbdee44af90135bffb8b5828f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.044925 24173 raft_consensus.cc:729] T 9761df0bbdee44af90135bffb8b5828f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.045116 24173 consensus_queue.cc:260] T 9761df0bbdee44af90135bffb8b5828f P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.045553 24173 ts_tablet_manager.cc:1398] T 9761df0bbdee44af90135bffb8b5828f P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:16:56.045859 24173 tablet_bootstrap.cc:492] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:56.047132 24173 tablet_bootstrap.cc:654] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.049149 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 16a1c5276f014410b740c87b9971fb48
I20240502 08:16:56.050002 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_63fdac96c4d7418a9c60737466b8b0a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:56.050575 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63fdac96c4d7418a9c60737466b8b0a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:56.052196 24181 ts_tablet_manager.cc:1888] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.052201 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet 16a1c5276f014410b740c87b9971fb48 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:56.052292 24181 log.cc:1196] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/24a4104f4cb64889ad66810788ad6b29
I20240502 08:16:56.052467 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 16a1c5276f014410b740c87b9971fb48 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:56.052639 24181 ts_tablet_manager.cc:1909] T 24a4104f4cb64889ad66810788ad6b29 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:56.052942 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 868) took 4109 ms. Trace:
I20240502 08:16:56.053017 24181 rpcz_store.cc:276] 0502 08:16:51.943496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.256250 (+312754us) service_pool.cc:225] Handling call
0502 08:16:55.935236 (+3678986us) ts_tablet_manager.cc:1120] Deleting tablet 24a4104f4cb64889ad66810788ad6b29
0502 08:16:55.935248 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.042916 (+107668us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.052109 (+ 9193us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.052924 (+ 815us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3678798}
I20240502 08:16:56.053426 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 24a4104f4cb64889ad66810788ad6b29 (table default.Table_307041e730f248a5bd4d265f08aad5e4 [id=f2316137abe74f5c8e79faf0c4638540]) successfully deleted
I20240502 08:16:56.031739 24174 tablet_bootstrap.cc:492] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:56.033079 25531 leader_election.cc:288] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:56.053887 24174 ts_tablet_manager.cc:1367] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.114s user 0.001s sys 0.002s
I20240502 08:16:56.054226 24180 ts_tablet_manager.cc:1909] T 07344b78894c42809391d0fe2a9cd8f7 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.050010 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet 16a1c5276f014410b740c87b9971fb48 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:56.056876 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 6e8f8d9101c440a5ab99fb6d3546fa41 (DEFAULT_TABLE table=default.Table_1060ebe6de4a49499e58dfa9a9ab5e20 [id=5875a30b7c60491b9afd3ad0a19c5fda]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.054772 24174 raft_consensus.cc:348] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.057186 24174 raft_consensus.cc:374] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.057258 24174 raft_consensus.cc:729] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
W20240502 08:16:56.033890 23723 leader_election.cc:341] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.057646 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 869) took 4062 ms. Trace:
I20240502 08:16:56.057623 23723 leader_election.cc:302] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:56.057713 24181 rpcz_store.cc:276] 0502 08:16:51.995391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.256473 (+261082us) service_pool.cc:225] Handling call
0502 08:16:56.057505 (+3801032us) ts_tablet_manager.cc:1120] Deleting tablet f685cbe705b149e69f9536e270a74a42
0502 08:16:56.057514 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.057630 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3800919}
W20240502 08:16:56.058092 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet f685cbe705b149e69f9536e270a74a42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f685cbe705b149e69f9536e270a74a42 already in progress: creating tablet
I20240502 08:16:56.059304 25536 consensus_queue.cc:1035] T 2f1d946bd6de48f1a8d8fe5f179d22b6 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:56.060833 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 870) took 4006 ms. Trace:
I20240502 08:16:56.060830 25158 raft_consensus.cc:2726] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:56.060907 24181 rpcz_store.cc:276] 0502 08:16:52.054079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.256600 (+202521us) service_pool.cc:225] Handling call
0502 08:16:56.060729 (+3804129us) ts_tablet_manager.cc:1120] Deleting tablet fe5e08ad6af2442bb0a8fcc80a99b532
0502 08:16:56.060737 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.060818 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3804022}
W20240502 08:16:56.061254 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet fe5e08ad6af2442bb0a8fcc80a99b532 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe5e08ad6af2442bb0a8fcc80a99b532 already in progress: creating tablet
I20240502 08:16:56.057502 24174 consensus_queue.cc:260] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.062103 24174 ts_tablet_manager.cc:1398] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:16:56.064468 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 871) took 4002 ms. Trace:
I20240502 08:16:56.064551 24181 rpcz_store.cc:276] 0502 08:16:52.062387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.256716 (+194329us) service_pool.cc:225] Handling call
0502 08:16:56.062959 (+3806243us) ts_tablet_manager.cc:1120] Deleting tablet 303a0a3748ee422faab856e1c4bff66d
0502 08:16:56.064358 (+ 1399us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.064453 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3806142}
W20240502 08:16:56.064951 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 303a0a3748ee422faab856e1c4bff66d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 303a0a3748ee422faab856e1c4bff66d already in progress: creating tablet
I20240502 08:16:56.065337 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 872) took 4002 ms. Trace:
I20240502 08:16:56.065461 24181 rpcz_store.cc:276] 0502 08:16:52.062521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.256832 (+194311us) service_pool.cc:225] Handling call
0502 08:16:56.065241 (+3808409us) ts_tablet_manager.cc:1120] Deleting tablet 29f6c3cea4b14364b6d073a92171119d
0502 08:16:56.065249 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.065321 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3808304}
W20240502 08:16:56.065979 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 29f6c3cea4b14364b6d073a92171119d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29f6c3cea4b14364b6d073a92171119d already in progress: creating tablet
I20240502 08:16:56.066305 24181 tablet_replica.cc:330] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:56.066466 24174 tablet_bootstrap.cc:492] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:56.066725 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 953) took 3035 ms. Trace:
I20240502 08:16:56.066891 24180 rpcz_store.cc:276] 0502 08:16:53.030871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.030927 (+ 56us) service_pool.cc:225] Handling call
0502 08:16:55.929082 (+2898155us) ts_tablet_manager.cc:1120] Deleting tablet 07344b78894c42809391d0fe2a9cd8f7
0502 08:16:55.929095 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.939152 (+ 10057us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.030712 (+ 91560us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.066706 (+ 35994us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2897975}
I20240502 08:16:56.067348 24180 tablet_replica.cc:330] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:56.067516 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 07344b78894c42809391d0fe2a9cd8f7 (table default.Table_4c4a02d4a6eb4c87971a03d670f5fdaf [id=6b6ffe47b9534a618f0912d78e7bb92a]) successfully deleted
I20240502 08:16:56.067582 24180 raft_consensus.cc:2229] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.067730 24180 raft_consensus.cc:2258] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.067796 24181 raft_consensus.cc:2229] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:56.068100 24174 tablet_bootstrap.cc:654] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.068117 24181 raft_consensus.cc:2258] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.068236 24180 ts_tablet_manager.cc:1875] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.068653 24181 ts_tablet_manager.cc:1875] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.069706 25536 raft_consensus.cc:482] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.069829 25536 raft_consensus.cc:504] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.070426 25536 leader_election.cc:288] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:56.070338 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 6e8f8d9101c440a5ab99fb6d3546fa41 (DEFAULT_TABLE table=default.Table_1060ebe6de4a49499e58dfa9a9ab5e20 [id=5875a30b7c60491b9afd3ad0a19c5fda]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.071033 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6746c25111d04f20a6f30ce0838d71f5" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:56.071436 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e8f8d9101c440a5ab99fb6d3546fa41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.072818 24173 tablet_bootstrap.cc:492] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:56.072943 24173 ts_tablet_manager.cc:1367] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.010s
I20240502 08:16:56.072791 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6746c25111d04f20a6f30ce0838d71f5" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:56.073737 23858 leader_election.cc:302] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:56.074226 25536 raft_consensus.cc:2726] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:56.075580 24180 ts_tablet_manager.cc:1888] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.075675 24180 log.cc:1196] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/1cd8338500f54604abcd6d42702c2eca
I20240502 08:16:56.076015 24173 raft_consensus.cc:348] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.076205 24173 raft_consensus.cc:374] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.076254 24173 raft_consensus.cc:729] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.076462 24173 consensus_queue.cc:260] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.076973 24173 ts_tablet_manager.cc:1398] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:16:56.077229 24173 tablet_bootstrap.cc:492] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:56.084023 24037 tablet_service.cc:1467] Processing CreateTablet for tablet f13bd33dd7c44ac19fa4e0bba3083483 (DEFAULT_TABLE table=default.Table_bcfbfe93e2a14297828b5837679bfd8d [id=5155802598fa400fb72b3d9d37c359e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.084352 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13bd33dd7c44ac19fa4e0bba3083483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.084589 24173 tablet_bootstrap.cc:654] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.088198 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 54310d86b1f9462a8f87728711ebf6fa (DEFAULT_TABLE table=default.Table_93b89623977c4ede951f10ea31e1762b [id=f978ad4c43364d5cbdad30b9b5a687a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.088642 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 5027fa2c80b544b8b1e2a4694bec7454 (DEFAULT_TABLE table=default.Table_963bbb5ed3184e6a95549deb63ffb35a [id=d330f8cbe8ce4b089d6f8a9435ed07cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.088989 24036 tablet_service.cc:1467] Processing CreateTablet for tablet f2a1a46a4e6f43669c72d5807e60c4dc (DEFAULT_TABLE table=default.Table_b59e0701b4d042f0b4d6277ade5ab169 [id=8d0d796950ba478b8058b8bb714ae329]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.089335 24028 tablet_service.cc:1467] Processing CreateTablet for tablet d6cdf43567be47c1b9d876017b31d157 (DEFAULT_TABLE table=default.Table_d40724ccadc148cd91ea36278515b65c [id=f8b9072ab9a2434a90efae6588087d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.089704 24022 tablet_service.cc:1467] Processing CreateTablet for tablet 40bbd76c59c649619abbcea1d36758ca (DEFAULT_TABLE table=default.Table_ec2cb5eeb2c34373b60522a9725b77e0 [id=214f79d839814815808dc415892e3472]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.090056 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 7f7ea2d3bc8a494ebdefa6de8ca389e0 (DEFAULT_TABLE table=default.Table_bbf9d7694fe942de8aa8bd4ed61336a5 [id=f1d6eb661b0143ee82e797cde61391c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.090368 24031 tablet_service.cc:1467] Processing CreateTablet for tablet e4e2309db5ca4c7cbafcdf7782121d23 (DEFAULT_TABLE table=default.Table_8e550c4078d04f7db883f8ed5e051554 [id=f9d35abc5f84451da41552124e9fd10b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.095160 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54310d86b1f9462a8f87728711ebf6fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.099512 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e2309db5ca4c7cbafcdf7782121d23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.101017 25405 consensus_queue.cc:1035] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:56.101476 24181 ts_tablet_manager.cc:1888] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.101568 24181 log.cc:1196] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/654ce33d1e3d472eb6ae6a726eb803bb
I20240502 08:16:56.101891 24181 ts_tablet_manager.cc:1909] T 654ce33d1e3d472eb6ae6a726eb803bb P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:56.102365 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6cdf43567be47c1b9d876017b31d157. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.102607 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 867) took 4168 ms. Trace:
I20240502 08:16:56.102684 24181 rpcz_store.cc:276] 0502 08:16:51.934440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.251794 (+317354us) service_pool.cc:225] Handling call
0502 08:16:56.066262 (+3814468us) ts_tablet_manager.cc:1120] Deleting tablet 654ce33d1e3d472eb6ae6a726eb803bb
0502 08:16:56.066270 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.071375 (+ 5105us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.098447 (+ 27072us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.102588 (+ 4141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3803524}
I20240502 08:16:56.103035 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 882) took 3519 ms. Trace:
I20240502 08:16:56.103096 24181 rpcz_store.cc:276] 0502 08:16:52.583173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.583216 (+ 43us) service_pool.cc:225] Handling call
0502 08:16:56.102938 (+3519722us) ts_tablet_manager.cc:1120] Deleting tablet 9f3c00e5393c47df924bcb5dcf2a0f19
0502 08:16:56.102946 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.103022 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3519559}
I20240502 08:16:56.103334 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 883) took 3520 ms. Trace:
I20240502 08:16:56.103394 24181 rpcz_store.cc:276] 0502 08:16:52.583254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.583410 (+ 156us) service_pool.cc:225] Handling call
0502 08:16:56.103260 (+3519850us) ts_tablet_manager.cc:1120] Deleting tablet d2bb4dba23ac4b4eb53b3f2d75d0442e
0502 08:16:56.103266 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.103324 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3519760}
I20240502 08:16:56.105835 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f7ea2d3bc8a494ebdefa6de8ca389e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.108896 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5027fa2c80b544b8b1e2a4694bec7454. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.078191 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 6e8f8d9101c440a5ab99fb6d3546fa41 (DEFAULT_TABLE table=default.Table_1060ebe6de4a49499e58dfa9a9ab5e20 [id=5875a30b7c60491b9afd3ad0a19c5fda]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.078393 24174 tablet_bootstrap.cc:492] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:56.110040 24174 ts_tablet_manager.cc:1367] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.000s
I20240502 08:16:56.110164 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e8f8d9101c440a5ab99fb6d3546fa41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.110203 24180 ts_tablet_manager.cc:1909] T 1cd8338500f54604abcd6d42702c2eca P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.110862 24174 raft_consensus.cc:348] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.111021 24174 raft_consensus.cc:374] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.111097 24174 raft_consensus.cc:729] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.111279 24174 consensus_queue.cc:260] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.111781 24174 ts_tablet_manager.cc:1398] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:56.112044 24174 tablet_bootstrap.cc:492] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:56.113477 24174 tablet_bootstrap.cc:654] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.078851 23751 tablet_service.cc:1467] Processing CreateTablet for tablet f13bd33dd7c44ac19fa4e0bba3083483 (DEFAULT_TABLE table=default.Table_bcfbfe93e2a14297828b5837679bfd8d [id=5155802598fa400fb72b3d9d37c359e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.114342 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13bd33dd7c44ac19fa4e0bba3083483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.114387 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e8f8d9101c440a5ab99fb6d3546fa41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.115453 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 954) took 3079 ms. Trace:
I20240502 08:16:56.115696 24180 rpcz_store.cc:276] 0502 08:16:53.036208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.036707 (+ 499us) service_pool.cc:225] Handling call
0502 08:16:56.067309 (+3030602us) ts_tablet_manager.cc:1120] Deleting tablet 1cd8338500f54604abcd6d42702c2eca
0502 08:16:56.067317 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.072908 (+ 5591us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.075494 (+ 2586us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.115409 (+ 39915us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3030449}
I20240502 08:16:56.116428 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 956) took 3062 ms. Trace:
I20240502 08:16:56.116601 24180 rpcz_store.cc:276] 0502 08:16:53.053890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.053937 (+ 47us) service_pool.cc:225] Handling call
0502 08:16:56.116124 (+3062187us) ts_tablet_manager.cc:1120] Deleting tablet 2ebf55c8024c483f9ef0afe6945a02ff
0502 08:16:56.116162 (+ 38us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.116318 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3062006}
I20240502 08:16:56.117213 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 957) took 3044 ms. Trace:
I20240502 08:16:56.079759 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 54310d86b1f9462a8f87728711ebf6fa (DEFAULT_TABLE table=default.Table_93b89623977c4ede951f10ea31e1762b [id=f978ad4c43364d5cbdad30b9b5a687a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.117421 24180 rpcz_store.cc:276] 0502 08:16:53.072911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.072962 (+ 51us) service_pool.cc:225] Handling call
0502 08:16:56.116999 (+3044037us) ts_tablet_manager.cc:1120] Deleting tablet 33bbebbca7464ea6ad59b81d7b7c560d
0502 08:16:56.117007 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.117144 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3043827}
I20240502 08:16:56.117787 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54310d86b1f9462a8f87728711ebf6fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.117973 24180 tablet_replica.cc:330] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:56.118319 24180 raft_consensus.cc:2229] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.118563 24180 raft_consensus.cc:2258] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.119913 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 855) took 4412 ms. Trace:
I20240502 08:16:56.119987 25012 rpcz_store.cc:276] 0502 08:16:51.707291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.782312 (+1075021us) service_pool.cc:225] Handling call
0502 08:16:55.918456 (+3136144us) ts_tablet_manager.cc:1120] Deleting tablet f797d3c6bdc4493398954fbbf9943971
0502 08:16:55.918465 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:55.923278 (+ 4813us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:55.957386 (+ 34108us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.119892 (+162506us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3136057}
I20240502 08:16:56.120290 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 849) took 5018 ms. Trace:
I20240502 08:16:56.120352 25012 rpcz_store.cc:276] 0502 08:16:51.101854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.747068 (+1645214us) service_pool.cc:225] Handling call
0502 08:16:56.120208 (+3373140us) ts_tablet_manager.cc:1120] Deleting tablet 642744194e22461f8316a78689fd99ff
0502 08:16:56.120217 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.120277 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3321912}
I20240502 08:16:56.120595 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 865) took 3131 ms. Trace:
I20240502 08:16:56.120646 25012 rpcz_store.cc:276] 0502 08:16:52.989604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.989651 (+ 47us) service_pool.cc:225] Handling call
0502 08:16:56.120515 (+3130864us) ts_tablet_manager.cc:1120] Deleting tablet f685cbe705b149e69f9536e270a74a42
0502 08:16:56.120522 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.120584 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3130694}
I20240502 08:16:56.120900 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 866) took 3130 ms. Trace:
I20240502 08:16:56.120954 25012 rpcz_store.cc:276] 0502 08:16:52.989981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.990020 (+ 39us) service_pool.cc:225] Handling call
0502 08:16:56.120813 (+3130793us) ts_tablet_manager.cc:1120] Deleting tablet 303a0a3748ee422faab856e1c4bff66d
0502 08:16:56.120821 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.120888 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3130553}
I20240502 08:16:56.121212 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 864) took 3131 ms. Trace:
I20240502 08:16:56.121268 25012 rpcz_store.cc:276] 0502 08:16:52.989510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.989927 (+ 417us) service_pool.cc:225] Handling call
0502 08:16:56.121134 (+3131207us) ts_tablet_manager.cc:1120] Deleting tablet fe5e08ad6af2442bb0a8fcc80a99b532
0502 08:16:56.121140 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.121200 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26,"tablet-delete.queue_time_us":3130868}
I20240502 08:16:56.080114 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 5027fa2c80b544b8b1e2a4694bec7454 (DEFAULT_TABLE table=default.Table_963bbb5ed3184e6a95549deb63ffb35a [id=d330f8cbe8ce4b089d6f8a9435ed07cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.121575 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 867) took 3131 ms. Trace:
I20240502 08:16:56.121803 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5027fa2c80b544b8b1e2a4694bec7454. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.121929 24175 raft_consensus.cc:374] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.121994 24175 raft_consensus.cc:729] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.122169 24175 consensus_queue.cc:260] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.122653 24175 ts_tablet_manager.cc:1398] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.114s user 0.021s sys 0.000s
I20240502 08:16:56.122871 24175 tablet_bootstrap.cc:492] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:56.124065 24175 tablet_bootstrap.cc:654] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.125296 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2a1a46a4e6f43669c72d5807e60c4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.129343 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40bbd76c59c649619abbcea1d36758ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.131351 24173 tablet_bootstrap.cc:492] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:56.131479 24173 ts_tablet_manager.cc:1367] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.000s
I20240502 08:16:56.132228 24173 raft_consensus.cc:348] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.132397 24173 raft_consensus.cc:374] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.132443 24173 raft_consensus.cc:729] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.132627 24173 consensus_queue.cc:260] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.133131 24173 ts_tablet_manager.cc:1398] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:56.133383 24173 tablet_bootstrap.cc:492] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:56.134915 24173 tablet_bootstrap.cc:654] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.137200 23765 tablet_service.cc:1467] Processing CreateTablet for tablet d6cdf43567be47c1b9d876017b31d157 (DEFAULT_TABLE table=default.Table_d40724ccadc148cd91ea36278515b65c [id=f8b9072ab9a2434a90efae6588087d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.137297 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 7f7ea2d3bc8a494ebdefa6de8ca389e0 (DEFAULT_TABLE table=default.Table_bbf9d7694fe942de8aa8bd4ed61336a5 [id=f1d6eb661b0143ee82e797cde61391c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.137579 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6cdf43567be47c1b9d876017b31d157. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.137755 23749 tablet_service.cc:1467] Processing CreateTablet for tablet f2a1a46a4e6f43669c72d5807e60c4dc (DEFAULT_TABLE table=default.Table_b59e0701b4d042f0b4d6277ade5ab169 [id=8d0d796950ba478b8058b8bb714ae329]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.137825 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f7ea2d3bc8a494ebdefa6de8ca389e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.138084 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2a1a46a4e6f43669c72d5807e60c4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.138051 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 40bbd76c59c649619abbcea1d36758ca (DEFAULT_TABLE table=default.Table_ec2cb5eeb2c34373b60522a9725b77e0 [id=214f79d839814815808dc415892e3472]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.138346 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40bbd76c59c649619abbcea1d36758ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.138530 23752 tablet_service.cc:1467] Processing CreateTablet for tablet e4e2309db5ca4c7cbafcdf7782121d23 (DEFAULT_TABLE table=default.Table_8e550c4078d04f7db883f8ed5e051554 [id=f9d35abc5f84451da41552124e9fd10b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.138803 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e2309db5ca4c7cbafcdf7782121d23. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:56.141749 23993 consensus_peers.cc:485] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:56.141764 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 55fb41b8f8f9487a81e98bae9cddaae1 (DEFAULT_TABLE table=default.Table_0d515c0c1c334feb9d4252e90ba9ad4e [id=1af5758bf1144300a4e635f7e1e2879f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.142113 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fb41b8f8f9487a81e98bae9cddaae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.147773 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 5027fa2c80b544b8b1e2a4694bec7454 (DEFAULT_TABLE table=default.Table_963bbb5ed3184e6a95549deb63ffb35a [id=d330f8cbe8ce4b089d6f8a9435ed07cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.147735 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 40bbd76c59c649619abbcea1d36758ca (DEFAULT_TABLE table=default.Table_ec2cb5eeb2c34373b60522a9725b77e0 [id=214f79d839814815808dc415892e3472]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.148173 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5027fa2c80b544b8b1e2a4694bec7454. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.148383 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40bbd76c59c649619abbcea1d36758ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.148414 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet f797d3c6bdc4493398954fbbf9943971 (table default.Table_7f44f7f1aed6460cb1e73392da0987b8 [id=07d8592503e843e18faedfcce845a974]) successfully deleted
W20240502 08:16:56.148684 23597 catalog_manager.cc:4797] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 642744194e22461f8316a78689fd99ff because the tablet was not found. No further retry: Not found: Tablet not found: 642744194e22461f8316a78689fd99ff
W20240502 08:16:56.148891 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet f685cbe705b149e69f9536e270a74a42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f685cbe705b149e69f9536e270a74a42 already in progress: creating tablet
W20240502 08:16:56.149077 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 303a0a3748ee422faab856e1c4bff66d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 303a0a3748ee422faab856e1c4bff66d already in progress: creating tablet
W20240502 08:16:56.149214 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet fe5e08ad6af2442bb0a8fcc80a99b532 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe5e08ad6af2442bb0a8fcc80a99b532 already in progress: creating tablet
I20240502 08:16:56.149824 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 7f7ea2d3bc8a494ebdefa6de8ca389e0 (DEFAULT_TABLE table=default.Table_bbf9d7694fe942de8aa8bd4ed61336a5 [id=f1d6eb661b0143ee82e797cde61391c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.150115 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f7ea2d3bc8a494ebdefa6de8ca389e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.153179 23889 tablet_service.cc:1467] Processing CreateTablet for tablet f2a1a46a4e6f43669c72d5807e60c4dc (DEFAULT_TABLE table=default.Table_b59e0701b4d042f0b4d6277ade5ab169 [id=8d0d796950ba478b8058b8bb714ae329]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.153512 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2a1a46a4e6f43669c72d5807e60c4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.154207 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 54310d86b1f9462a8f87728711ebf6fa (DEFAULT_TABLE table=default.Table_93b89623977c4ede951f10ea31e1762b [id=f978ad4c43364d5cbdad30b9b5a687a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.154476 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54310d86b1f9462a8f87728711ebf6fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.157974 23891 tablet_service.cc:1467] Processing CreateTablet for tablet d6cdf43567be47c1b9d876017b31d157 (DEFAULT_TABLE table=default.Table_d40724ccadc148cd91ea36278515b65c [id=f8b9072ab9a2434a90efae6588087d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.158303 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6cdf43567be47c1b9d876017b31d157. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.158744 24180 ts_tablet_manager.cc:1875] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.162309 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 1cd8338500f54604abcd6d42702c2eca (table default.Table_482a1b0b7a064e02805ac3b86fc62f99 [id=3c1d614616374739aeb928e797479df4]) successfully deleted
W20240502 08:16:56.162590 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 2ebf55c8024c483f9ef0afe6945a02ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ebf55c8024c483f9ef0afe6945a02ff already in progress: creating tablet
W20240502 08:16:56.162798 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 33bbebbca7464ea6ad59b81d7b7c560d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33bbebbca7464ea6ad59b81d7b7c560d already in progress: creating tablet
I20240502 08:16:56.163700 23887 tablet_service.cc:1467] Processing CreateTablet for tablet f13bd33dd7c44ac19fa4e0bba3083483 (DEFAULT_TABLE table=default.Table_bcfbfe93e2a14297828b5837679bfd8d [id=5155802598fa400fb72b3d9d37c359e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.164026 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13bd33dd7c44ac19fa4e0bba3083483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.164861 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 55fb41b8f8f9487a81e98bae9cddaae1 (DEFAULT_TABLE table=default.Table_0d515c0c1c334feb9d4252e90ba9ad4e [id=1af5758bf1144300a4e635f7e1e2879f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.165031 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 654ce33d1e3d472eb6ae6a726eb803bb (table default.Table_6f4d25ef5b90455ca233af7b70b9073a [id=2fe4c270673b4204aaccc289d3b9d929]) successfully deleted
I20240502 08:16:56.165205 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fb41b8f8f9487a81e98bae9cddaae1. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:56.165246 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 9f3c00e5393c47df924bcb5dcf2a0f19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f3c00e5393c47df924bcb5dcf2a0f19 already in progress: creating tablet
I20240502 08:16:56.168355 23916 raft_consensus.cc:3037] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:56.176472 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 91167335ab4441a1a2fc8d70cebdd2eb (DEFAULT_TABLE table=default.Table_7c9232b45ef64cb58387478ac1b14b45 [id=92bda79c30d940ddb02950c6ade6837f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.176930 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91167335ab4441a1a2fc8d70cebdd2eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.178066 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 3b10fecf092b4db18060971ef8f90872 (DEFAULT_TABLE table=default.Table_bac8a6673ffc44e7825538aa4257329e [id=945bdcf8f2854a19bf0fcd24117d8bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.178890 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b10fecf092b4db18060971ef8f90872. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.180147 24173 tablet_bootstrap.cc:492] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:56.180279 24173 ts_tablet_manager.cc:1367] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.001s
I20240502 08:16:56.181119 24173 raft_consensus.cc:348] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.181285 24173 raft_consensus.cc:374] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.181332 24173 raft_consensus.cc:729] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.121655 25012 rpcz_store.cc:276] 0502 08:16:52.990339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.990382 (+ 43us) service_pool.cc:225] Handling call
0502 08:16:56.121477 (+3131095us) ts_tablet_manager.cc:1120] Deleting tablet 29f6c3cea4b14364b6d073a92171119d
0502 08:16:56.121486 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.121561 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3130632}
I20240502 08:16:56.181531 24173 consensus_queue.cc:260] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.182456 24173 ts_tablet_manager.cc:1398] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:16:56.182940 24173 tablet_bootstrap.cc:492] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:56.185258 24174 tablet_bootstrap.cc:492] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:56.185446 24174 ts_tablet_manager.cc:1367] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.073s user 0.011s sys 0.015s
I20240502 08:16:56.185974 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 50d60243655e464c999398db95063c15 (DEFAULT_TABLE table=default.Table_2223dafc11344718b775bac0d51fa2b6 [id=f78d98fce9bf467982390f3cff53a268]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.186318 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d60243655e464c999398db95063c15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.186941 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 50d60243655e464c999398db95063c15 (DEFAULT_TABLE table=default.Table_2223dafc11344718b775bac0d51fa2b6 [id=f78d98fce9bf467982390f3cff53a268]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.187368 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d60243655e464c999398db95063c15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.192533 24174 raft_consensus.cc:348] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.192821 24174 raft_consensus.cc:374] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.192932 24174 raft_consensus.cc:729] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.193200 24174 consensus_queue.cc:260] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.193372 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 91167335ab4441a1a2fc8d70cebdd2eb (DEFAULT_TABLE table=default.Table_7c9232b45ef64cb58387478ac1b14b45 [id=92bda79c30d940ddb02950c6ade6837f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.193804 24174 ts_tablet_manager.cc:1398] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 08:16:56.194118 24174 tablet_bootstrap.cc:492] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:56.194340 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 3b10fecf092b4db18060971ef8f90872 (DEFAULT_TABLE table=default.Table_bac8a6673ffc44e7825538aa4257329e [id=945bdcf8f2854a19bf0fcd24117d8bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.194682 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7be16f77c74551aebbf6b06acd9100 (DEFAULT_TABLE table=default.Table_9e72800839ec45fead8b8b2c5afe4799 [id=3cde30910c38423a9b41e6f6dab30978]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.193812 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91167335ab4441a1a2fc8d70cebdd2eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.196838 24174 tablet_bootstrap.cc:654] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.198505 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7be16f77c74551aebbf6b06acd9100. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:56.199605 23996 consensus_peers.cc:485] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:56.203524 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b10fecf092b4db18060971ef8f90872. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.206737 25423 consensus_queue.cc:1035] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:56.186762 23886 tablet_service.cc:1467] Processing CreateTablet for tablet e4e2309db5ca4c7cbafcdf7782121d23 (DEFAULT_TABLE table=default.Table_8e550c4078d04f7db883f8ed5e051554 [id=f9d35abc5f84451da41552124e9fd10b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.183677 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 876) took 3103 ms. Trace:
I20240502 08:16:56.218614 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e2309db5ca4c7cbafcdf7782121d23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.218672 25012 rpcz_store.cc:276] 0502 08:16:53.079867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.079908 (+ 41us) service_pool.cc:225] Handling call
0502 08:16:56.183536 (+3103628us) ts_tablet_manager.cc:1120] Deleting tablet 87a3df4fbb224dd0b4ec6141584d374b
0502 08:16:56.183549 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.183653 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3103428}
I20240502 08:16:56.219069 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 877) took 3135 ms. Trace:
I20240502 08:16:56.219147 25012 rpcz_store.cc:276] 0502 08:16:53.083381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.083433 (+ 52us) service_pool.cc:225] Handling call
0502 08:16:56.218945 (+3135512us) ts_tablet_manager.cc:1120] Deleting tablet 9f3c00e5393c47df924bcb5dcf2a0f19
0502 08:16:56.218955 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.219052 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3135345}
I20240502 08:16:56.219446 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 878) took 3135 ms. Trace:
I20240502 08:16:56.219516 25012 rpcz_store.cc:276] 0502 08:16:53.083708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.083773 (+ 65us) service_pool.cc:225] Handling call
0502 08:16:56.219355 (+3135582us) ts_tablet_manager.cc:1120] Deleting tablet d2bb4dba23ac4b4eb53b3f2d75d0442e
0502 08:16:56.219364 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.219433 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3135456}
I20240502 08:16:56.219794 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 879) took 3135 ms. Trace:
I20240502 08:16:56.219856 25012 rpcz_store.cc:276] 0502 08:16:53.084331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.084551 (+ 220us) service_pool.cc:225] Handling call
0502 08:16:56.219700 (+3135149us) ts_tablet_manager.cc:1120] Deleting tablet 2085ced52fff46ebaa98324cbb2d8c11
0502 08:16:56.219709 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.219780 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3135013}
I20240502 08:16:56.220177 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 880) took 3134 ms. Trace:
I20240502 08:16:56.220247 25012 rpcz_store.cc:276] 0502 08:16:53.086623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.086829 (+ 206us) service_pool.cc:225] Handling call
0502 08:16:56.220086 (+3133257us) ts_tablet_manager.cc:1120] Deleting tablet ee3d91d55ffd49ceb20d2c33408f5a5a
0502 08:16:56.220095 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.220164 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3133069}
I20240502 08:16:56.220477 25012 tablet_replica.cc:330] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:56.220935 24037 tablet_service.cc:1467] Processing CreateTablet for tablet ba13189883454982a99f535701556c05 (DEFAULT_TABLE table=default.Table_bd032329c0574a53a1d3b7d977c4783b [id=ea6a434a58974b5fa427c9be27d8bd82]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.221274 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba13189883454982a99f535701556c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.223793 24173 tablet_bootstrap.cc:654] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.226977 24029 tablet_service.cc:1467] Processing CreateTablet for tablet d8150e91625a413d89e1b71b1c86048e (DEFAULT_TABLE table=default.Table_fc4c2996ae4c4e8ea466bf9625b823f8 [id=f0202e3d3721423b812ae77573f5ea1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.228050 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8150e91625a413d89e1b71b1c86048e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.229444 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7be16f77c74551aebbf6b06acd9100 (DEFAULT_TABLE table=default.Table_9e72800839ec45fead8b8b2c5afe4799 [id=3cde30910c38423a9b41e6f6dab30978]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.229800 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7be16f77c74551aebbf6b06acd9100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.233775 24038 tablet_service.cc:1467] Processing CreateTablet for tablet b1dcb923e2364b358b6e443a1ede703c (DEFAULT_TABLE table=default.Table_6140ac5b9ad042a08e94f15c3c2efd16 [id=95d469fab066477ea87282478ca0d835]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.234606 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1dcb923e2364b358b6e443a1ede703c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.242439 23751 tablet_service.cc:1467] Processing CreateTablet for tablet d8150e91625a413d89e1b71b1c86048e (DEFAULT_TABLE table=default.Table_fc4c2996ae4c4e8ea466bf9625b823f8 [id=f0202e3d3721423b812ae77573f5ea1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.242866 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8150e91625a413d89e1b71b1c86048e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.243359 23759 tablet_service.cc:1467] Processing CreateTablet for tablet b1dcb923e2364b358b6e443a1ede703c (DEFAULT_TABLE table=default.Table_6140ac5b9ad042a08e94f15c3c2efd16 [id=95d469fab066477ea87282478ca0d835]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:56.249413 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 29f6c3cea4b14364b6d073a92171119d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29f6c3cea4b14364b6d073a92171119d already in progress: creating tablet
W20240502 08:16:56.249789 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 87a3df4fbb224dd0b4ec6141584d374b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87a3df4fbb224dd0b4ec6141584d374b already in progress: creating tablet
W20240502 08:16:56.250028 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 9f3c00e5393c47df924bcb5dcf2a0f19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f3c00e5393c47df924bcb5dcf2a0f19 already in progress: creating tablet
W20240502 08:16:56.250227 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet d2bb4dba23ac4b4eb53b3f2d75d0442e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2bb4dba23ac4b4eb53b3f2d75d0442e already in progress: creating tablet
W20240502 08:16:56.250443 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 2085ced52fff46ebaa98324cbb2d8c11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2085ced52fff46ebaa98324cbb2d8c11 already in progress: creating tablet
W20240502 08:16:56.250617 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet ee3d91d55ffd49ceb20d2c33408f5a5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee3d91d55ffd49ceb20d2c33408f5a5a already in progress: creating tablet
I20240502 08:16:56.257792 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 881) took 3674 ms. Trace:
W20240502 08:16:56.258250 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet d2bb4dba23ac4b4eb53b3f2d75d0442e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2bb4dba23ac4b4eb53b3f2d75d0442e already in progress: creating tablet
I20240502 08:16:56.258066 24181 rpcz_store.cc:276] 0502 08:16:52.583091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.583781 (+ 690us) service_pool.cc:225] Handling call
0502 08:16:56.257658 (+3673877us) ts_tablet_manager.cc:1120] Deleting tablet 87a3df4fbb224dd0b4ec6141584d374b
0502 08:16:56.257669 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.257772 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3673753}
W20240502 08:16:56.260279 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 87a3df4fbb224dd0b4ec6141584d374b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87a3df4fbb224dd0b4ec6141584d374b already in progress: creating tablet
I20240502 08:16:56.260752 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 884) took 3677 ms. Trace:
I20240502 08:16:56.260977 24173 tablet_bootstrap.cc:492] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:56.265874 24173 ts_tablet_manager.cc:1367] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.003s
I20240502 08:16:56.266806 24173 raft_consensus.cc:348] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.267062 24173 raft_consensus.cc:374] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.267201 24173 raft_consensus.cc:729] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.267513 24173 consensus_queue.cc:260] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.270449 24173 ts_tablet_manager.cc:1398] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:16:56.270738 24173 tablet_bootstrap.cc:492] T 25c089f0c1b14e419f250d03d4eaad12 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:56.272272 24173 tablet_bootstrap.cc:654] T 25c089f0c1b14e419f250d03d4eaad12 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.273489 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 7fe583cbf4af493e82d3ec0508ea0f0b
I20240502 08:16:56.274307 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_655a2929f122445780cd4db021cb35f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:56.275141 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b673548cd88d4f51975f7433f1509cc9" } new_table_name: "default.Table_e2743b70968340b69aea47c559536937" modify_external_catalogs: true
I20240502 08:16:56.276712 24173 tablet_bootstrap.cc:492] T 25c089f0c1b14e419f250d03d4eaad12 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:56.276845 24173 ts_tablet_manager.cc:1367] T 25c089f0c1b14e419f250d03d4eaad12 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240502 08:16:56.276813 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7d9efd3ee8d94c048fb1ce4498a4167b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:56.277228 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d9efd3ee8d94c048fb1ce4498a4167b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:56.277585 24173 raft_consensus.cc:348] T 25c089f0c1b14e419f250d03d4eaad12 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.277776 24173 raft_consensus.cc:374] T 25c089f0c1b14e419f250d03d4eaad12 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.277827 24173 raft_consensus.cc:729] T 25c089f0c1b14e419f250d03d4eaad12 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.278044 24173 consensus_queue.cc:260] T 25c089f0c1b14e419f250d03d4eaad12 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.278268 24181 rpcz_store.cc:276] 0502 08:16:52.583328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.585379 (+ 2051us) service_pool.cc:225] Handling call
0502 08:16:56.260635 (+3675256us) ts_tablet_manager.cc:1120] Deleting tablet 2085ced52fff46ebaa98324cbb2d8c11
0502 08:16:56.260644 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.260734 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3675039}
I20240502 08:16:56.278460 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_978b0604ce1c4638bcd51dc8389941d9" } new_table_name: "default.Table_4481e3bc8b3548b0a7ecef01d39f80e8" modify_external_catalogs: true
W20240502 08:16:56.278789 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 2085ced52fff46ebaa98324cbb2d8c11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2085ced52fff46ebaa98324cbb2d8c11 already in progress: creating tablet
I20240502 08:16:56.279064 24181 tablet_replica.cc:330] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:56.279287 24181 raft_consensus.cc:2229] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.279453 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fe583cbf4af493e82d3ec0508ea0f0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:56.281708 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_22d85074cecf41dbbe8ca2743891e567"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:56.282141 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22d85074cecf41dbbe8ca2743891e567 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:56.282974 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_655a2929f122445780cd4db021cb35f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:56.291236 24181 raft_consensus.cc:2258] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.291517 25423 raft_consensus.cc:482] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.291627 25423 raft_consensus.cc:504] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.292217 25423 leader_election.cc:288] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:56.292923 24181 ts_tablet_manager.cc:1875] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.293381 24173 ts_tablet_manager.cc:1398] T 25c089f0c1b14e419f250d03d4eaad12 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 08:16:56.297371 24173 tablet_bootstrap.cc:492] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:56.299713 24173 tablet_bootstrap.cc:654] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.220652 25012 raft_consensus.cc:2229] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.300204 25012 raft_consensus.cc:2258] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.246811 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 3b10fecf092b4db18060971ef8f90872 (DEFAULT_TABLE table=default.Table_bac8a6673ffc44e7825538aa4257329e [id=945bdcf8f2854a19bf0fcd24117d8bcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.300796 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b10fecf092b4db18060971ef8f90872. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.306505 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_3a88ce68a50f4471b706d37631957df0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:56.247058 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7be16f77c74551aebbf6b06acd9100 (DEFAULT_TABLE table=default.Table_9e72800839ec45fead8b8b2c5afe4799 [id=3cde30910c38423a9b41e6f6dab30978]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:56.307227 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a88ce68a50f4471b706d37631957df0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:56.307511 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7be16f77c74551aebbf6b06acd9100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.308084 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0f452d0e88414d98a241e34d44354db3" } new_table_name: "default.Table_36c237ae3b544378a5f1652677455de2" modify_external_catalogs: true
I20240502 08:16:56.309181 23607 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 9761df0bbdee44af90135bffb8b5828f (table default.Table_8ba240c391da43978f93a27dce48e557 [id=1b5536950db244d1978b154da0be33e0]). Expected version 2 got 0
I20240502 08:16:56.309360 23607 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet af6efeebaebc495aba017a922e243c39 (table default.Table_82f82cd6e702420283c288c10f905d90 [id=f293b22818344aea9eb6232141432183]). Expected version 1 got 0
I20240502 08:16:56.309495 23607 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 7ed919909289410e867881baab841f53 (table default.Table_2edad641920b44ce9a4d5b7503f0279a [id=ee3912407a934444ad2c62f7b83d93ac]). Expected version 2 got 0
I20240502 08:16:56.247097 23900 tablet_service.cc:1467] Processing CreateTablet for tablet d8150e91625a413d89e1b71b1c86048e (DEFAULT_TABLE table=default.Table_fc4c2996ae4c4e8ea466bf9625b823f8 [id=f0202e3d3721423b812ae77573f5ea1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.310114 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8150e91625a413d89e1b71b1c86048e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.310765 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_825b08eccbb0430395675758faf7b2f8" } modify_external_catalogs: true
I20240502 08:16:56.310928 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_825b08eccbb0430395675758faf7b2f8" } modify_external_catalogs: true
I20240502 08:16:56.312649 23897 tablet_service.cc:1514] Processing DeleteTablet for tablet 128c88af54d24ffd895e9de19cdb4c8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:56.312768 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet c00f29fb76b44d6ea4f76963d2d0751c with delete_type TABLET_DATA_DELETED (replaced by tablet a17fb4f2f02e43dc9f7cfdb7c73f7774) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:56.312948 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d04312bd81f46ed8c49c42b1af1aec9 with delete_type TABLET_DATA_DELETED (replaced by tablet b45f3233dae94be1a1d0d8fb426d7f2c) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:56.313103 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet 955e7bd2216646c6aef5acce8d5b0d77 with delete_type TABLET_DATA_DELETED (replaced by tablet 4645143a00d440a7b6d56c5e30d249f1) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:56.313360 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f02efa659e457ab57bc32ed3f66bb9 with delete_type TABLET_DATA_DELETED (replaced by tablet df06e0edb57f45e5ac71b568c14769cd) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:56.313541 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c8131846dc41678827b86e87be18d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:56.247289 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 91167335ab4441a1a2fc8d70cebdd2eb (DEFAULT_TABLE table=default.Table_7c9232b45ef64cb58387478ac1b14b45 [id=92bda79c30d940ddb02950c6ade6837f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.313932 23892 tablet_service.cc:1514] Processing DeleteTablet for tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:56.247289 23895 tablet_service.cc:1467] Processing CreateTablet for tablet b1dcb923e2364b358b6e443a1ede703c (DEFAULT_TABLE table=default.Table_6140ac5b9ad042a08e94f15c3c2efd16 [id=95d469fab066477ea87282478ca0d835]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.314476 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3c059ec418cd4c6dbf7bf93c0856437f" } new_table_name: "default.Table_d45052f775a8497fad354f27c3def079" modify_external_catalogs: true
I20240502 08:16:56.260186 23888 tablet_service.cc:1467] Processing CreateTablet for tablet ba13189883454982a99f535701556c05 (DEFAULT_TABLE table=default.Table_bd032329c0574a53a1d3b7d977c4783b [id=ea6a434a58974b5fa427c9be27d8bd82]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.317822 23602 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 7ed919909289410e867881baab841f53 (table default.Table_2edad641920b44ce9a4d5b7503f0279a [id=ee3912407a934444ad2c62f7b83d93ac]). Expected version 2 got 0
I20240502 08:16:56.260777 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 50d60243655e464c999398db95063c15 (DEFAULT_TABLE table=default.Table_2223dafc11344718b775bac0d51fa2b6 [id=f78d98fce9bf467982390f3cff53a268]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.292789 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b811b81ddf2b4dc9b5e5f80331bff294" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:56.318684 23920 raft_consensus.cc:2454] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:56.319159 23996 leader_election.cc:302] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:56.319435 25423 raft_consensus.cc:2781] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:16:56.319518 25423 raft_consensus.cc:482] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:56.244127 24175 tablet_bootstrap.cc:492] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:56.319581 25423 raft_consensus.cc:3037] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:16:56.278992 23903 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fe583cbf4af493e82d3ec0508ea0f0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:56.319676 24175 ts_tablet_manager.cc:1367] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.197s user 0.010s sys 0.012s
I20240502 08:16:56.269906 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 55fb41b8f8f9487a81e98bae9cddaae1 (DEFAULT_TABLE table=default.Table_0d515c0c1c334feb9d4252e90ba9ad4e [id=1af5758bf1144300a4e635f7e1e2879f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.322031 25423 raft_consensus.cc:504] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.322189 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_59fc9761e46d4ff0bad5029d2b0b25fc" } modify_external_catalogs: true
I20240502 08:16:56.322429 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_59fc9761e46d4ff0bad5029d2b0b25fc" } modify_external_catalogs: true
I20240502 08:16:56.322808 25423 leader_election.cc:288] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 3 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:56.325637 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b811b81ddf2b4dc9b5e5f80331bff294" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:56.325824 23920 raft_consensus.cc:3037] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Advancing to term 3
I20240502 08:16:56.328305 23920 raft_consensus.cc:2454] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 3.
I20240502 08:16:56.328807 23996 leader_election.cc:302] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:56.329066 25423 raft_consensus.cc:2781] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:16:56.329284 25423 raft_consensus.cc:686] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 3 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:56.329525 25423 consensus_queue.cc:237] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.329954 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet 592ab8523b8d479d9caead321d5888c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:56.331064 24180 ts_tablet_manager.cc:1888] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:56.332648 24175 raft_consensus.cc:348] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.332804 24175 raft_consensus.cc:374] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.332849 24175 raft_consensus.cc:729] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.333046 24175 consensus_queue.cc:260] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.333572 24175 ts_tablet_manager.cc:1398] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240502 08:16:56.337374 25012 ts_tablet_manager.cc:1875] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.337782 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a4104f4cb64889ad66810788ad6b29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:56.338081 25531 raft_consensus.cc:482] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.338208 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet 654ce33d1e3d472eb6ae6a726eb803bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:56.338281 25536 raft_consensus.cc:482] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.338379 25536 raft_consensus.cc:504] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.338505 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fe583cbf4af493e82d3ec0508ea0f0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:56.339154 25536 leader_election.cc:288] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:56.341969 23762 tablet_service.cc:1467] Processing CreateTablet for tablet ba13189883454982a99f535701556c05 (DEFAULT_TABLE table=default.Table_bd032329c0574a53a1d3b7d977c4783b [id=ea6a434a58974b5fa427c9be27d8bd82]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:56.342660 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07344b78894c42809391d0fe2a9cd8f7" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:56.343307 23858 leader_election.cc:341] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 07344b78894c42809391d0fe2a9cd8f7
I20240502 08:16:56.343788 24175 tablet_bootstrap.cc:492] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:56.343933 23861 leader_election.cc:341] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.344040 23861 leader_election.cc:302] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:56.344486 25536 raft_consensus.cc:2726] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:56.350070 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1dcb923e2364b358b6e443a1ede703c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.350358 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba13189883454982a99f535701556c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.353677 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b811b81ddf2b4dc9b5e5f80331bff294" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:56.353981 24180 log.cc:1196] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/955e7bd2216646c6aef5acce8d5b0d77
I20240502 08:16:56.354156 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b811b81ddf2b4dc9b5e5f80331bff294" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:56.355907 23783 raft_consensus.cc:2454] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 2.
I20240502 08:16:56.356493 24180 ts_tablet_manager.cc:1909] T 955e7bd2216646c6aef5acce8d5b0d77 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.358835 24175 tablet_bootstrap.cc:654] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.361747 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d60243655e464c999398db95063c15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.361789 25531 raft_consensus.cc:504] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.362066 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91167335ab4441a1a2fc8d70cebdd2eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.362421 25531 leader_election.cc:288] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:56.362648 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba13189883454982a99f535701556c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.362874 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9761df0bbdee44af90135bffb8b5828f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:56.363060 24051 raft_consensus.cc:2454] T 9761df0bbdee44af90135bffb8b5828f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:56.363420 23723 leader_election.cc:302] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:56.363945 25531 raft_consensus.cc:2781] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:56.364013 25531 raft_consensus.cc:482] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:56.364059 25531 raft_consensus.cc:3037] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:56.365108 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fb41b8f8f9487a81e98bae9cddaae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.365710 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1dcb923e2364b358b6e443a1ede703c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:56.365883 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 958) took 3232 ms. Trace:
I20240502 08:16:56.365967 24180 rpcz_store.cc:276] 0502 08:16:53.124494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.124560 (+ 66us) service_pool.cc:225] Handling call
0502 08:16:56.117837 (+2993277us) ts_tablet_manager.cc:1120] Deleting tablet 955e7bd2216646c6aef5acce8d5b0d77
0502 08:16:56.117845 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.210193 (+ 92348us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.233709 (+ 23516us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.357354 (+123645us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2993017}
I20240502 08:16:56.366287 24180 tablet_replica.cc:330] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:56.366465 24180 raft_consensus.cc:2229] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.366478 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 955e7bd2216646c6aef5acce8d5b0d77 (table default.Table_b3278c5d6d0b4593ac33bc9977f5c3b1 [id=e86031550cec46a793785c0b3859fa92]) successfully deleted
I20240502 08:16:56.366591 24180 raft_consensus.cc:2258] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.366894 25531 raft_consensus.cc:504] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.367072 24180 ts_tablet_manager.cc:1875] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.368031 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9761df0bbdee44af90135bffb8b5828f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:56.368644 25531 leader_election.cc:288] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:56.368965 23723 leader_election.cc:341] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.369304 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9761df0bbdee44af90135bffb8b5828f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:56.369800 23723 leader_election.cc:341] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.370635 24174 tablet_bootstrap.cc:492] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:56.370777 24174 ts_tablet_manager.cc:1367] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.177s user 0.007s sys 0.009s
I20240502 08:16:56.371541 24174 raft_consensus.cc:348] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.371784 24174 raft_consensus.cc:374] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.371884 24174 raft_consensus.cc:729] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.372604 24174 consensus_queue.cc:260] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.373239 24174 ts_tablet_manager.cc:1398] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:56.376363 24174 tablet_bootstrap.cc:492] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:56.368510 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9761df0bbdee44af90135bffb8b5828f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:56.376925 24051 raft_consensus.cc:3037] T 9761df0bbdee44af90135bffb8b5828f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:56.377172 24175 tablet_bootstrap.cc:492] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:56.377285 24175 ts_tablet_manager.cc:1367] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.000s
I20240502 08:16:56.377928 24175 raft_consensus.cc:348] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.378096 24175 raft_consensus.cc:374] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.378146 24175 raft_consensus.cc:729] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.378324 24175 consensus_queue.cc:260] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.378811 24175 ts_tablet_manager.cc:1398] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:16:56.382597 24174 tablet_bootstrap.cc:654] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.317255 24181 ts_tablet_manager.cc:1888] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 08:16:56.383715 24181 log.cc:1196] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/050c16739efb4037a4adc78a2ffd3cba
I20240502 08:16:56.384166 24181 ts_tablet_manager.cc:1909] T 050c16739efb4037a4adc78a2ffd3cba P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:56.384548 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 885) took 3798 ms. Trace:
I20240502 08:16:56.384686 24181 rpcz_store.cc:276] 0502 08:16:52.585754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.586093 (+ 339us) service_pool.cc:225] Handling call
0502 08:16:56.279019 (+3692926us) ts_tablet_manager.cc:1120] Deleting tablet 050c16739efb4037a4adc78a2ffd3cba
0502 08:16:56.279030 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.314339 (+ 35309us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.317141 (+ 2802us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.384527 (+ 67386us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11737,"tablet-delete.queue_time_us":3692768}
I20240502 08:16:56.385298 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 050c16739efb4037a4adc78a2ffd3cba (table default.Table_3a66619663c642f5be35fd8249184e1a [id=ba62a673079d424992daa0f6b0c5d6b9]) successfully deleted
I20240502 08:16:56.385797 24181 tablet_replica.cc:330] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:56.386066 24181 raft_consensus.cc:2229] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.386371 24181 raft_consensus.cc:2258] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.386483 24180 ts_tablet_manager.cc:1888] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:56.386600 24180 log.cc:1196] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/33c8131846dc41678827b86e87be18d3
I20240502 08:16:56.386886 24180 ts_tablet_manager.cc:1909] T 33c8131846dc41678827b86e87be18d3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.387426 24181 ts_tablet_manager.cc:1875] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.387818 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 959) took 3249 ms. Trace:
I20240502 08:16:56.387908 24180 rpcz_store.cc:276] 0502 08:16:53.138237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.138287 (+ 50us) service_pool.cc:225] Handling call
0502 08:16:56.366239 (+3227952us) ts_tablet_manager.cc:1120] Deleting tablet 33c8131846dc41678827b86e87be18d3
0502 08:16:56.366249 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.371784 (+ 5535us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.386385 (+ 14601us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.387799 (+ 1414us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3227729}
I20240502 08:16:56.388358 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 33c8131846dc41678827b86e87be18d3 (table default.Table_bd9708a3da8240b4aa9444c8e9ed30d2 [id=5849a3c8b28e47439b73e2137187fac9]) successfully deleted
I20240502 08:16:56.389256 24051 raft_consensus.cc:2454] T 9761df0bbdee44af90135bffb8b5828f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:56.389672 23723 leader_election.cc:302] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:56.389798 24180 tablet_replica.cc:330] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:56.389969 24180 raft_consensus.cc:2229] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.390064 25531 raft_consensus.cc:2781] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:56.390162 25531 raft_consensus.cc:686] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:56.390275 24180 raft_consensus.cc:2258] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.390369 25531 consensus_queue.cc:237] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.394511 24175 tablet_bootstrap.cc:492] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:56.396124 24175 tablet_bootstrap.cc:654] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.396131 24173 tablet_bootstrap.cc:492] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:56.396247 24173 ts_tablet_manager.cc:1367] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.099s user 0.001s sys 0.001s
I20240502 08:16:56.396963 24173 raft_consensus.cc:348] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.397138 24173 raft_consensus.cc:374] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.397188 24173 raft_consensus.cc:729] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.397394 24173 consensus_queue.cc:260] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.397861 24173 ts_tablet_manager.cc:1398] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:56.408529 24180 ts_tablet_manager.cc:1875] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.418511 24173 tablet_bootstrap.cc:492] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:56.418829 24174 tablet_bootstrap.cc:492] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:56.419003 24174 ts_tablet_manager.cc:1367] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.004s
I20240502 08:16:56.419785 24174 raft_consensus.cc:348] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.420029 24174 raft_consensus.cc:374] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.420106 24173 tablet_bootstrap.cc:654] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.420127 24174 raft_consensus.cc:729] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.420428 24174 consensus_queue.cc:260] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.430310 24174 ts_tablet_manager.cc:1398] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240502 08:16:56.430651 24174 tablet_bootstrap.cc:492] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:56.430611 25317 raft_consensus.cc:482] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.430886 25317 raft_consensus.cc:504] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.431814 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df08f0d3af4c4c7e8abce656714b1b7f" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:56.432015 23783 raft_consensus.cc:2454] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:56.432044 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df08f0d3af4c4c7e8abce656714b1b7f" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:56.432384 23993 leader_election.cc:302] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:56.432629 25413 raft_consensus.cc:2781] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:56.432689 25413 raft_consensus.cc:482] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:56.432739 25413 raft_consensus.cc:3037] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:56.433169 24175 tablet_bootstrap.cc:492] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:56.433302 24175 ts_tablet_manager.cc:1367] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.003s
I20240502 08:16:56.434003 24175 raft_consensus.cc:348] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.434252 24175 raft_consensus.cc:374] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.434299 24175 raft_consensus.cc:729] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.434535 24175 consensus_queue.cc:260] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.435192 25413 raft_consensus.cc:504] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.435724 25413 leader_election.cc:288] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:56.436504 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df08f0d3af4c4c7e8abce656714b1b7f" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:56.436556 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df08f0d3af4c4c7e8abce656714b1b7f" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:56.436691 23783 raft_consensus.cc:3037] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:56.436964 23996 leader_election.cc:341] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:56.437297 23996 leader_election.cc:341] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.437405 25317 leader_election.cc:288] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:56.438067 24180 ts_tablet_manager.cc:1888] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:56.438159 24180 log.cc:1196] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/fbef606ce7ac499d9bd84b49d804621c
I20240502 08:16:56.438457 24180 ts_tablet_manager.cc:1909] T fbef606ce7ac499d9bd84b49d804621c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.439095 23783 raft_consensus.cc:2454] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:56.439131 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 960) took 3300 ms. Trace:
I20240502 08:16:56.439208 24180 rpcz_store.cc:276] 0502 08:16:53.138952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.138996 (+ 44us) service_pool.cc:225] Handling call
0502 08:16:56.389752 (+3250756us) ts_tablet_manager.cc:1120] Deleting tablet fbef606ce7ac499d9bd84b49d804621c
0502 08:16:56.389761 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.412330 (+ 22569us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.437967 (+ 25637us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.439113 (+ 1146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3250612}
I20240502 08:16:56.439604 23993 leader_election.cc:302] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:56.439615 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet fbef606ce7ac499d9bd84b49d804621c (table default.Table_1e7f2516f34b483d913cf5080a8febec [id=d5c976b2debc472fa1d7c28ae73ca32a]) successfully deleted
I20240502 08:16:56.440201 25413 raft_consensus.cc:2781] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:56.440517 25413 raft_consensus.cc:686] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:56.440786 25413 consensus_queue.cc:237] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.441141 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 965) took 3195 ms. Trace:
I20240502 08:16:56.441221 24180 rpcz_store.cc:276] 0502 08:16:53.245756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.246561 (+ 805us) service_pool.cc:225] Handling call
0502 08:16:56.439473 (+3192912us) ts_tablet_manager.cc:1120] Deleting tablet 418ab502915c4e4ea9998dd2d0180c48
0502 08:16:56.439481 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.441122 (+ 1641us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3192753}
I20240502 08:16:56.441793 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 974) took 3065 ms. Trace:
I20240502 08:16:56.441874 24180 rpcz_store.cc:276] 0502 08:16:53.376762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.376819 (+ 57us) service_pool.cc:225] Handling call
0502 08:16:56.441447 (+3064628us) ts_tablet_manager.cc:1120] Deleting tablet 40fab016ed4f489aa3ccdeafcfbe1477
0502 08:16:56.441455 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.441776 (+ 321us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3064328}
I20240502 08:16:56.442132 24180 tablet_replica.cc:330] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
W20240502 08:16:56.442306 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 40fab016ed4f489aa3ccdeafcfbe1477 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40fab016ed4f489aa3ccdeafcfbe1477 already in progress: creating tablet
I20240502 08:16:56.442333 24180 raft_consensus.cc:2229] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.442484 24180 raft_consensus.cc:2258] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.442560 25531 raft_consensus.cc:482] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.442674 25531 raft_consensus.cc:504] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.442870 24175 ts_tablet_manager.cc:1398] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 08:16:56.443002 24180 ts_tablet_manager.cc:1875] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.443235 25531 leader_election.cc:288] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:56.444020 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592eeecbf5514215b6464bd030081e4b" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:56.444522 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592eeecbf5514215b6464bd030081e4b" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:56.445030 23723 leader_election.cc:341] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:56.445408 23723 leader_election.cc:341] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.445554 23723 leader_election.cc:302] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:56.446106 25531 raft_consensus.cc:2726] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:56.447337 24175 tablet_bootstrap.cc:492] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:56.448194 24174 tablet_bootstrap.cc:654] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.451789 23788 raft_consensus.cc:3037] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:56.455603 25184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=538 overwritten=0 applied=534 ignored=0} inserts{seen=502 ignored=0} mutations{seen=623 ignored=0} orphaned_commits=0. Pending: 4 replicates
W20240502 08:16:56.459316 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 968a301fd817452a8b4552e959d2e371 consensus state has no leader
I20240502 08:16:56.459420 24180 ts_tablet_manager.cc:1888] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.459510 24180 log.cc:1196] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/577b149f52e44d5d9a9f6b9d85a6864d
I20240502 08:16:56.459882 24180 ts_tablet_manager.cc:1909] T 577b149f52e44d5d9a9f6b9d85a6864d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.460551 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 976) took 3083 ms. Trace:
I20240502 08:16:56.460625 24180 rpcz_store.cc:276] 0502 08:16:53.376969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.377206 (+ 237us) service_pool.cc:225] Handling call
0502 08:16:56.442093 (+3064887us) ts_tablet_manager.cc:1120] Deleting tablet 577b149f52e44d5d9a9f6b9d85a6864d
0502 08:16:56.442101 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.445865 (+ 3764us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.459325 (+ 13460us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.460533 (+ 1208us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3063334}
I20240502 08:16:56.460984 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 975) took 3084 ms. Trace:
I20240502 08:16:56.461112 24174 tablet_bootstrap.cc:492] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:56.461042 24180 rpcz_store.cc:276] 0502 08:16:53.376867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.388040 (+ 11173us) service_pool.cc:225] Handling call
0502 08:16:56.460871 (+3072831us) ts_tablet_manager.cc:1120] Deleting tablet df87b8b1ff504031bf007880792192dd
0502 08:16:56.460879 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.460971 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3072500}
I20240502 08:16:56.461079 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 577b149f52e44d5d9a9f6b9d85a6864d (table default.Table_1600e0e09856405ea0d90ac82aa47fda [id=89eea7c9830f47d2bd1c95044d5af4d6]) successfully deleted
I20240502 08:16:56.461210 24174 ts_tablet_manager.cc:1367] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.001s
I20240502 08:16:56.461319 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 980) took 3038 ms. Trace:
I20240502 08:16:56.461405 24180 rpcz_store.cc:276] 0502 08:16:53.423021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.423079 (+ 58us) service_pool.cc:225] Handling call
0502 08:16:56.461222 (+3038143us) ts_tablet_manager.cc:1120] Deleting tablet 711a38d7c47749f8a7f59e52b81b51ee
0502 08:16:56.461229 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.461304 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3037947}
W20240502 08:16:56.461494 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet df87b8b1ff504031bf007880792192dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df87b8b1ff504031bf007880792192dd already in progress: creating tablet
I20240502 08:16:56.461673 24180 tablet_replica.cc:330] T dd3078cc1f6247609f8bd017ed076b45 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:56.461835 24180 raft_consensus.cc:2229] T dd3078cc1f6247609f8bd017ed076b45 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240502 08:16:56.461911 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 711a38d7c47749f8a7f59e52b81b51ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 711a38d7c47749f8a7f59e52b81b51ee already in progress: creating tablet
I20240502 08:16:56.461962 24180 raft_consensus.cc:2258] T dd3078cc1f6247609f8bd017ed076b45 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.461946 24174 raft_consensus.cc:348] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:16:56.462098 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e20e126b1c9546e4a194bf9842f444a5 consensus state has no leader
I20240502 08:16:56.462116 24174 raft_consensus.cc:374] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.462160 24174 raft_consensus.cc:729] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.462319 24174 consensus_queue.cc:260] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.462782 24174 ts_tablet_manager.cc:1398] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:56.464577 24180 ts_tablet_manager.cc:1875] T dd3078cc1f6247609f8bd017ed076b45 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.464771 25317 consensus_queue.cc:1035] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:56.465054 24174 tablet_bootstrap.cc:492] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:56.466296 25184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1: Bootstrap complete.
I20240502 08:16:56.470532 25502 consensus_queue.cc:1035] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:56.472054 24174 tablet_bootstrap.cc:654] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.480031 24181 ts_tablet_manager.cc:1888] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:16:56.482421 24181 log.cc:1196] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/77c49da1e0f0435ba241a6a66526e790
I20240502 08:16:56.483302 24180 ts_tablet_manager.cc:1888] T dd3078cc1f6247609f8bd017ed076b45 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.483582 24180 log.cc:1196] T dd3078cc1f6247609f8bd017ed076b45 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/dd3078cc1f6247609f8bd017ed076b45
I20240502 08:16:56.485951 24181 ts_tablet_manager.cc:1909] T 77c49da1e0f0435ba241a6a66526e790 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
W20240502 08:16:56.486583 23858 consensus_peers.cc:485] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 128c88af54d24ffd895e9de19cdb4c8a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:56.487615 24180 ts_tablet_manager.cc:1909] T dd3078cc1f6247609f8bd017ed076b45 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.488523 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 994) took 2920 ms. Trace:
I20240502 08:16:56.488593 24180 rpcz_store.cc:276] 0502 08:16:53.567711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.568440 (+ 729us) service_pool.cc:225] Handling call
0502 08:16:56.461609 (+2893169us) ts_tablet_manager.cc:1120] Deleting tablet dd3078cc1f6247609f8bd017ed076b45
0502 08:16:56.461619 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.480464 (+ 18845us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.483225 (+ 2761us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.488504 (+ 5279us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2892993}
I20240502 08:16:56.488921 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 995) took 2918 ms. Trace:
I20240502 08:16:56.488988 24180 rpcz_store.cc:276] 0502 08:16:53.570460 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.570510 (+ 50us) service_pool.cc:225] Handling call
0502 08:16:56.488822 (+2918312us) ts_tablet_manager.cc:1120] Deleting tablet 88acb3b0c2b7480e917ed68b7646b47d
0502 08:16:56.488830 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.488908 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2918062}
I20240502 08:16:56.489076 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet dd3078cc1f6247609f8bd017ed076b45 (table default.Table_bac7f229ce9541b39c0a70951431b689 [id=f9f2bf985f124269982869050bbecd70]) successfully deleted
I20240502 08:16:56.489457 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 996) took 2901 ms. Trace:
I20240502 08:16:56.489524 24180 rpcz_store.cc:276] 0502 08:16:53.588045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.592089 (+ 4044us) service_pool.cc:225] Handling call
0502 08:16:56.489195 (+2897106us) ts_tablet_manager.cc:1120] Deleting tablet 765fdb66f12941f0ac16223e931e576e
0502 08:16:56.489202 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.489442 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2896903}
W20240502 08:16:56.489684 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 88acb3b0c2b7480e917ed68b7646b47d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88acb3b0c2b7480e917ed68b7646b47d already in progress: creating tablet
I20240502 08:16:56.489841 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 997) took 2893 ms. Trace:
W20240502 08:16:56.490161 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 765fdb66f12941f0ac16223e931e576e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 765fdb66f12941f0ac16223e931e576e already in progress: creating tablet
I20240502 08:16:56.489902 24180 rpcz_store.cc:276] 0502 08:16:53.595921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.597212 (+ 1291us) service_pool.cc:225] Handling call
0502 08:16:56.489752 (+2892540us) ts_tablet_manager.cc:1120] Deleting tablet 552e46f69230474189278911cfeec6c7
0502 08:16:56.489760 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.489827 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2892340}
I20240502 08:16:56.490944 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 998) took 2893 ms. Trace:
I20240502 08:16:56.491019 24180 rpcz_store.cc:276] 0502 08:16:53.597140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.597474 (+ 334us) service_pool.cc:225] Handling call
0502 08:16:56.490859 (+2893385us) ts_tablet_manager.cc:1120] Deleting tablet df266081ed8345dda989b3050f605685
0502 08:16:56.490866 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.490929 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2892995}
W20240502 08:16:56.491184 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 552e46f69230474189278911cfeec6c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 552e46f69230474189278911cfeec6c7 already in progress: creating tablet
I20240502 08:16:56.491295 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1007) took 2517 ms. Trace:
I20240502 08:16:56.491358 24180 rpcz_store.cc:276] 0502 08:16:53.973958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.983305 (+ 9347us) service_pool.cc:225] Handling call
0502 08:16:56.491207 (+2507902us) ts_tablet_manager.cc:1120] Deleting tablet 7a0ff1c3818a49d2b13f57ea6a81c3d6
0502 08:16:56.491215 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.491281 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2507288}
I20240502 08:16:56.491641 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1008) took 2514 ms. Trace:
I20240502 08:16:56.490193 24174 tablet_bootstrap.cc:492] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
W20240502 08:16:56.491906 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet df266081ed8345dda989b3050f605685 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df266081ed8345dda989b3050f605685 already in progress: creating tablet
I20240502 08:16:56.492017 24174 ts_tablet_manager.cc:1367] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
W20240502 08:16:56.492123 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 7a0ff1c3818a49d2b13f57ea6a81c3d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a0ff1c3818a49d2b13f57ea6a81c3d6 already in progress: creating tablet
I20240502 08:16:56.492621 25423 consensus_queue.cc:1035] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:56.493232 24174 raft_consensus.cc:348] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.493438 24174 raft_consensus.cc:374] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.493505 24174 raft_consensus.cc:729] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.493700 24174 consensus_queue.cc:260] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.494414 24174 ts_tablet_manager.cc:1398] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:56.494752 24174 tablet_bootstrap.cc:492] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:56.495899 25531 raft_consensus.cc:482] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.495457 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 895) took 3470 ms. Trace:
I20240502 08:16:56.496032 25531 raft_consensus.cc:504] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.496666 25531 leader_election.cc:288] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:56.496774 24174 tablet_bootstrap.cc:654] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.496237 24181 rpcz_store.cc:276] 0502 08:16:53.025141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.025188 (+ 47us) service_pool.cc:225] Handling call
0502 08:16:56.385729 (+3360541us) ts_tablet_manager.cc:1120] Deleting tablet 77c49da1e0f0435ba241a6a66526e790
0502 08:16:56.385742 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.446673 (+ 60931us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.478724 (+ 32051us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.495310 (+ 16586us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3360329}
I20240502 08:16:56.497546 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b4c563d6bd4a44bb3bab58a3fcdc07" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:56.498172 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 77c49da1e0f0435ba241a6a66526e790 (table default.Table_8a9cdf1f53b0494694035cd549988bf0 [id=0de18f9c7e194e698b4f37603bfd4780]) successfully deleted
W20240502 08:16:56.498433 23723 leader_election.cc:341] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.498661 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b4c563d6bd4a44bb3bab58a3fcdc07" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:56.498845 24181 tablet_replica.cc:330] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
W20240502 08:16:56.499086 23723 leader_election.cc:341] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.499166 23723 leader_election.cc:302] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:56.499497 25531 raft_consensus.cc:2726] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:56.500327 24181 raft_consensus.cc:2229] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.501374 24181 raft_consensus.cc:2258] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.491699 24180 rpcz_store.cc:276] 0502 08:16:53.977244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.979792 (+ 2548us) service_pool.cc:225] Handling call
0502 08:16:56.491553 (+2511761us) ts_tablet_manager.cc:1120] Deleting tablet 518cb4b86ea34e95bef51c5c2a64d89b
0502 08:16:56.491560 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.491627 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13227,"tablet-delete.queue_time_us":2507535}
I20240502 08:16:56.505272 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1006) took 2541 ms. Trace:
W20240502 08:16:56.505432 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 518cb4b86ea34e95bef51c5c2a64d89b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 518cb4b86ea34e95bef51c5c2a64d89b already in progress: creating tablet
I20240502 08:16:56.505360 24180 rpcz_store.cc:276] 0502 08:16:53.964286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.015754 (+ 51468us) service_pool.cc:225] Handling call
0502 08:16:56.505151 (+2489397us) ts_tablet_manager.cc:1120] Deleting tablet 17d6f4f6e9ee46c28b7e7513f791c81f
0502 08:16:56.505162 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.505254 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2457068}
I20240502 08:16:56.505777 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1009) took 2445 ms. Trace:
I20240502 08:16:56.505853 24180 rpcz_store.cc:276] 0502 08:16:54.060624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.106731 (+ 46107us) service_pool.cc:225] Handling call
0502 08:16:56.505662 (+2398931us) ts_tablet_manager.cc:1120] Deleting tablet ac7a7cec2e154d6fa095d68d1ba3308d
0502 08:16:56.505673 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.505760 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2398686}
I20240502 08:16:56.506146 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1010) took 2309 ms. Trace:
I20240502 08:16:56.506215 24180 rpcz_store.cc:276] 0502 08:16:54.196476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.218064 (+ 21588us) service_pool.cc:225] Handling call
0502 08:16:56.506044 (+2287980us) ts_tablet_manager.cc:1120] Deleting tablet e4ac2b5073e34cd98e0246620fc1515d
0502 08:16:56.506053 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.506131 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2287440}
W20240502 08:16:56.506363 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 17d6f4f6e9ee46c28b7e7513f791c81f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17d6f4f6e9ee46c28b7e7513f791c81f already in progress: creating tablet
I20240502 08:16:56.506546 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1018) took 1760 ms. Trace:
W20240502 08:16:56.506624 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet ac7a7cec2e154d6fa095d68d1ba3308d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac7a7cec2e154d6fa095d68d1ba3308d already in progress: creating tablet
I20240502 08:16:56.506615 24180 rpcz_store.cc:276] 0502 08:16:54.745963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.746162 (+ 199us) service_pool.cc:225] Handling call
0502 08:16:56.506447 (+1760285us) ts_tablet_manager.cc:1120] Deleting tablet 31775513f3a446b08c0aac25ae0cab20
0502 08:16:56.506456 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.506531 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1760044}
I20240502 08:16:56.506815 24180 tablet_replica.cc:330] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:56.506980 24180 raft_consensus.cc:2229] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:56.507282 24180 raft_consensus.cc:2258] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.508952 24180 ts_tablet_manager.cc:1875] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:56.509760 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e4ac2b5073e34cd98e0246620fc1515d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4ac2b5073e34cd98e0246620fc1515d already in progress: creating tablet
W20240502 08:16:56.510023 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 31775513f3a446b08c0aac25ae0cab20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31775513f3a446b08c0aac25ae0cab20 already in progress: creating tablet
I20240502 08:16:56.510444 24181 ts_tablet_manager.cc:1875] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.510691 25184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 1 FOLLOWER]: Replica starting. Triggering 4 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:56.515074 25494 raft_consensus.cc:482] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.515202 25494 raft_consensus.cc:504] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.515818 25494 leader_election.cc:288] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:56.516448 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b8d81f071d74b218fcfd4f4e8c20bab" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:56.517439 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b8d81f071d74b218fcfd4f4e8c20bab" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:56.517928 23858 leader_election.cc:302] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:56.518365 25494 raft_consensus.cc:2726] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:56.519356 25531 raft_consensus.cc:482] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.519825 25531 raft_consensus.cc:504] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.521991 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d151770ee44130adfd822518b9b017" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:56.522740 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d151770ee44130adfd822518b9b017" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:56.522437 23723 leader_election.cc:341] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:56.523414 23723 leader_election.cc:341] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.523517 23723 leader_election.cc:302] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:56.523550 24180 ts_tablet_manager.cc:1888] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.523689 24180 log.cc:1196] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/78ddfc7b74644a0bbf8222d5f90e6658
I20240502 08:16:56.522004 25531 leader_election.cc:288] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:56.525136 25296 raft_consensus.cc:2726] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:56.528101 24180 ts_tablet_manager.cc:1909] T 78ddfc7b74644a0bbf8222d5f90e6658 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.529196 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1021) took 1782 ms. Trace:
I20240502 08:16:56.529326 24180 rpcz_store.cc:276] 0502 08:16:54.746224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.746490 (+ 266us) service_pool.cc:225] Handling call
0502 08:16:56.506768 (+1760278us) ts_tablet_manager.cc:1120] Deleting tablet 78ddfc7b74644a0bbf8222d5f90e6658
0502 08:16:56.506775 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.515774 (+ 8999us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.523425 (+ 7651us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.529153 (+ 5728us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1760092}
I20240502 08:16:56.529805 24180 tablet_replica.cc:330] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:56.529927 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 78ddfc7b74644a0bbf8222d5f90e6658 (table default.Table_0b7c860d457f421294ca1d37248e40b5 [id=c06501dd41a640279c769c9ab3926ad5]) successfully deleted
I20240502 08:16:56.530977 24180 raft_consensus.cc:2229] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:56.531541 24180 raft_consensus.cc:2258] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.532172 24174 tablet_bootstrap.cc:492] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:56.532332 24174 ts_tablet_manager.cc:1367] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.003s
I20240502 08:16:56.532518 24180 ts_tablet_manager.cc:1875] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.533133 24174 raft_consensus.cc:348] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.533353 24174 raft_consensus.cc:374] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.533473 24174 raft_consensus.cc:729] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.533757 24174 consensus_queue.cc:260] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.534271 24174 ts_tablet_manager.cc:1398] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:56.534559 24174 tablet_bootstrap.cc:492] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:56.536188 24174 tablet_bootstrap.cc:654] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.540946 25184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f3c07a31414c4f8fb61c44dc9f16e1, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.542326 25184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 534, Last appended: 1.538, Last appended by leader: 538, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:56.543692 24174 tablet_bootstrap.cc:492] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:56.543911 24174 ts_tablet_manager.cc:1367] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20240502 08:16:56.544782 24174 raft_consensus.cc:348] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.545094 24174 raft_consensus.cc:374] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.545215 24174 raft_consensus.cc:729] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.545524 24174 consensus_queue.cc:260] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.546139 24174 ts_tablet_manager.cc:1398] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:56.546491 24174 tablet_bootstrap.cc:492] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:56.547042 24175 tablet_bootstrap.cc:654] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.548049 25296 raft_consensus.cc:482] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.548120 24174 tablet_bootstrap.cc:654] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.548219 25296 raft_consensus.cc:504] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.548808 25296 leader_election.cc:288] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:56.549397 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f1ab141585441db46d3027833932b0" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:56.549505 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f1ab141585441db46d3027833932b0" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:56.550756 23723 leader_election.cc:341] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:56.551007 23723 leader_election.cc:341] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.551221 23723 leader_election.cc:302] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:56.551680 25581 tablet.cc:1722] T 7a19310948e248d1a85479c41c61bda2 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20240502 08:16:56.551832 25296 raft_consensus.cc:2726] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:56.558514 24173 tablet_bootstrap.cc:492] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:56.558692 24173 ts_tablet_manager.cc:1367] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.140s user 0.003s sys 0.002s
I20240502 08:16:56.559376 24173 raft_consensus.cc:348] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.560565 24180 ts_tablet_manager.cc:1888] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.560709 24180 log.cc:1196] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/45f02efa659e457ab57bc32ed3f66bb9
I20240502 08:16:56.561105 24180 ts_tablet_manager.cc:1909] T 45f02efa659e457ab57bc32ed3f66bb9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.562480 25012 ts_tablet_manager.cc:1888] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.566650 24174 tablet_bootstrap.cc:492] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:56.566825 24174 ts_tablet_manager.cc:1367] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.001s
I20240502 08:16:56.567651 24174 raft_consensus.cc:348] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.568058 24174 raft_consensus.cc:374] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.568145 24174 raft_consensus.cc:729] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.568392 24174 consensus_queue.cc:260] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.569027 24174 ts_tablet_manager.cc:1398] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:56.569330 24174 tablet_bootstrap.cc:492] T ec10ff7251a5442a8ecc1881b34db74d P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:56.569949 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1025) took 1823 ms. Trace:
I20240502 08:16:56.570060 24180 rpcz_store.cc:276] 0502 08:16:54.746733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.746771 (+ 38us) service_pool.cc:225] Handling call
0502 08:16:56.529730 (+1782959us) ts_tablet_manager.cc:1120] Deleting tablet 45f02efa659e457ab57bc32ed3f66bb9
0502 08:16:56.529743 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.557800 (+ 28057us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.560460 (+ 2660us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.569877 (+ 9417us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1782799}
I20240502 08:16:56.570420 24180 tablet_replica.cc:330] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:56.570554 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 45f02efa659e457ab57bc32ed3f66bb9 (table default.Table_54b9e5042d19438ab2690d06917ec10c [id=27f4152aaee24a3bad52e0390b8fa649]) successfully deleted
I20240502 08:16:56.570597 24180 raft_consensus.cc:2229] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:56.570897 24180 raft_consensus.cc:2258] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.571421 24174 tablet_bootstrap.cc:654] T ec10ff7251a5442a8ecc1881b34db74d P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.572507 25531 raft_consensus.cc:482] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.572654 25531 raft_consensus.cc:504] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.573230 25531 leader_election.cc:288] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:56.573268 24180 ts_tablet_manager.cc:1875] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.578630 24173 raft_consensus.cc:374] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.578764 24173 raft_consensus.cc:729] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.579145 24173 consensus_queue.cc:260] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.579872 24173 ts_tablet_manager.cc:1398] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20240502 08:16:56.580252 24173 tablet_bootstrap.cc:492] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:56.565188 25582 tablet.cc:1722] T 7a19310948e248d1a85479c41c61bda2 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20240502 08:16:56.573827 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f3f09fc1f94c00a47b9087b98e221d" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:56.582319 23723 leader_election.cc:341] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.584076 24180 ts_tablet_manager.cc:1888] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.584213 24180 log.cc:1196] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/4b8d81f071d74b218fcfd4f4e8c20bab
I20240502 08:16:56.584553 24180 ts_tablet_manager.cc:1909] T 4b8d81f071d74b218fcfd4f4e8c20bab P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.585336 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1024) took 1838 ms. Trace:
I20240502 08:16:56.565423 23922 raft_consensus.cc:3037] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:56.585431 24180 rpcz_store.cc:276] 0502 08:16:54.746645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.746838 (+ 193us) service_pool.cc:225] Handling call
0502 08:16:56.570327 (+1823489us) ts_tablet_manager.cc:1120] Deleting tablet 4b8d81f071d74b218fcfd4f4e8c20bab
0502 08:16:56.570336 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.576547 (+ 6211us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.583985 (+ 7438us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.585317 (+ 1332us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1823362}
I20240502 08:16:56.592800 24180 tablet_replica.cc:330] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:56.592973 24180 raft_consensus.cc:2229] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.592969 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 4b8d81f071d74b218fcfd4f4e8c20bab (table default.Table_dbe385cc3f6b48d39bee298ca9490248 [id=aa825e5ca73c4dafa9585126d5c48df7]) successfully deleted
I20240502 08:16:56.593117 24180 raft_consensus.cc:2258] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.593665 24180 ts_tablet_manager.cc:1875] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.569731 24175 tablet_bootstrap.cc:492] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:56.594839 24175 ts_tablet_manager.cc:1367] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.148s user 0.012s sys 0.013s
I20240502 08:16:56.595518 24175 raft_consensus.cc:348] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.595798 24175 raft_consensus.cc:374] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.595898 24175 raft_consensus.cc:729] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.596180 24175 consensus_queue.cc:260] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.596805 24175 ts_tablet_manager.cc:1398] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:56.596591 24181 ts_tablet_manager.cc:1888] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.597136 24175 tablet_bootstrap.cc:492] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:56.597445 24181 log.cc:1196] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/dd3078cc1f6247609f8bd017ed076b45
I20240502 08:16:56.598701 24175 tablet_bootstrap.cc:654] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.600296 24181 ts_tablet_manager.cc:1909] T dd3078cc1f6247609f8bd017ed076b45 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:56.574550 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f3f09fc1f94c00a47b9087b98e221d" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:56.604462 23723 leader_election.cc:341] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.604579 23723 leader_election.cc:302] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:56.578123 25588 tablet.cc:1722] T 7a19310948e248d1a85479c41c61bda2 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20240502 08:16:56.605918 25012 log.cc:1196] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/c5c377e037894728ac084b6500ad20de
I20240502 08:16:56.606290 25012 ts_tablet_manager.cc:1909] T c5c377e037894728ac084b6500ad20de P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:56.606583 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 882) took 3441 ms. Trace:
I20240502 08:16:56.606659 25012 rpcz_store.cc:276] 0502 08:16:53.165598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.165647 (+ 49us) service_pool.cc:225] Handling call
0502 08:16:56.220437 (+3054790us) ts_tablet_manager.cc:1120] Deleting tablet c5c377e037894728ac084b6500ad20de
0502 08:16:56.220446 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.375062 (+154616us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.562390 (+187328us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.606565 (+ 44175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31742,"tablet-delete.queue_time_us":3054586}
I20240502 08:16:56.607620 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 883) took 3441 ms. Trace:
I20240502 08:16:56.607695 25012 rpcz_store.cc:276] 0502 08:16:53.166111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.166153 (+ 42us) service_pool.cc:225] Handling call
0502 08:16:56.607498 (+3441345us) ts_tablet_manager.cc:1120] Deleting tablet c48cea94856f43b6861c3ba360414888
0502 08:16:56.607508 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.607604 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3441190}
W20240502 08:16:56.608132 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet c48cea94856f43b6861c3ba360414888 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c48cea94856f43b6861c3ba360414888 already in progress: creating tablet
I20240502 08:16:56.608467 25012 tablet_replica.cc:330] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:56.608587 25531 raft_consensus.cc:2726] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:56.608634 25012 raft_consensus.cc:2229] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.608774 25012 raft_consensus.cc:2258] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.609002 25531 raft_consensus.cc:482] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.609113 25531 raft_consensus.cc:504] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.609321 25012 ts_tablet_manager.cc:1875] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.608587 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 896) took 3582 ms. Trace:
I20240502 08:16:56.610714 25531 leader_election.cc:288] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:56.609757 24181 rpcz_store.cc:276] 0502 08:16:53.025537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.025632 (+ 95us) service_pool.cc:225] Handling call
0502 08:16:56.498464 (+3472832us) ts_tablet_manager.cc:1120] Deleting tablet dd3078cc1f6247609f8bd017ed076b45
0502 08:16:56.498474 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.563115 (+ 64641us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.595938 (+ 32823us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.608365 (+ 12427us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3472549}
I20240502 08:16:56.611724 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet dd3078cc1f6247609f8bd017ed076b45 (table default.Table_bac7f229ce9541b39c0a70951431b689 [id=f9f2bf985f124269982869050bbecd70]) successfully deleted
I20240502 08:16:56.612392 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592ab8523b8d479d9caead321d5888c3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:56.612656 24051 raft_consensus.cc:2454] T 592ab8523b8d479d9caead321d5888c3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:56.612708 24180 ts_tablet_manager.cc:1888] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:56.612818 24180 log.cc:1196] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/ee022e8e86f8477a8fa1e58d52644b03
I20240502 08:16:56.613107 24180 ts_tablet_manager.cc:1909] T ee022e8e86f8477a8fa1e58d52644b03 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.613134 23723 leader_election.cc:302] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:56.613617 25531 raft_consensus.cc:2781] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:56.613690 25531 raft_consensus.cc:482] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:56.613745 25531 raft_consensus.cc:3037] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:56.613943 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 894) took 3588 ms. Trace:
I20240502 08:16:56.614248 24181 rpcz_store.cc:276] 0502 08:16:53.025047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.025807 (+ 760us) service_pool.cc:225] Handling call
0502 08:16:56.612054 (+3586247us) ts_tablet_manager.cc:1120] Deleting tablet 8075b03643a14a01ab75921fc3d3dace
0502 08:16:56.612065 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.613720 (+ 1655us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3585847}
I20240502 08:16:56.616151 25531 raft_consensus.cc:504] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.616003 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 898) took 3585 ms. Trace:
I20240502 08:16:56.616744 25531 leader_election.cc:288] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:56.617188 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 8075b03643a14a01ab75921fc3d3dace because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8075b03643a14a01ab75921fc3d3dace already in progress: creating tablet
I20240502 08:16:56.616434 24181 rpcz_store.cc:276] 0502 08:16:53.030970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.031013 (+ 43us) service_pool.cc:225] Handling call
0502 08:16:56.615380 (+3584367us) ts_tablet_manager.cc:1120] Deleting tablet 07344b78894c42809391d0fe2a9cd8f7
0502 08:16:56.615495 (+ 115us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.615980 (+ 485us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3584238}
W20240502 08:16:56.617769 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 07344b78894c42809391d0fe2a9cd8f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07344b78894c42809391d0fe2a9cd8f7 already in progress: creating tablet
I20240502 08:16:56.618247 24181 tablet_replica.cc:330] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:56.618494 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592ab8523b8d479d9caead321d5888c3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:56.618716 23920 raft_consensus.cc:3037] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:56.618677 25531 consensus_queue.cc:1035] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:56.619611 24181 raft_consensus.cc:2229] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.620252 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592ab8523b8d479d9caead321d5888c3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:56.620571 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1023) took 1874 ms. Trace:
I20240502 08:16:56.620491 24181 raft_consensus.cc:2258] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.620651 24180 rpcz_store.cc:276] 0502 08:16:54.746542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.746763 (+ 221us) service_pool.cc:225] Handling call
0502 08:16:56.592749 (+1845986us) ts_tablet_manager.cc:1120] Deleting tablet ee022e8e86f8477a8fa1e58d52644b03
0502 08:16:56.592760 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.596517 (+ 3757us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.612608 (+ 16091us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.620554 (+ 7946us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6678,"tablet-delete.queue_time_us":1845685}
I20240502 08:16:56.620949 24180 tablet_replica.cc:330] T 6e7432f1df94431a83cf68356e3e78a3 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:56.620468 24053 raft_consensus.cc:3037] T 592ab8523b8d479d9caead321d5888c3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:56.621122 24180 raft_consensus.cc:2229] T 6e7432f1df94431a83cf68356e3e78a3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.621177 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet ee022e8e86f8477a8fa1e58d52644b03 (table default.Table_d6f5ca82488a40a69121f1b779d8f874 [id=0422356f66a74ad29c2e13e1ad36b654]) successfully deleted
I20240502 08:16:56.621294 24180 raft_consensus.cc:2258] T 6e7432f1df94431a83cf68356e3e78a3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.621824 24180 ts_tablet_manager.cc:1875] T 6e7432f1df94431a83cf68356e3e78a3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.622651 24181 ts_tablet_manager.cc:1875] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.624931 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592ab8523b8d479d9caead321d5888c3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:56.627884 25584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } } }
W20240502 08:16:56.635654 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d2f7ccb4385d4c3cad54758d765e8452 consensus state has no leader
I20240502 08:16:56.635818 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_79bb10e705c047138e1c4363673a65e6" } modify_external_catalogs: true
I20240502 08:16:56.635908 23603 master_service.cc:946] ReplaceTablet: received request to replace tablet edd7244b9f5741a8aa1c6835fe544676 from {username='slave'} at 127.0.0.1:36668
W20240502 08:16:56.636195 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 598cba0f8b9f4c8dae9e34872c4212fa consensus state has no leader
I20240502 08:16:56.636510 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8e4b36ac2e7b41559cb0941df14e55a1" } modify_external_catalogs: true
I20240502 08:16:56.636685 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8e4b36ac2e7b41559cb0941df14e55a1" } modify_external_catalogs: true
I20240502 08:16:56.637207 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 548ab3f9d0d84527b1092ae715ec2bdd
I20240502 08:16:56.638037 23610 master_service.cc:946] ReplaceTablet: received request to replace tablet 96013fa1fa014b1891eb621cc3a81cdc from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:56.639288 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_79bb10e705c047138e1c4363673a65e6" } modify_external_catalogs: true
I20240502 08:16:56.641021 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet 9467974d67c64c7eb5e488ac51f1240d from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:56.645092 24175 tablet_bootstrap.cc:492] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:56.629479 25184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:56.629827 25113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" candidate_term: 2 candidate_status { last_received { term: 1 index: 1056 } } ignore_live_leader: false dest_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" is_pre_election: true
W20240502 08:16:56.645910 25113 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38224 (request call id 1) took 4586 ms (client timeout 746 ms). Trace:
W20240502 08:16:56.646080 25113 rpcz_store.cc:269] 0502 08:16:52.059212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:52.059295 (+ 83us) service_pool.cc:225] Handling call
0502 08:16:56.645845 (+4586550us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:56.637648 25115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" candidate_term: 2 candidate_status { last_received { term: 1 index: 1056 } } ignore_live_leader: false dest_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" is_pre_election: true
W20240502 08:16:56.647017 25115 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38224 (request call id 0) took 5143 ms (client timeout 746 ms). Trace:
W20240502 08:16:56.647166 25115 rpcz_store.cc:269] 0502 08:16:51.503755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.503815 (+ 60us) service_pool.cc:225] Handling call
0502 08:16:56.646990 (+5143175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 08:16:56.647536 24175 ts_tablet_manager.cc:1367] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.000s
I20240502 08:16:56.647656 24180 ts_tablet_manager.cc:1888] T 6e7432f1df94431a83cf68356e3e78a3 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.647774 24180 log.cc:1196] T 6e7432f1df94431a83cf68356e3e78a3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/6e7432f1df94431a83cf68356e3e78a3
I20240502 08:16:56.648164 24180 ts_tablet_manager.cc:1909] T 6e7432f1df94431a83cf68356e3e78a3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.648566 24175 raft_consensus.cc:348] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.648891 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5889de778d2842aba3b3a02fb663425a" } modify_external_catalogs: true
I20240502 08:16:56.649013 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1022) took 1902 ms. Trace:
I20240502 08:16:56.649113 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5889de778d2842aba3b3a02fb663425a" } modify_external_catalogs: true
I20240502 08:16:56.649143 24180 rpcz_store.cc:276] 0502 08:16:54.746425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.747082 (+ 657us) service_pool.cc:225] Handling call
0502 08:16:56.620908 (+1873826us) ts_tablet_manager.cc:1120] Deleting tablet 6e7432f1df94431a83cf68356e3e78a3
0502 08:16:56.620917 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.624548 (+ 3631us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.647538 (+ 22990us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.648968 (+ 1430us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1873674}
I20240502 08:16:56.649511 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1019) took 1903 ms. Trace:
I20240502 08:16:56.649613 24180 rpcz_store.cc:276] 0502 08:16:54.746048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.749466 (+ 3418us) service_pool.cc:225] Handling call
0502 08:16:56.649386 (+1899920us) ts_tablet_manager.cc:1120] Deleting tablet 37b499184f31454c96f66cfdf9143d14
0502 08:16:56.649394 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.649479 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1899751}
I20240502 08:16:56.649904 24180 tablet_replica.cc:330] T 5d04312bd81f46ed8c49c42b1af1aec9 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:56.649926 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 6e7432f1df94431a83cf68356e3e78a3 (table default.Table_a89a2cdb74684df5bcbe15389816c50b [id=04819651040e46c580d553290e3fa250]) successfully deleted
W20240502 08:16:56.650468 23993 consensus_peers.cc:485] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33c8131846dc41678827b86e87be18d3. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:56.650493 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 548ab3f9d0d84527b1092ae715ec2bdd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:56.650540 24180 raft_consensus.cc:2229] T 5d04312bd81f46ed8c49c42b1af1aec9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.650691 24180 raft_consensus.cc:2258] T 5d04312bd81f46ed8c49c42b1af1aec9 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.651005 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_38fd1abca8cf4743bee0a0cb23748c90" } new_table_name: "default.Table_34217583459a498eaa704feafc2a5939" modify_external_catalogs: true
I20240502 08:16:56.651113 24175 raft_consensus.cc:374] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.651178 24175 raft_consensus.cc:729] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.651345 24180 ts_tablet_manager.cc:1875] T 5d04312bd81f46ed8c49c42b1af1aec9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.651389 24175 consensus_queue.cc:260] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.651846 24175 ts_tablet_manager.cc:1398] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:16:56.652094 24175 tablet_bootstrap.cc:492] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:56.653465 24175 tablet_bootstrap.cc:654] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:56.653811 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 37b499184f31454c96f66cfdf9143d14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37b499184f31454c96f66cfdf9143d14 already in progress: creating tablet
I20240502 08:16:56.655197 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet 548ab3f9d0d84527b1092ae715ec2bdd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:56.656044 25584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:56.657222 24180 ts_tablet_manager.cc:1888] T 5d04312bd81f46ed8c49c42b1af1aec9 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.657352 24180 log.cc:1196] T 5d04312bd81f46ed8c49c42b1af1aec9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/5d04312bd81f46ed8c49c42b1af1aec9
I20240502 08:16:56.657733 24180 ts_tablet_manager.cc:1909] T 5d04312bd81f46ed8c49c42b1af1aec9 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.658427 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1020) took 1912 ms. Trace:
I20240502 08:16:56.658519 24180 rpcz_store.cc:276] 0502 08:16:54.746140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.749660 (+ 3520us) service_pool.cc:225] Handling call
0502 08:16:56.649790 (+1900130us) ts_tablet_manager.cc:1120] Deleting tablet 5d04312bd81f46ed8c49c42b1af1aec9
0502 08:16:56.649854 (+ 64us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.654182 (+ 4328us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.657118 (+ 2936us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.658360 (+ 1242us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1900006}
I20240502 08:16:56.658910 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1032) took 1880 ms. Trace:
I20240502 08:16:56.658973 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 5d04312bd81f46ed8c49c42b1af1aec9 (table default.Table_9741459a96e649fb8151cc28b642523e [id=2b9169f285754bfda0f4f8dc7f5cb653]) successfully deleted
I20240502 08:16:56.658996 24180 rpcz_store.cc:276] 0502 08:16:54.778439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.779021 (+ 582us) service_pool.cc:225] Handling call
0502 08:16:56.658769 (+1879748us) ts_tablet_manager.cc:1120] Deleting tablet 870edf582ab14ad1b57c4c4469b19954
0502 08:16:56.658778 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.658859 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1879585}
I20240502 08:16:56.659241 24180 tablet_replica.cc:330] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
W20240502 08:16:56.659495 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 870edf582ab14ad1b57c4c4469b19954 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 870edf582ab14ad1b57c4c4469b19954 already in progress: creating tablet
I20240502 08:16:56.654673 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet 548ab3f9d0d84527b1092ae715ec2bdd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:56.668273 23434 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 46745.7us
I20240502 08:16:56.668498 24180 raft_consensus.cc:2229] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:56.668995 24180 raft_consensus.cc:2258] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.669826 23920 raft_consensus.cc:2454] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:56.670248 23723 leader_election.cc:302] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:56.670607 25531 raft_consensus.cc:2781] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:56.670725 25531 raft_consensus.cc:686] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:56.670948 25531 consensus_queue.cc:237] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.671085 24180 ts_tablet_manager.cc:1875] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.674674 24173 tablet_bootstrap.cc:654] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.678635 24180 ts_tablet_manager.cc:1888] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.679615 24053 raft_consensus.cc:2454] T 592ab8523b8d479d9caead321d5888c3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:56.681859 24180 log.cc:1196] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/b59c2f93d6514c9cbf358ab96f1a06b5
I20240502 08:16:56.682446 24180 ts_tablet_manager.cc:1909] T b59c2f93d6514c9cbf358ab96f1a06b5 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.683199 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1017) took 1937 ms. Trace:
I20240502 08:16:56.684619 24180 rpcz_store.cc:276] 0502 08:16:54.745871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.874345 (+128474us) service_pool.cc:225] Handling call
0502 08:16:56.659170 (+1784825us) ts_tablet_manager.cc:1120] Deleting tablet b59c2f93d6514c9cbf358ab96f1a06b5
0502 08:16:56.659176 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.675919 (+ 16743us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.678552 (+ 2633us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.683183 (+ 4631us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1784643}
I20240502 08:16:56.685070 24180 tablet_replica.cc:330] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:56.683595 25531 consensus_queue.cc:1035] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:56.685221 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet b59c2f93d6514c9cbf358ab96f1a06b5 (table default.Table_bbf9d7694fe942de8aa8bd4ed61336a5 [id=f1d6eb661b0143ee82e797cde61391c0]) successfully deleted
I20240502 08:16:56.687479 24181 ts_tablet_manager.cc:1888] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.687611 24181 log.cc:1196] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/45f02efa659e457ab57bc32ed3f66bb9
I20240502 08:16:56.687997 24181 ts_tablet_manager.cc:1909] T 45f02efa659e457ab57bc32ed3f66bb9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:56.688917 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 897) took 3658 ms. Trace:
I20240502 08:16:56.689041 24181 rpcz_store.cc:276] 0502 08:16:53.030871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.031181 (+ 310us) service_pool.cc:225] Handling call
0502 08:16:56.617941 (+3586760us) ts_tablet_manager.cc:1120] Deleting tablet 45f02efa659e457ab57bc32ed3f66bb9
0502 08:16:56.617955 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.680099 (+ 62144us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.687360 (+ 7261us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.688894 (+ 1534us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3586506}
I20240502 08:16:56.689401 24181 tablet_replica.cc:330] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:56.689574 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 45f02efa659e457ab57bc32ed3f66bb9 (table default.Table_54b9e5042d19438ab2690d06917ec10c [id=27f4152aaee24a3bad52e0390b8fa649]) successfully deleted
I20240502 08:16:56.689656 24181 raft_consensus.cc:2229] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:56.689755 24180 raft_consensus.cc:2229] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.689653 25413 raft_consensus.cc:482] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.689882 24180 raft_consensus.cc:2258] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.689947 24181 raft_consensus.cc:2258] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.689888 25413 raft_consensus.cc:504] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.690363 24180 ts_tablet_manager.cc:1875] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.690363 24181 ts_tablet_manager.cc:1875] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.690907 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a19bb8705c45c0bfc537d3a6bf8ae2" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:56.691234 25413 leader_election.cc:288] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:56.691491 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a19bb8705c45c0bfc537d3a6bf8ae2" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:56.691792 23920 raft_consensus.cc:2379] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 149ce744feab49b5aa5bcaa14bd6aba5 in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:16:56.692216 23996 leader_election.cc:302] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:56.692968 25413 raft_consensus.cc:3037] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:56.695806 25413 raft_consensus.cc:2726] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:56.696619 24180 ts_tablet_manager.cc:1888] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:56.696727 24180 log.cc:1196] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/e47459b5f5bb4ecb96f76240fefa1f5d
I20240502 08:16:56.697031 24180 ts_tablet_manager.cc:1909] T e47459b5f5bb4ecb96f76240fefa1f5d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.698177 24175 tablet_bootstrap.cc:492] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:56.698364 24175 ts_tablet_manager.cc:1367] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.046s user 0.007s sys 0.003s
I20240502 08:16:56.699136 24175 raft_consensus.cc:348] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.703529 24175 raft_consensus.cc:374] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.703704 24175 raft_consensus.cc:729] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.704146 24175 consensus_queue.cc:260] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.705454 24175 ts_tablet_manager.cc:1398] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 08:16:56.706164 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1035) took 1841 ms. Trace:
I20240502 08:16:56.706328 25413 raft_consensus.cc:482] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.706266 24180 rpcz_store.cc:276] 0502 08:16:54.864710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.894924 (+ 30214us) service_pool.cc:225] Handling call
0502 08:16:56.685024 (+1790100us) ts_tablet_manager.cc:1120] Deleting tablet e47459b5f5bb4ecb96f76240fefa1f5d
0502 08:16:56.685033 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.693903 (+ 8870us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.696541 (+ 2638us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.706139 (+ 9598us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4428,"tablet-delete.queue_time_us":1789879}
I20240502 08:16:56.706470 25413 raft_consensus.cc:504] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.706796 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1015) took 1961 ms. Trace:
I20240502 08:16:56.706899 24180 rpcz_store.cc:276] 0502 08:16:54.745642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.899479 (+153837us) service_pool.cc:225] Handling call
0502 08:16:56.706640 (+1807161us) ts_tablet_manager.cc:1120] Deleting tablet bb9b75d6be924d0291f94fc4a439dea6
0502 08:16:56.706654 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.706774 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1806984}
I20240502 08:16:56.707541 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet e47459b5f5bb4ecb96f76240fefa1f5d (table default.Table_922974ba60ab43e5af71756d9209fd0d [id=51cd52edc7ae46738d9ffa2bd8083fdb]) successfully deleted
W20240502 08:16:56.707872 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet bb9b75d6be924d0291f94fc4a439dea6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb9b75d6be924d0291f94fc4a439dea6 already in progress: creating tablet
W20240502 08:16:56.708418 23993 leader_election.cc:341] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: e47459b5f5bb4ecb96f76240fefa1f5d
I20240502 08:16:56.709419 25413 leader_election.cc:288] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:56.709805 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1016) took 1964 ms. Trace:
I20240502 08:16:56.709897 24180 rpcz_store.cc:276] 0502 08:16:54.745776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.899835 (+154059us) service_pool.cc:225] Handling call
0502 08:16:56.707085 (+1807250us) ts_tablet_manager.cc:1120] Deleting tablet 2c28c47b4f814573971227b2208d1f83
0502 08:16:56.707093 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.709787 (+ 2694us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1807118}
I20240502 08:16:56.710218 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1036) took 1601 ms. Trace:
I20240502 08:16:56.710291 24180 rpcz_store.cc:276] 0502 08:16:55.108939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.108987 (+ 48us) service_pool.cc:225] Handling call
0502 08:16:56.710124 (+1601137us) ts_tablet_manager.cc:1120] Deleting tablet e9941681b26d4c97b06a91155043b040
0502 08:16:56.710132 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.710203 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1600965}
I20240502 08:16:56.710510 24180 tablet_replica.cc:330] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
W20240502 08:16:56.710495 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 2c28c47b4f814573971227b2208d1f83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c28c47b4f814573971227b2208d1f83 already in progress: creating tablet
W20240502 08:16:56.711092 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e9941681b26d4c97b06a91155043b040 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9941681b26d4c97b06a91155043b040 already in progress: creating tablet
I20240502 08:16:56.710680 24180 raft_consensus.cc:2229] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.713972 24180 raft_consensus.cc:2258] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.709219 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47459b5f5bb4ecb96f76240fefa1f5d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:56.716861 23996 leader_election.cc:341] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.716966 23996 leader_election.cc:302] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:56.717092 24174 tablet_bootstrap.cc:492] T ec10ff7251a5442a8ecc1881b34db74d P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:56.717206 24174 ts_tablet_manager.cc:1367] T ec10ff7251a5442a8ecc1881b34db74d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.148s user 0.004s sys 0.015s
I20240502 08:16:56.717447 25413 raft_consensus.cc:2726] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:56.717492 24180 ts_tablet_manager.cc:1875] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.717917 24175 tablet_bootstrap.cc:492] T a2da8b9d3f374d8a812336057c0ce7d2 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:56.717900 24174 raft_consensus.cc:348] T ec10ff7251a5442a8ecc1881b34db74d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.718065 24174 raft_consensus.cc:374] T ec10ff7251a5442a8ecc1881b34db74d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.718111 24174 raft_consensus.cc:729] T ec10ff7251a5442a8ecc1881b34db74d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.718284 24174 consensus_queue.cc:260] T ec10ff7251a5442a8ecc1881b34db74d P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.718708 24174 ts_tablet_manager.cc:1398] T ec10ff7251a5442a8ecc1881b34db74d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:16:56.718931 24174 tablet_bootstrap.cc:492] T 4c395a29c7a1485eab39ca107dd91fca P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:56.720009 24174 tablet_bootstrap.cc:654] T 4c395a29c7a1485eab39ca107dd91fca P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.720196 24181 ts_tablet_manager.cc:1888] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.721649 24181 log.cc:1196] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/1cd8338500f54604abcd6d42702c2eca
I20240502 08:16:56.722774 24180 ts_tablet_manager.cc:1888] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:56.722858 24180 log.cc:1196] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/89f57e3a39d54b7ab372d4b145b7a696
I20240502 08:16:56.723145 24180 ts_tablet_manager.cc:1909] T 89f57e3a39d54b7ab372d4b145b7a696 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.723421 24181 ts_tablet_manager.cc:1909] T 1cd8338500f54604abcd6d42702c2eca P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:56.723661 24175 tablet_bootstrap.cc:654] T a2da8b9d3f374d8a812336057c0ce7d2 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.723852 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1037) took 1607 ms. Trace:
I20240502 08:16:56.723946 24180 rpcz_store.cc:276] 0502 08:16:55.115986 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.116048 (+ 62us) service_pool.cc:225] Handling call
0502 08:16:56.710472 (+1594424us) ts_tablet_manager.cc:1120] Deleting tablet 89f57e3a39d54b7ab372d4b145b7a696
0502 08:16:56.710479 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.720199 (+ 9720us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.722695 (+ 2496us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.723832 (+ 1137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1594205}
I20240502 08:16:56.724256 24180 tablet_replica.cc:330] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:56.724439 24180 raft_consensus.cc:2229] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:56.724584 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 89f57e3a39d54b7ab372d4b145b7a696 (table default.Table_28b3620523074c23866e772b117bf0ee [id=b09ad60d787b4740a493cb73c22b94a9]) successfully deleted
I20240502 08:16:56.724704 24180 raft_consensus.cc:2258] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.726706 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 899) took 3657 ms. Trace:
I20240502 08:16:56.726838 24181 rpcz_store.cc:276] 0502 08:16:53.069013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.069089 (+ 76us) service_pool.cc:225] Handling call
0502 08:16:56.689351 (+3620262us) ts_tablet_manager.cc:1120] Deleting tablet 1cd8338500f54604abcd6d42702c2eca
0502 08:16:56.689360 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.694868 (+ 5508us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.697532 (+ 2664us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.726679 (+ 29147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3620081}
I20240502 08:16:56.732256 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 900) took 3663 ms. Trace:
I20240502 08:16:56.732394 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 1cd8338500f54604abcd6d42702c2eca (table default.Table_482a1b0b7a064e02805ac3b86fc62f99 [id=3c1d614616374739aeb928e797479df4]) successfully deleted
I20240502 08:16:56.732347 24181 rpcz_store.cc:276] 0502 08:16:53.069112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.070222 (+ 1110us) service_pool.cc:225] Handling call
0502 08:16:56.732100 (+3661878us) ts_tablet_manager.cc:1120] Deleting tablet 2ebf55c8024c483f9ef0afe6945a02ff
0502 08:16:56.732113 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.732237 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3661699}
I20240502 08:16:56.732731 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 901) took 3654 ms. Trace:
W20240502 08:16:56.732858 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 2ebf55c8024c483f9ef0afe6945a02ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ebf55c8024c483f9ef0afe6945a02ff already in progress: creating tablet
I20240502 08:16:56.734102 24180 ts_tablet_manager.cc:1875] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.732806 24181 rpcz_store.cc:276] 0502 08:16:53.078772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.078824 (+ 52us) service_pool.cc:225] Handling call
0502 08:16:56.732632 (+3653808us) ts_tablet_manager.cc:1120] Deleting tablet 33bbebbca7464ea6ad59b81d7b7c560d
0502 08:16:56.732641 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.732717 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3653610}
I20240502 08:16:56.738224 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 906) took 3529 ms. Trace:
I20240502 08:16:56.738303 24181 rpcz_store.cc:276] 0502 08:16:53.208868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.208955 (+ 87us) service_pool.cc:225] Handling call
0502 08:16:56.738124 (+3529169us) ts_tablet_manager.cc:1120] Deleting tablet 418ab502915c4e4ea9998dd2d0180c48
0502 08:16:56.738132 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.738210 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3528971}
I20240502 08:16:56.738559 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 915) took 3417 ms. Trace:
I20240502 08:16:56.738626 24181 rpcz_store.cc:276] 0502 08:16:53.320964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.321017 (+ 53us) service_pool.cc:225] Handling call
0502 08:16:56.738483 (+3417466us) ts_tablet_manager.cc:1120] Deleting tablet 40fab016ed4f489aa3ccdeafcfbe1477
0502 08:16:56.738489 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.738547 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3417290}
W20240502 08:16:56.738837 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 33bbebbca7464ea6ad59b81d7b7c560d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33bbebbca7464ea6ad59b81d7b7c560d already in progress: creating tablet
I20240502 08:16:56.738878 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 916) took 3416 ms. Trace:
W20240502 08:16:56.739192 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 40fab016ed4f489aa3ccdeafcfbe1477 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40fab016ed4f489aa3ccdeafcfbe1477 already in progress: creating tablet
I20240502 08:16:56.739773 24180 ts_tablet_manager.cc:1888] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.739874 24180 log.cc:1196] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/c00f29fb76b44d6ea4f76963d2d0751c
I20240502 08:16:56.740228 24180 ts_tablet_manager.cc:1909] T c00f29fb76b44d6ea4f76963d2d0751c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.741010 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1038) took 1623 ms. Trace:
I20240502 08:16:56.741111 24180 rpcz_store.cc:276] 0502 08:16:55.117125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.117176 (+ 51us) service_pool.cc:225] Handling call
0502 08:16:56.724211 (+1607035us) ts_tablet_manager.cc:1120] Deleting tablet c00f29fb76b44d6ea4f76963d2d0751c
0502 08:16:56.724221 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.737121 (+ 12900us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.739689 (+ 2568us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.740978 (+ 1289us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1606844}
I20240502 08:16:56.741417 24180 tablet_replica.cc:330] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:56.741588 24180 raft_consensus.cc:2229] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.741741 24180 raft_consensus.cc:2258] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.742215 24180 ts_tablet_manager.cc:1875] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.743682 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet c00f29fb76b44d6ea4f76963d2d0751c (table default.Table_c2a25d0d51cd40769e5045ff80c9c3fc [id=a1004106109d47e7a1f9c6337480d3c0]) successfully deleted
I20240502 08:16:56.738934 24181 rpcz_store.cc:276] 0502 08:16:53.321945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.321997 (+ 52us) service_pool.cc:225] Handling call
0502 08:16:56.738800 (+3416803us) ts_tablet_manager.cc:1120] Deleting tablet df87b8b1ff504031bf007880792192dd
0502 08:16:56.738807 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.738867 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3416650}
I20240502 08:16:56.743935 24181 tablet_replica.cc:330] T 1ba1baa2279045b3adc58d187a12cbf3 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:56.744100 24181 raft_consensus.cc:2229] T 1ba1baa2279045b3adc58d187a12cbf3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.744235 24181 raft_consensus.cc:2258] T 1ba1baa2279045b3adc58d187a12cbf3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.747308 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 96013fa1fa014b1891eb621cc3a81cdc
I20240502 08:16:56.747699 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet edd7244b9f5741a8aa1c6835fe544676
I20240502 08:16:56.747917 23610 catalog_manager.cc:6554] ReplaceTablet: tablet 96013fa1fa014b1891eb621cc3a81cdc deleted and replaced by tablet 7ddf9a1299944755bee1cf0178c6edf8
I20240502 08:16:56.748128 23603 catalog_manager.cc:6554] ReplaceTablet: tablet edd7244b9f5741a8aa1c6835fe544676 deleted and replaced by tablet 9eaf8d65f0b4472eb8ed0ef6fc722577
I20240502 08:16:56.748342 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_456a3cba3b8e4f09b405cd14e3785a23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:56.748464 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7dd4773d221543cb990a5a6067ef2db1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:56.748987 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_456a3cba3b8e4f09b405cd14e3785a23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:56.749246 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 92b830e128ac439eba286f29dbb20ee4
I20240502 08:16:56.749334 24173 tablet_bootstrap.cc:492] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:56.749459 24173 ts_tablet_manager.cc:1367] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.169s user 0.000s sys 0.004s
I20240502 08:16:56.749814 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c4e6c4c8a4d54368aa8547df0627bfee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:56.750243 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4e6c4c8a4d54368aa8547df0627bfee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:56.750221 24173 raft_consensus.cc:348] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.750418 24173 raft_consensus.cc:374] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.750470 24173 raft_consensus.cc:729] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.750674 24173 consensus_queue.cc:260] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.744727 24181 ts_tablet_manager.cc:1875] T 1ba1baa2279045b3adc58d187a12cbf3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.751256 24173 ts_tablet_manager.cc:1398] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:56.751286 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e5de31a3db014615989d9e018521bfc9
I20240502 08:16:56.751534 24173 tablet_bootstrap.cc:492] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:56.751895 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8d08b791cfc34eef87f84aff3275577d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:56.752256 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d08b791cfc34eef87f84aff3275577d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:56.753024 24173 tablet_bootstrap.cc:654] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:56.748989 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7dd4773d221543cb990a5a6067ef2db1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:56.754262 24174 tablet_bootstrap.cc:492] T 4c395a29c7a1485eab39ca107dd91fca P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:56.754384 24174 ts_tablet_manager.cc:1367] T 4c395a29c7a1485eab39ca107dd91fca P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.003s
I20240502 08:16:56.755034 24174 raft_consensus.cc:348] T 4c395a29c7a1485eab39ca107dd91fca P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.755203 24174 raft_consensus.cc:374] T 4c395a29c7a1485eab39ca107dd91fca P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.755254 24174 raft_consensus.cc:729] T 4c395a29c7a1485eab39ca107dd91fca P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.755424 24174 consensus_queue.cc:260] T 4c395a29c7a1485eab39ca107dd91fca P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.755687 24175 tablet_bootstrap.cc:492] T a2da8b9d3f374d8a812336057c0ce7d2 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:56.755805 24175 ts_tablet_manager.cc:1367] T a2da8b9d3f374d8a812336057c0ce7d2 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.001s
I20240502 08:16:56.755905 25531 raft_consensus.cc:482] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.756004 25531 raft_consensus.cc:504] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:16:56.749821 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1be95e0576c143e5a94c7aaf8d3e0625 consensus state has no leader
I20240502 08:16:56.756455 24181 ts_tablet_manager.cc:1888] T 1ba1baa2279045b3adc58d187a12cbf3 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.756558 24181 log.cc:1196] T 1ba1baa2279045b3adc58d187a12cbf3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/1ba1baa2279045b3adc58d187a12cbf3
I20240502 08:16:56.756564 25531 leader_election.cc:288] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:56.756506 24175 raft_consensus.cc:348] T a2da8b9d3f374d8a812336057c0ce7d2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.756703 24175 raft_consensus.cc:374] T a2da8b9d3f374d8a812336057c0ce7d2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.756759 24175 raft_consensus.cc:729] T a2da8b9d3f374d8a812336057c0ce7d2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.756675 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_dfb2d9793c694bc1845e757adebff4e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:56.756908 24181 ts_tablet_manager.cc:1909] T 1ba1baa2279045b3adc58d187a12cbf3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:56.756978 24175 consensus_queue.cc:260] T a2da8b9d3f374d8a812336057c0ce7d2 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:56.757133 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfb2d9793c694bc1845e757adebff4e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:56.757297 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63afd6e4fee24edeaba1209105f1c3e6" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:56.757426 24175 ts_tablet_manager.cc:1398] T a2da8b9d3f374d8a812336057c0ce7d2 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:16:56.757588 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 922) took 3315 ms. Trace:
I20240502 08:16:56.757650 24181 rpcz_store.cc:276] 0502 08:16:53.442416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.442466 (+ 50us) service_pool.cc:225] Handling call
0502 08:16:56.743895 (+3301429us) ts_tablet_manager.cc:1120] Deleting tablet 1ba1baa2279045b3adc58d187a12cbf3
0502 08:16:56.743903 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.753821 (+ 9918us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.756369 (+ 2548us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.757546 (+ 1177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3301212}
I20240502 08:16:56.757922 24181 tablet_replica.cc:330] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
W20240502 08:16:56.757908 23723 leader_election.cc:341] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.758069 24181 raft_consensus.cc:2229] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:56.758318 24181 raft_consensus.cc:2258] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.758327 24174 ts_tablet_manager.cc:1398] T 4c395a29c7a1485eab39ca107dd91fca P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:16:56.758571 24174 tablet_bootstrap.cc:492] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:56.758754 24181 ts_tablet_manager.cc:1875] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.758855 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 96013fa1fa014b1891eb621cc3a81cdc with delete_type TABLET_DATA_DELETED (replaced by tablet 7ddf9a1299944755bee1cf0178c6edf8) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:56.759054 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet edd7244b9f5741a8aa1c6835fe544676 with delete_type TABLET_DATA_DELETED (replaced by tablet 9eaf8d65f0b4472eb8ed0ef6fc722577) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:56.759217 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b830e128ac439eba286f29dbb20ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:56.759354 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet e5de31a3db014615989d9e018521bfc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:56.759871 24175 tablet_bootstrap.cc:492] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:56.761147 24175 tablet_bootstrap.cc:654] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:56.762619 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet df87b8b1ff504031bf007880792192dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df87b8b1ff504031bf007880792192dd already in progress: creating tablet
I20240502 08:16:56.762826 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 1ba1baa2279045b3adc58d187a12cbf3 (table default.Table_1704983c4496461d9fa8ea54781f5cb1 [id=1dbff884223d4ad9843fe2512d1f295c]) successfully deleted
I20240502 08:16:56.763513 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet 96013fa1fa014b1891eb621cc3a81cdc with delete_type TABLET_DATA_DELETED (replaced by tablet 7ddf9a1299944755bee1cf0178c6edf8) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:56.763674 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet edd7244b9f5741a8aa1c6835fe544676 with delete_type TABLET_DATA_DELETED (replaced by tablet 9eaf8d65f0b4472eb8ed0ef6fc722577) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:56.763783 23888 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b830e128ac439eba286f29dbb20ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:56.763907 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet e5de31a3db014615989d9e018521bfc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:56.766806 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet 96013fa1fa014b1891eb621cc3a81cdc with delete_type TABLET_DATA_DELETED (replaced by tablet 7ddf9a1299944755bee1cf0178c6edf8) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:56.767000 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet edd7244b9f5741a8aa1c6835fe544676 with delete_type TABLET_DATA_DELETED (replaced by tablet 9eaf8d65f0b4472eb8ed0ef6fc722577) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:56.767175 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b830e128ac439eba286f29dbb20ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:56.767320 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet e5de31a3db014615989d9e018521bfc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:56.769656 23972 heartbeater.cc:347] Connected to a master server at 127.22.225.125:36189
I20240502 08:16:56.769773 23576 master_runner.cc:186] Error getting master registration for 127.22.225.125:36189: OK, instance_id { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" instance_seqno: 1714637809148659 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240502 08:16:56.770009 23576 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240502 08:16:56.786840 25531 consensus_queue.cc:1035] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:56.787017 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63afd6e4fee24edeaba1209105f1c3e6" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:56.788015 23723 leader_election.cc:341] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.788136 23723 leader_election.cc:302] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:56.788499 24181 ts_tablet_manager.cc:1888] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.788502 24180 ts_tablet_manager.cc:1888] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:56.788585 25531 raft_consensus.cc:2726] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:56.788620 24181 log.cc:1196] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/cb7b37f776df47da9aa6e7fba82a8999
I20240502 08:16:56.788769 24107 heartbeater.cc:347] Connected to a master server at 127.22.225.125:36189
I20240502 08:16:56.789124 24181 ts_tablet_manager.cc:1909] T cb7b37f776df47da9aa6e7fba82a8999 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:56.788620 24180 log.cc:1196] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/18d284f71937459db0c0676ce4568175
I20240502 08:16:56.791818 24180 ts_tablet_manager.cc:1909] T 18d284f71937459db0c0676ce4568175 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.792697 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1039) took 1674 ms. Trace:
I20240502 08:16:56.792804 24180 rpcz_store.cc:276] 0502 08:16:55.118247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.118293 (+ 46us) service_pool.cc:225] Handling call
0502 08:16:56.741370 (+1623077us) ts_tablet_manager.cc:1120] Deleting tablet 18d284f71937459db0c0676ce4568175
0502 08:16:56.741380 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.744710 (+ 3330us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.788418 (+ 43708us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.792675 (+ 4257us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1622902}
I20240502 08:16:56.793257 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1040) took 1625 ms. Trace:
I20240502 08:16:56.793342 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 18d284f71937459db0c0676ce4568175 (table default.Table_7af51f5f2feb42ca8dbe067300bbbfd9 [id=7eff9f0e4b304a01b8d8c521bfdd1563]) successfully deleted
I20240502 08:16:56.793344 24180 rpcz_store.cc:276] 0502 08:16:55.168180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.180245 (+ 12065us) service_pool.cc:225] Handling call
0502 08:16:56.793124 (+1612879us) ts_tablet_manager.cc:1120] Deleting tablet 89fefde55e514e5fabae3f4e21f44ca3
0502 08:16:56.793136 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.793240 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1612234}
I20240502 08:16:56.793660 24180 tablet_replica.cc:330] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:56.793884 24180 raft_consensus.cc:2229] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240502 08:16:56.793913 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 89fefde55e514e5fabae3f4e21f44ca3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89fefde55e514e5fabae3f4e21f44ca3 already in progress: creating tablet
I20240502 08:16:56.794045 24180 raft_consensus.cc:2258] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.794615 24180 ts_tablet_manager.cc:1875] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:56.794872 23596 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet e20e126b1c9546e4a194bf9842f444a5 (table default.Table_4481e3bc8b3548b0a7ecef01d39f80e8 [id=f8b12b3bb4904c18be60b567cb05b739], current schema version=5) failed: Not found: failed to reset TS proxy: No leader found: Tablet e20e126b1c9546e4a194bf9842f444a5 consensus state has no leader
I20240502 08:16:56.797744 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 921) took 3355 ms. Trace:
I20240502 08:16:56.797858 24181 rpcz_store.cc:276] 0502 08:16:53.442305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.442806 (+ 501us) service_pool.cc:225] Handling call
0502 08:16:56.757880 (+3315074us) ts_tablet_manager.cc:1120] Deleting tablet cb7b37f776df47da9aa6e7fba82a8999
0502 08:16:56.757890 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.761277 (+ 3387us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.788369 (+ 27092us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.797717 (+ 9348us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3314928}
I20240502 08:16:56.798365 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 920) took 3356 ms. Trace:
I20240502 08:16:56.798457 24181 rpcz_store.cc:276] 0502 08:16:53.442183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.451464 (+ 9281us) service_pool.cc:225] Handling call
0502 08:16:56.798220 (+3346756us) ts_tablet_manager.cc:1120] Deleting tablet 711a38d7c47749f8a7f59e52b81b51ee
0502 08:16:56.798232 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.798338 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3343530}
I20240502 08:16:56.798796 24181 tablet_replica.cc:330] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:56.798424 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet cb7b37f776df47da9aa6e7fba82a8999 (table default.Table_3def35abf8ad4ff49c5ae77acb410e01 [id=5550f8374136404e84d93cc14b9cf834]) successfully deleted
I20240502 08:16:56.799014 24181 raft_consensus.cc:2229] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
W20240502 08:16:56.799055 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 711a38d7c47749f8a7f59e52b81b51ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 711a38d7c47749f8a7f59e52b81b51ee already in progress: creating tablet
I20240502 08:16:56.798626 24174 tablet_bootstrap.cc:654] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.799381 24181 raft_consensus.cc:2258] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.800025 24181 ts_tablet_manager.cc:1875] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.802949 25012 ts_tablet_manager.cc:1888] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.809607 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6b8552ddc1814ad5a20c72190929a5d9" } new_table_name: "default.Table_a394560ca3454cb88c3cbc2e87ab4fa7" modify_external_catalogs: true
I20240502 08:16:56.810145 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_15baf520893a4ad28df2c15c35649e15" } new_table_name: "default.Table_94b096765c2f49359eae0ab0f0ff178d" modify_external_catalogs: true
I20240502 08:16:56.810633 23610 master_service.cc:946] ReplaceTablet: received request to replace tablet 870edf582ab14ad1b57c4c4469b19954 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:56.811542 25158 consensus_queue.cc:1035] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:56.812131 24174 tablet_bootstrap.cc:492] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:56.812261 24174 ts_tablet_manager.cc:1367] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.002s
I20240502 08:16:56.809576 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b49e692bc51f437281bdebe78679a3ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:56.815299 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b49e692bc51f437281bdebe78679a3ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:56.817066 24181 ts_tablet_manager.cc:1888] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.817194 24181 log.cc:1196] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/577b149f52e44d5d9a9f6b9d85a6864d
I20240502 08:16:56.817659 24181 ts_tablet_manager.cc:1909] T 577b149f52e44d5d9a9f6b9d85a6864d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:56.818069 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 923) took 3321 ms. Trace:
I20240502 08:16:56.818192 24181 rpcz_store.cc:276] 0502 08:16:53.496711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.497113 (+ 402us) service_pool.cc:225] Handling call
0502 08:16:56.798736 (+3301623us) ts_tablet_manager.cc:1120] Deleting tablet 577b149f52e44d5d9a9f6b9d85a6864d
0502 08:16:56.798749 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.807757 (+ 9008us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.816911 (+ 9154us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.818044 (+ 1133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3301451}
I20240502 08:16:56.818614 24181 tablet_replica.cc:330] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:56.818862 24181 raft_consensus.cc:2229] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.819059 24181 raft_consensus.cc:2258] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.819047 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 577b149f52e44d5d9a9f6b9d85a6864d (table default.Table_1600e0e09856405ea0d90ac82aa47fda [id=89eea7c9830f47d2bd1c95044d5af4d6]) successfully deleted
I20240502 08:16:56.819708 24181 ts_tablet_manager.cc:1875] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.820257 24174 raft_consensus.cc:348] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.820430 24174 raft_consensus.cc:374] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.820477 24174 raft_consensus.cc:729] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.820647 24174 consensus_queue.cc:260] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.829022 24180 ts_tablet_manager.cc:1888] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:56.829144 24180 log.cc:1196] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/df08f0d3af4c4c7e8abce656714b1b7f
I20240502 08:16:56.829232 24174 ts_tablet_manager.cc:1398] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240502 08:16:56.829483 24180 ts_tablet_manager.cc:1909] T df08f0d3af4c4c7e8abce656714b1b7f P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.831663 24173 tablet_bootstrap.cc:492] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:56.831804 24173 ts_tablet_manager.cc:1367] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.080s user 0.003s sys 0.000s
W20240502 08:16:56.832233 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa1703fbb5bc42d1990d8064a14f2519 consensus state has no leader
I20240502 08:16:56.832455 24173 raft_consensus.cc:348] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.832634 24173 raft_consensus.cc:374] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.832684 24173 raft_consensus.cc:729] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.832866 24173 consensus_queue.cc:260] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.835232 24175 tablet_bootstrap.cc:492] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:56.835376 24175 ts_tablet_manager.cc:1367] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.017s
I20240502 08:16:56.836191 24175 raft_consensus.cc:348] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.836390 24175 raft_consensus.cc:374] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.836445 24175 raft_consensus.cc:729] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.838390 24181 ts_tablet_manager.cc:1888] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.838500 24181 log.cc:1196] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/4b8d81f071d74b218fcfd4f4e8c20bab
I20240502 08:16:56.803100 25012 log.cc:1196] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/e2126bf2d4ab44628b52481dc2ec3a99
I20240502 08:16:56.841593 25012 ts_tablet_manager.cc:1909] T e2126bf2d4ab44628b52481dc2ec3a99 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:56.847532 24173 ts_tablet_manager.cc:1398] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 08:16:56.847959 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 884) took 3681 ms. Trace:
I20240502 08:16:56.848500 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_bdb591e6a3e6408692d56430f64e36b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:56.848896 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9975eed13bd34c0e8e61352b573d680a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:56.849038 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdb591e6a3e6408692d56430f64e36b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:56.849352 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9975eed13bd34c0e8e61352b573d680a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:56.850206 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_81825c4d3ce8457ba17891e473aacddc" } new_table_name: "default.Table_2911b4ae0383471eafe8761c4674beed" modify_external_catalogs: true
I20240502 08:16:56.850585 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cda7da2a134f43a8a3c79a6609c3e52e" } new_table_name: "default.Table_4191e0f787d948bda654f8bbb6d8d24e" modify_external_catalogs: true
I20240502 08:16:56.848205 25494 raft_consensus.cc:482] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.851584 24175 consensus_queue.cc:260] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.851863 25494 raft_consensus.cc:504] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.851791 25012 rpcz_store.cc:276] 0502 08:16:53.166521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.166612 (+ 91us) service_pool.cc:225] Handling call
0502 08:16:56.608423 (+3441811us) ts_tablet_manager.cc:1120] Deleting tablet e2126bf2d4ab44628b52481dc2ec3a99
0502 08:16:56.608432 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.698461 (+ 90029us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.802665 (+104204us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.847937 (+ 45272us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3441647}
I20240502 08:16:56.852806 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet e2126bf2d4ab44628b52481dc2ec3a99 (table default.Table_5b4e617d1c8949daa34aa358c1528f88 [id=92cbf164660d48bd974bf8a26b8dfb0d]) successfully deleted
I20240502 08:16:56.852905 25494 leader_election.cc:288] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:56.853389 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59c2f93d6514c9cbf358ab96f1a06b5" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:56.854038 24173 tablet_bootstrap.cc:492] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:56.854245 24181 ts_tablet_manager.cc:1909] T 4b8d81f071d74b218fcfd4f4e8c20bab P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:56.854525 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 937) took 3299 ms. Trace:
I20240502 08:16:56.854589 24181 rpcz_store.cc:276] 0502 08:16:53.555289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.555467 (+ 178us) service_pool.cc:225] Handling call
0502 08:16:56.818558 (+3263091us) ts_tablet_manager.cc:1120] Deleting tablet 4b8d81f071d74b218fcfd4f4e8c20bab
0502 08:16:56.818571 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.832478 (+ 13907us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.838292 (+ 5814us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.854508 (+ 16216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3262887}
I20240502 08:16:56.854856 24181 tablet_replica.cc:330] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:56.855031 24181 raft_consensus.cc:2229] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:56.855314 24181 raft_consensus.cc:2258] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.855825 24181 ts_tablet_manager.cc:1875] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.856317 24173 tablet_bootstrap.cc:654] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:56.856719 23858 leader_election.cc:341] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: b59c2f93d6514c9cbf358ab96f1a06b5
I20240502 08:16:56.856854 23858 leader_election.cc:302] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:56.857024 25494 raft_consensus.cc:2726] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:56.857331 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 4b8d81f071d74b218fcfd4f4e8c20bab (table default.Table_dbe385cc3f6b48d39bee298ca9490248 [id=aa825e5ca73c4dafa9585126d5c48df7]) successfully deleted
I20240502 08:16:56.858283 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ecb54a9c637e45e6ac9bbc1012d1d78f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:56.858700 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecb54a9c637e45e6ac9bbc1012d1d78f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:56.860249 25012 tablet_replica.cc:330] T 1ba1baa2279045b3adc58d187a12cbf3 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:56.860548 25012 raft_consensus.cc:2229] T 1ba1baa2279045b3adc58d187a12cbf3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.862579 25012 raft_consensus.cc:2258] T 1ba1baa2279045b3adc58d187a12cbf3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.852234 24175 ts_tablet_manager.cc:1398] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.017s user 0.001s sys 0.002s
I20240502 08:16:56.868101 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_74c703e708b34d03891d74e02e877c53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:56.868193 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5af2a55b1ec14ac0ab16ca949eb002b8" } modify_external_catalogs: true
I20240502 08:16:56.868422 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5af2a55b1ec14ac0ab16ca949eb002b8" } modify_external_catalogs: true
W20240502 08:16:56.868628 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74c703e708b34d03891d74e02e877c53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:56.870364 24175 tablet_bootstrap.cc:492] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:16:56.875212 23996 consensus_peers.cc:485] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:16:56.875294 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 981b7a7292884b1fb48af99db6e762c0 consensus state has no leader
I20240502 08:16:56.877245 24174 tablet_bootstrap.cc:492] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:56.878146 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1041) took 1621 ms. Trace:
I20240502 08:16:56.878232 24180 rpcz_store.cc:276] 0502 08:16:55.256670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.262477 (+ 5807us) service_pool.cc:225] Handling call
0502 08:16:56.793612 (+1531135us) ts_tablet_manager.cc:1120] Deleting tablet df08f0d3af4c4c7e8abce656714b1b7f
0502 08:16:56.793621 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.797536 (+ 3915us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.828929 (+ 31393us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.878126 (+ 49197us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1530883}
I20240502 08:16:56.878502 24180 tablet_replica.cc:330] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:56.878677 24180 raft_consensus.cc:2229] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.878795 24180 raft_consensus.cc:2258] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.878777 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet df08f0d3af4c4c7e8abce656714b1b7f (table default.Table_59f82dbde55b406ba5271520c5bab104 [id=8e9acf4c0dbf4590b82aaee07243ed0c]) successfully deleted
I20240502 08:16:56.879280 24180 ts_tablet_manager.cc:1875] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.882660 24174 tablet_bootstrap.cc:654] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.887444 24175 tablet_bootstrap.cc:654] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.887787 25012 ts_tablet_manager.cc:1875] T 1ba1baa2279045b3adc58d187a12cbf3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.901158 24181 ts_tablet_manager.cc:1888] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.901274 24181 log.cc:1196] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/751808dfc1964b539f6f836d786156aa
I20240502 08:16:56.901621 24181 ts_tablet_manager.cc:1909] T 751808dfc1964b539f6f836d786156aa P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:56.902315 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 938) took 3346 ms. Trace:
I20240502 08:16:56.904361 24174 tablet_bootstrap.cc:492] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:56.904492 24174 ts_tablet_manager.cc:1367] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240502 08:16:56.905076 25158 raft_consensus.cc:482] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.905211 25158 raft_consensus.cc:504] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.905611 24180 ts_tablet_manager.cc:1888] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:56.905735 24180 log.cc:1196] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/25c089f0c1b14e419f250d03d4eaad12
I20240502 08:16:56.905875 25158 leader_election.cc:288] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:56.906018 24180 ts_tablet_manager.cc:1909] T 25c089f0c1b14e419f250d03d4eaad12 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:56.906746 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1042) took 1578 ms. Trace:
I20240502 08:16:56.906690 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3aa2621fdb43fd8299f81af486ef1e" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:56.906896 24053 raft_consensus.cc:2454] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:56.906833 24180 rpcz_store.cc:276] 0502 08:16:55.328250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.332353 (+ 4103us) service_pool.cc:225] Handling call
0502 08:16:56.878465 (+1546112us) ts_tablet_manager.cc:1120] Deleting tablet 25c089f0c1b14e419f250d03d4eaad12
0502 08:16:56.878472 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.894160 (+ 15688us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.905506 (+ 11346us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.906727 (+ 1221us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1543022}
I20240502 08:16:56.907176 24180 tablet_replica.cc:330] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:56.907398 24180 raft_consensus.cc:2229] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.907492 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 25c089f0c1b14e419f250d03d4eaad12 (table default.Table_94efd28ada5b4024ab65e4d5feda0b38 [id=15b42ca7619645c8ac3cff50044f069b]) successfully deleted
I20240502 08:16:56.907541 24180 raft_consensus.cc:2258] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.908006 24180 ts_tablet_manager.cc:1875] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.907974 23723 leader_election.cc:302] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:56.908432 25158 raft_consensus.cc:2781] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:56.908507 25158 raft_consensus.cc:482] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:56.908568 25158 raft_consensus.cc:3037] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:56.911403 25158 raft_consensus.cc:504] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.912649 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3aa2621fdb43fd8299f81af486ef1e" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:56.912839 24053 raft_consensus.cc:3037] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:56.902395 24181 rpcz_store.cc:276] 0502 08:16:53.555415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.555702 (+ 287us) service_pool.cc:225] Handling call
0502 08:16:56.854818 (+3299116us) ts_tablet_manager.cc:1120] Deleting tablet 751808dfc1964b539f6f836d786156aa
0502 08:16:56.854826 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.897294 (+ 42468us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.901079 (+ 3785us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.902294 (+ 1215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3298995}
I20240502 08:16:56.913862 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 939) took 3343 ms. Trace:
I20240502 08:16:56.913905 25012 ts_tablet_manager.cc:1888] T 1ba1baa2279045b3adc58d187a12cbf3 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.914021 25012 log.cc:1196] T 1ba1baa2279045b3adc58d187a12cbf3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/1ba1baa2279045b3adc58d187a12cbf3
I20240502 08:16:56.913949 24181 rpcz_store.cc:276] 0502 08:16:53.570303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.570491 (+ 188us) service_pool.cc:225] Handling call
0502 08:16:56.913737 (+3343246us) ts_tablet_manager.cc:1120] Deleting tablet 88acb3b0c2b7480e917ed68b7646b47d
0502 08:16:56.913748 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.913845 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3289482}
I20240502 08:16:56.914302 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 940) took 3270 ms. Trace:
I20240502 08:16:56.914371 25012 ts_tablet_manager.cc:1909] T 1ba1baa2279045b3adc58d187a12cbf3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:56.914386 24181 rpcz_store.cc:276] 0502 08:16:53.643390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.643868 (+ 478us) service_pool.cc:225] Handling call
0502 08:16:56.914195 (+3270327us) ts_tablet_manager.cc:1120] Deleting tablet 765fdb66f12941f0ac16223e931e576e
0502 08:16:56.914205 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.914287 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3269679}
I20240502 08:16:56.914667 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 942) took 3265 ms. Trace:
I20240502 08:16:56.914685 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 887) took 3714 ms. Trace:
I20240502 08:16:56.914732 24181 rpcz_store.cc:276] 0502 08:16:53.648982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.649116 (+ 134us) service_pool.cc:225] Handling call
0502 08:16:56.914583 (+3265467us) ts_tablet_manager.cc:1120] Deleting tablet df266081ed8345dda989b3050f605685
0502 08:16:56.914590 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.914653 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3264388}
I20240502 08:16:56.914764 25012 rpcz_store.cc:276] 0502 08:16:53.200128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.200227 (+ 99us) service_pool.cc:225] Handling call
0502 08:16:56.860199 (+3659972us) ts_tablet_manager.cc:1120] Deleting tablet 1ba1baa2279045b3adc58d187a12cbf3
0502 08:16:56.860210 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.900900 (+ 40690us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.913805 (+ 12905us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.914650 (+ 845us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3659733}
I20240502 08:16:56.915015 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 941) took 3266 ms. Trace:
I20240502 08:16:56.915117 25012 tablet_replica.cc:330] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:56.915076 24181 rpcz_store.cc:276] 0502 08:16:53.648875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.649179 (+ 304us) service_pool.cc:225] Handling call
0502 08:16:56.914929 (+3265750us) ts_tablet_manager.cc:1120] Deleting tablet 552e46f69230474189278911cfeec6c7
0502 08:16:56.914937 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.915002 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3254680}
I20240502 08:16:56.915246 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 1ba1baa2279045b3adc58d187a12cbf3 (table default.Table_1704983c4496461d9fa8ea54781f5cb1 [id=1dbff884223d4ad9843fe2512d1f295c]) successfully deleted
I20240502 08:16:56.915320 25012 raft_consensus.cc:2229] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.915343 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 950) took 2887 ms. Trace:
I20240502 08:16:56.915398 24181 rpcz_store.cc:276] 0502 08:16:54.027840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.029327 (+ 1487us) service_pool.cc:225] Handling call
0502 08:16:56.915255 (+2885928us) ts_tablet_manager.cc:1120] Deleting tablet 17d6f4f6e9ee46c28b7e7513f791c81f
0502 08:16:56.915263 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.915328 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2885575}
I20240502 08:16:56.915511 25012 raft_consensus.cc:2258] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.915680 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 952) took 2878 ms. Trace:
I20240502 08:16:56.915746 24181 rpcz_store.cc:276] 0502 08:16:54.037067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.037400 (+ 333us) service_pool.cc:225] Handling call
0502 08:16:56.915574 (+2878174us) ts_tablet_manager.cc:1120] Deleting tablet 518cb4b86ea34e95bef51c5c2a64d89b
0502 08:16:56.915581 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.915667 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2877944}
I20240502 08:16:56.916015 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 951) took 2879 ms. Trace:
I20240502 08:16:56.916059 25012 ts_tablet_manager.cc:1875] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.916088 24181 rpcz_store.cc:276] 0502 08:16:54.036993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.038986 (+ 1993us) service_pool.cc:225] Handling call
0502 08:16:56.915932 (+2876946us) ts_tablet_manager.cc:1120] Deleting tablet 7a0ff1c3818a49d2b13f57ea6a81c3d6
0502 08:16:56.915940 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.916003 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2876755}
I20240502 08:16:56.916354 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 953) took 2853 ms. Trace:
I20240502 08:16:56.916414 24181 rpcz_store.cc:276] 0502 08:16:54.062753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.065926 (+ 3173us) service_pool.cc:225] Handling call
0502 08:16:56.916263 (+2850337us) ts_tablet_manager.cc:1120] Deleting tablet ac7a7cec2e154d6fa095d68d1ba3308d
0502 08:16:56.916270 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.916339 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2844907}
I20240502 08:16:56.916667 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 954) took 2718 ms. Trace:
I20240502 08:16:56.916723 24181 rpcz_store.cc:276] 0502 08:16:54.198198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.198271 (+ 73us) service_pool.cc:225] Handling call
0502 08:16:56.916587 (+2718316us) ts_tablet_manager.cc:1120] Deleting tablet e4ac2b5073e34cd98e0246620fc1515d
0502 08:16:56.916594 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.916655 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2718098}
I20240502 08:16:56.916985 24181 tablet_replica.cc:330] T 1da01e2c2f054ccfb53383e70e16da84 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:56.917201 24181 raft_consensus.cc:2229] T 1da01e2c2f054ccfb53383e70e16da84 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.917366 24181 raft_consensus.cc:2258] T 1da01e2c2f054ccfb53383e70e16da84 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.918879 24053 raft_consensus.cc:2454] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:56.919982 23723 leader_election.cc:302] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:56.920243 25531 raft_consensus.cc:2781] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:56.920346 25531 raft_consensus.cc:686] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:56.920562 25531 consensus_queue.cc:237] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.925926 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3aa2621fdb43fd8299f81af486ef1e" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:56.926618 23723 leader_election.cc:341] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.927577 25158 leader_election.cc:288] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:56.928002 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3aa2621fdb43fd8299f81af486ef1e" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:56.928691 23723 leader_election.cc:341] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.905084 24174 raft_consensus.cc:348] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.929241 24174 raft_consensus.cc:374] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.929348 24174 raft_consensus.cc:729] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.929682 24174 consensus_queue.cc:260] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.930346 24174 ts_tablet_manager.cc:1398] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240502 08:16:56.931530 24174 tablet_bootstrap.cc:492] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
W20240502 08:16:56.931982 23858 consensus_peers.cc:485] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd3078cc1f6247609f8bd017ed076b45. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:56.933643 25408 raft_consensus.cc:482] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.933796 25408 raft_consensus.cc:504] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.934526 25408 leader_election.cc:288] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:56.935139 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f57e3a39d54b7ab372d4b145b7a696" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:56.935684 23858 leader_election.cc:341] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 89f57e3a39d54b7ab372d4b145b7a696
W20240502 08:16:56.935976 23861 leader_election.cc:341] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.936010 24181 ts_tablet_manager.cc:1875] T 1da01e2c2f054ccfb53383e70e16da84 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.936093 23861 leader_election.cc:302] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:56.936628 25408 raft_consensus.cc:2726] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:56.937443 24174 tablet_bootstrap.cc:654] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.941711 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 751808dfc1964b539f6f836d786156aa (table default.Table_ad81523678c3450c880efd1ecfc5f95e [id=911d4d5a86e54a5696d37a2bd9f69733]) successfully deleted
W20240502 08:16:56.942020 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 88acb3b0c2b7480e917ed68b7646b47d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88acb3b0c2b7480e917ed68b7646b47d already in progress: creating tablet
W20240502 08:16:56.942242 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 765fdb66f12941f0ac16223e931e576e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 765fdb66f12941f0ac16223e931e576e already in progress: creating tablet
W20240502 08:16:56.942481 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet df266081ed8345dda989b3050f605685 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df266081ed8345dda989b3050f605685 already in progress: creating tablet
W20240502 08:16:56.942659 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 552e46f69230474189278911cfeec6c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 552e46f69230474189278911cfeec6c7 already in progress: creating tablet
W20240502 08:16:56.944164 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 17d6f4f6e9ee46c28b7e7513f791c81f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17d6f4f6e9ee46c28b7e7513f791c81f already in progress: creating tablet
I20240502 08:16:56.944557 24180 ts_tablet_manager.cc:1888] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:56.944679 24180 log.cc:1196] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/090bec30f5664421b9bfe3f2810afd14
I20240502 08:16:56.944963 24180 ts_tablet_manager.cc:1909] T 090bec30f5664421b9bfe3f2810afd14 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
W20240502 08:16:56.945338 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 518cb4b86ea34e95bef51c5c2a64d89b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 518cb4b86ea34e95bef51c5c2a64d89b already in progress: creating tablet
I20240502 08:16:56.949301 24175 tablet_bootstrap.cc:492] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:56.949509 24175 ts_tablet_manager.cc:1367] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.004s
I20240502 08:16:56.950394 24175 raft_consensus.cc:348] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.950670 24175 raft_consensus.cc:374] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:56.950795 24175 raft_consensus.cc:729] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.951066 24175 consensus_queue.cc:260] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.953924 24175 ts_tablet_manager.cc:1398] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:56.954324 24175 tablet_bootstrap.cc:492] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:56.955986 24175 tablet_bootstrap.cc:654] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.967239 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b4d8ce6957a042f486a41196af1caaaa" } modify_external_catalogs: true
I20240502 08:16:56.967474 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b4d8ce6957a042f486a41196af1caaaa" } modify_external_catalogs: true
I20240502 08:16:56.968091 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8d08b791cfc34eef87f84aff3275577d" } modify_external_catalogs: true
I20240502 08:16:56.968243 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8d08b791cfc34eef87f84aff3275577d" } modify_external_catalogs: true
I20240502 08:16:56.969889 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_44816133bdb24a48a436fa62d832ed30" } new_table_name: "default.Table_2e569d61a7b94fb89013bed2831ec5c1" modify_external_catalogs: true
I20240502 08:16:56.971241 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1043) took 1373 ms. Trace:
I20240502 08:16:56.971287 24181 ts_tablet_manager.cc:1888] T 1da01e2c2f054ccfb53383e70e16da84 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:56.971416 24181 log.cc:1196] T 1da01e2c2f054ccfb53383e70e16da84 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/1da01e2c2f054ccfb53383e70e16da84
I20240502 08:16:56.971515 24180 rpcz_store.cc:276] 0502 08:16:55.597818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.597892 (+ 74us) service_pool.cc:225] Handling call
0502 08:16:56.907092 (+1309200us) ts_tablet_manager.cc:1120] Deleting tablet 090bec30f5664421b9bfe3f2810afd14
0502 08:16:56.907100 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.934770 (+ 27670us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.944448 (+ 9678us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.971214 (+ 26766us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1308982}
I20240502 08:16:56.971805 24181 ts_tablet_manager.cc:1909] T 1da01e2c2f054ccfb53383e70e16da84 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:56.972191 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1045) took 1335 ms. Trace:
I20240502 08:16:56.972322 24180 rpcz_store.cc:276] 0502 08:16:55.637150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.637197 (+ 47us) service_pool.cc:225] Handling call
0502 08:16:56.971983 (+1334786us) ts_tablet_manager.cc:1120] Deleting tablet 0281c8123eab4c48bddc377f5695c6e0
0502 08:16:56.971998 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.972172 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1334573}
I20240502 08:16:56.972637 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 955) took 2675 ms. Trace:
I20240502 08:16:56.972733 24181 rpcz_store.cc:276] 0502 08:16:54.297381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.297495 (+ 114us) service_pool.cc:225] Handling call
0502 08:16:56.916937 (+2619442us) ts_tablet_manager.cc:1120] Deleting tablet 1da01e2c2f054ccfb53383e70e16da84
0502 08:16:56.916947 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.944183 (+ 27236us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.971166 (+ 26983us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.972614 (+ 1448us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2619250}
I20240502 08:16:56.973106 24181 tablet_replica.cc:330] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:56.973304 24181 raft_consensus.cc:2229] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:56.972658 25012 ts_tablet_manager.cc:1888] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 08:16:56.973953 25012 log.cc:1196] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/050c16739efb4037a4adc78a2ffd3cba
I20240502 08:16:56.974138 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1049) took 1320 ms. Trace:
I20240502 08:16:56.974618 24180 rpcz_store.cc:276] 0502 08:16:55.653457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.657609 (+ 4152us) service_pool.cc:225] Handling call
0502 08:16:56.973100 (+1315491us) ts_tablet_manager.cc:1120] Deleting tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6
0502 08:16:56.973110 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.974119 (+ 1009us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1315332}
I20240502 08:16:56.974825 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 090bec30f5664421b9bfe3f2810afd14 (table default.Table_70034c04442f49f6a42cb66dccfd1cc1 [id=74be9e9508024ebeb4623c488e21faf9]) successfully deleted
W20240502 08:16:56.975139 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 0281c8123eab4c48bddc377f5695c6e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0281c8123eab4c48bddc377f5695c6e0 already in progress: creating tablet
I20240502 08:16:56.975189 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1050) took 1321 ms. Trace:
I20240502 08:16:56.975260 24180 rpcz_store.cc:276] 0502 08:16:55.653578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.657798 (+ 4220us) service_pool.cc:225] Handling call
0502 08:16:56.975087 (+1317289us) ts_tablet_manager.cc:1120] Deleting tablet 39cc4ef1174a4358bad777b0471c5f76
0502 08:16:56.975096 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.975174 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1317166}
I20240502 08:16:56.975548 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1047) took 1322 ms. Trace:
I20240502 08:16:56.975631 24180 rpcz_store.cc:276] 0502 08:16:55.652896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.664644 (+ 11748us) service_pool.cc:225] Handling call
0502 08:16:56.975459 (+1310815us) ts_tablet_manager.cc:1120] Deleting tablet 558a3a4a8383439bb64af1d3eaa64572
0502 08:16:56.975467 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.975535 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1310603}
I20240502 08:16:56.976750 25012 ts_tablet_manager.cc:1909] T 050c16739efb4037a4adc78a2ffd3cba P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:56.975937 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1055) took 1274 ms. Trace:
I20240502 08:16:56.978912 24180 rpcz_store.cc:276] 0502 08:16:55.701223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.701279 (+ 56us) service_pool.cc:225] Handling call
0502 08:16:56.975836 (+1274557us) ts_tablet_manager.cc:1120] Deleting tablet bb57cb00e884420bacc48f16739eb29c
0502 08:16:56.975846 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.975920 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1274373}
I20240502 08:16:56.979285 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 886) took 3777 ms. Trace:
I20240502 08:16:56.979279 25531 raft_consensus.cc:482] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:56.979389 25012 rpcz_store.cc:276] 0502 08:16:53.200006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.202971 (+ 2965us) service_pool.cc:225] Handling call
0502 08:16:56.915048 (+3712077us) ts_tablet_manager.cc:1120] Deleting tablet 050c16739efb4037a4adc78a2ffd3cba
0502 08:16:56.915056 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.946996 (+ 31940us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.963972 (+ 16976us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:56.977551 (+ 13579us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3711836}
I20240502 08:16:56.979429 25531 raft_consensus.cc:504] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:56.980131 25531 leader_election.cc:288] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:56.980338 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1056) took 1276 ms. Trace:
I20240502 08:16:56.980427 24180 rpcz_store.cc:276] 0502 08:16:55.703638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.703685 (+ 47us) service_pool.cc:225] Handling call
0502 08:16:56.979140 (+1275455us) ts_tablet_manager.cc:1120] Deleting tablet 36dece738afd42b99f3f38d7fd11a0af
0502 08:16:56.980229 (+ 1089us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.980319 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1275307}
I20240502 08:16:56.980865 24175 tablet_bootstrap.cc:492] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:56.981024 24175 ts_tablet_manager.cc:1367] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20240502 08:16:56.981168 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e645bb6e2948e091e08433944a52f4" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:56.981801 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6 already in progress: creating tablet
W20240502 08:16:56.982056 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 39cc4ef1174a4358bad777b0471c5f76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39cc4ef1174a4358bad777b0471c5f76 already in progress: creating tablet
W20240502 08:16:56.982205 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 558a3a4a8383439bb64af1d3eaa64572 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 558a3a4a8383439bb64af1d3eaa64572 already in progress: creating tablet
I20240502 08:16:56.982156 24175 raft_consensus.cc:348] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:56.982329 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet bb57cb00e884420bacc48f16739eb29c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb57cb00e884420bacc48f16739eb29c already in progress: creating tablet
I20240502 08:16:56.982369 24175 raft_consensus.cc:374] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 08:16:56.982488 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 36dece738afd42b99f3f38d7fd11a0af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36dece738afd42b99f3f38d7fd11a0af already in progress: creating tablet
I20240502 08:16:56.982620 24175 raft_consensus.cc:729] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.982925 24175 consensus_queue.cc:260] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:56.983436 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e645bb6e2948e091e08433944a52f4" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:56.983850 24175 ts_tablet_manager.cc:1398] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240502 08:16:56.983839 23723 leader_election.cc:341] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:56.984051 23723 leader_election.cc:341] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:56.984141 23723 leader_election.cc:302] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:56.984220 24175 tablet_bootstrap.cc:492] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:56.984628 25531 raft_consensus.cc:2726] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:56.984946 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1057) took 1278 ms. Trace:
I20240502 08:16:56.985028 24180 rpcz_store.cc:276] 0502 08:16:55.706608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.706656 (+ 48us) service_pool.cc:225] Handling call
0502 08:16:56.980649 (+1273993us) ts_tablet_manager.cc:1120] Deleting tablet dfd897ba97204b6788fd2d99cf2a0303
0502 08:16:56.980658 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.984927 (+ 4269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1273778}
I20240502 08:16:56.985680 24180 tablet_replica.cc:330] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:56.985925 24180 raft_consensus.cc:2229] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:56.986157 24175 tablet_bootstrap.cc:654] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:56.986239 24180 raft_consensus.cc:2258] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.986477 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 896) took 3744 ms. Trace:
I20240502 08:16:56.987061 24180 ts_tablet_manager.cc:1875] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:56.989051 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet dfd897ba97204b6788fd2d99cf2a0303 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfd897ba97204b6788fd2d99cf2a0303 already in progress: creating tablet
W20240502 08:16:56.989256 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 506644a1ead54a1faeaa8583ef96b449 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 506644a1ead54a1faeaa8583ef96b449 already in progress: creating tablet
W20240502 08:16:56.989432 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet fc87f91093b94b209caa6f89691bcde3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc87f91093b94b209caa6f89691bcde3 already in progress: creating tablet
I20240502 08:16:56.993718 23603 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 524a26074c3c4c91bf57fb9b077adb54
W20240502 08:16:56.994148 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5edadeb53a354c39a33ff68ffbde5ad1 consensus state has no leader
I20240502 08:16:56.996271 24181 raft_consensus.cc:2258] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:56.996987 24181 ts_tablet_manager.cc:1875] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:56.997901 25012 rpcz_store.cc:276] 0502 08:16:53.235721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.235866 (+ 145us) service_pool.cc:225] Handling call
0502 08:16:56.979634 (+3743768us) ts_tablet_manager.cc:1120] Deleting tablet 8075b03643a14a01ab75921fc3d3dace
0502 08:16:56.979644 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.979734 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3743585}
I20240502 08:16:56.998476 25012 tablet_replica.cc:330] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:56.998878 25012 raft_consensus.cc:2229] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:56.999292 25012 raft_consensus.cc:2258] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.000536 25012 ts_tablet_manager.cc:1875] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:56.948915 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 7a0ff1c3818a49d2b13f57ea6a81c3d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a0ff1c3818a49d2b13f57ea6a81c3d6 already in progress: creating tablet
W20240502 08:16:57.002790 23597 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 53884.3us
W20240502 08:16:57.003091 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet ac7a7cec2e154d6fa095d68d1ba3308d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac7a7cec2e154d6fa095d68d1ba3308d already in progress: creating tablet
W20240502 08:16:57.003388 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e4ac2b5073e34cd98e0246620fc1515d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4ac2b5073e34cd98e0246620fc1515d already in progress: creating tablet
I20240502 08:16:57.003597 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 1da01e2c2f054ccfb53383e70e16da84 (table default.Table_3ef466d7a87644e99070934d65e5c851 [id=f596bcb8d6f04ac680be38e763a33750]) successfully deleted
I20240502 08:16:57.003862 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 050c16739efb4037a4adc78a2ffd3cba (table default.Table_3a66619663c642f5be35fd8249184e1a [id=ba62a673079d424992daa0f6b0c5d6b9]) successfully deleted
W20240502 08:16:57.004077 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 8075b03643a14a01ab75921fc3d3dace because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8075b03643a14a01ab75921fc3d3dace already in progress: creating tablet
I20240502 08:16:57.007546 25413 raft_consensus.cc:482] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.007678 25413 raft_consensus.cc:504] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.008273 25413 leader_election.cc:288] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:57.008949 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c395a29c7a1485eab39ca107dd91fca" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:57.009543 23996 leader_election.cc:341] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.012815 25012 ts_tablet_manager.cc:1888] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:16:57.008930 24180 ts_tablet_manager.cc:1888] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:57.014055 24180 log.cc:1196] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/d6a19bb8705c45c0bfc537d3a6bf8ae2
I20240502 08:16:57.014508 24180 ts_tablet_manager.cc:1909] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:57.016348 24180 tablet_replica.cc:330] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:57.013794 24175 tablet_bootstrap.cc:492] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:57.016657 24175 ts_tablet_manager.cc:1367] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.002s
I20240502 08:16:57.017472 24175 raft_consensus.cc:348] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.017796 24175 raft_consensus.cc:374] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:57.018059 24175 raft_consensus.cc:729] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:57.018364 24175 consensus_queue.cc:260] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.018904 24175 ts_tablet_manager.cc:1398] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:57.019258 24175 tablet_bootstrap.cc:492] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:57.020813 24175 tablet_bootstrap.cc:654] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:57.022311 25012 log.cc:1196] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/77c49da1e0f0435ba241a6a66526e790
I20240502 08:16:57.023232 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 (table default.Table_c67035cbec4e42109cb3d466e0a6ad18 [id=b2cb0ed2926d4be7ade0790203b57e2e]) successfully deleted
I20240502 08:16:57.023391 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_465db433d6a5443bac1914682bc40769"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.023967 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_465db433d6a5443bac1914682bc40769 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.025177 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6e99bd5e0b2a4692a1400b64cb638f26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:57.025458 25413 raft_consensus.cc:482] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.025643 25413 raft_consensus.cc:504] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:16:57.025777 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e99bd5e0b2a4692a1400b64cb638f26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.026422 25413 leader_election.cc:288] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:57.023523 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 16a1c5276f014410b740c87b9971fb48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16a1c5276f014410b740c87b9971fb48 already in progress: creating tablet
W20240502 08:16:57.026901 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 7fe583cbf4af493e82d3ec0508ea0f0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fe583cbf4af493e82d3ec0508ea0f0b already in progress: creating tablet
W20240502 08:16:57.027135 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 548ab3f9d0d84527b1092ae715ec2bdd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 548ab3f9d0d84527b1092ae715ec2bdd already in progress: creating tablet
I20240502 08:16:57.027199 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec10ff7251a5442a8ecc1881b34db74d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:57.027416 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 96013fa1fa014b1891eb621cc3a81cdc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96013fa1fa014b1891eb621cc3a81cdc already in progress: creating tablet
W20240502 08:16:57.027905 23996 leader_election.cc:341] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.028417 25012 ts_tablet_manager.cc:1909] T 77c49da1e0f0435ba241a6a66526e790 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:57.010217 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c395a29c7a1485eab39ca107dd91fca" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:57.027819 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec10ff7251a5442a8ecc1881b34db74d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:57.028661 23783 raft_consensus.cc:2454] T 4c395a29c7a1485eab39ca107dd91fca P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:57.028672 23788 raft_consensus.cc:2454] T ec10ff7251a5442a8ecc1881b34db74d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:57.029129 23993 leader_election.cc:302] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:57.029096 24180 raft_consensus.cc:2229] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:57.029409 24180 raft_consensus.cc:2258] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.029489 23993 leader_election.cc:302] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:57.029677 25405 raft_consensus.cc:2781] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:57.029735 25405 raft_consensus.cc:482] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:57.029796 25405 raft_consensus.cc:3037] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:57.029866 25413 raft_consensus.cc:2781] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:57.029914 25413 raft_consensus.cc:482] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:57.029959 25413 raft_consensus.cc:3037] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:57.029953 24180 ts_tablet_manager.cc:1875] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.031208 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 897) took 3794 ms. Trace:
I20240502 08:16:57.031324 25012 rpcz_store.cc:276] 0502 08:16:53.236832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.237004 (+ 172us) service_pool.cc:225] Handling call
0502 08:16:56.998389 (+3761385us) ts_tablet_manager.cc:1120] Deleting tablet 77c49da1e0f0435ba241a6a66526e790
0502 08:16:56.998399 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.003704 (+ 5305us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.012648 (+ 8944us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.031170 (+ 18522us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3761215}
I20240502 08:16:57.031810 25012 tablet_replica.cc:330] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:57.032035 25012 raft_consensus.cc:2229] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:57.032192 25012 raft_consensus.cc:2258] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.032243 25405 raft_consensus.cc:504] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.032784 25405 leader_election.cc:288] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:57.033308 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c395a29c7a1485eab39ca107dd91fca" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:57.033777 23996 leader_election.cc:341] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.033815 24181 ts_tablet_manager.cc:1888] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:57.033926 24181 log.cc:1196] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/5d04312bd81f46ed8c49c42b1af1aec9
I20240502 08:16:57.034296 24181 ts_tablet_manager.cc:1909] T 5d04312bd81f46ed8c49c42b1af1aec9 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:57.035219 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c395a29c7a1485eab39ca107dd91fca" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:57.035413 23788 raft_consensus.cc:3037] T 4c395a29c7a1485eab39ca107dd91fca P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:57.037122 25413 raft_consensus.cc:504] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.038609 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec10ff7251a5442a8ecc1881b34db74d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:57.038772 23783 raft_consensus.cc:3037] T ec10ff7251a5442a8ecc1881b34db74d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:57.040448 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_df3f96e9bc0e43ef9c4d791737052252"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.040942 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df3f96e9bc0e43ef9c4d791737052252 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.041816 23783 raft_consensus.cc:2454] T ec10ff7251a5442a8ecc1881b34db74d P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:57.042361 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 956) took 2590 ms. Trace:
I20240502 08:16:57.042452 24181 rpcz_store.cc:276] 0502 08:16:54.451773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.467578 (+ 15805us) service_pool.cc:225] Handling call
0502 08:16:56.973055 (+2505477us) ts_tablet_manager.cc:1120] Deleting tablet 5d04312bd81f46ed8c49c42b1af1aec9
0502 08:16:56.973066 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:56.999936 (+ 26870us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.033709 (+ 33773us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.042338 (+ 8629us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2496120}
I20240502 08:16:57.042882 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 959) took 2281 ms. Trace:
I20240502 08:16:57.042961 24181 rpcz_store.cc:276] 0502 08:16:54.761727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.761785 (+ 58us) service_pool.cc:225] Handling call
0502 08:16:57.042738 (+2280953us) ts_tablet_manager.cc:1120] Deleting tablet bb9b75d6be924d0291f94fc4a439dea6
0502 08:16:57.042751 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.042865 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2280747}
I20240502 08:16:57.043236 24181 tablet_replica.cc:330] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:57.043435 24181 raft_consensus.cc:2229] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:57.043620 24181 raft_consensus.cc:2258] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.043756 23993 leader_election.cc:302] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:57.043980 25405 raft_consensus.cc:2781] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:57.044106 25405 raft_consensus.cc:686] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:57.044186 24181 ts_tablet_manager.cc:1875] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.044313 25405 consensus_queue.cc:237] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.044294 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8eb9563cd4c74c64a4ba798132ddf385"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.044778 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8eb9563cd4c74c64a4ba798132ddf385 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.045457 25413 leader_election.cc:288] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:57.047062 23788 raft_consensus.cc:2454] T 4c395a29c7a1485eab39ca107dd91fca P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:57.047724 25012 ts_tablet_manager.cc:1875] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.047827 23993 leader_election.cc:302] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:57.050845 25413 raft_consensus.cc:2781] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:57.051045 25413 raft_consensus.cc:686] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:57.051296 25413 consensus_queue.cc:237] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.057998 24175 tablet_bootstrap.cc:492] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:57.058199 24175 ts_tablet_manager.cc:1367] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.000s
I20240502 08:16:57.060813 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b20da99d7e65450cb4f49c3397e1e54f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.061295 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b20da99d7e65450cb4f49c3397e1e54f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.061990 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_35d118eaf50b48edaa15a45c6cef67b8" } new_table_name: "default.Table_08cf327ecab84357a007ce35456fedea" modify_external_catalogs: true
I20240502 08:16:57.011250 24174 tablet_bootstrap.cc:492] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:57.065994 24174 ts_tablet_manager.cc:1367] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.135s user 0.003s sys 0.000s
I20240502 08:16:57.066485 24181 ts_tablet_manager.cc:1888] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:57.066624 24181 log.cc:1196] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/b59c2f93d6514c9cbf358ab96f1a06b5
I20240502 08:16:57.066749 24175 raft_consensus.cc:348] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.067041 24181 ts_tablet_manager.cc:1909] T b59c2f93d6514c9cbf358ab96f1a06b5 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:57.067071 24175 raft_consensus.cc:374] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:57.067028 24174 raft_consensus.cc:348] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.067240 24174 raft_consensus.cc:374] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:57.067322 24175 raft_consensus.cc:729] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:57.067355 24174 raft_consensus.cc:729] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:57.067633 24174 consensus_queue.cc:260] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.067708 24175 consensus_queue.cc:260] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.067911 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 961) took 2305 ms. Trace:
I20240502 08:16:57.068003 24181 rpcz_store.cc:276] 0502 08:16:54.761951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.762000 (+ 49us) service_pool.cc:225] Handling call
0502 08:16:57.043190 (+2281190us) ts_tablet_manager.cc:1120] Deleting tablet b59c2f93d6514c9cbf358ab96f1a06b5
0502 08:16:57.043200 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.063743 (+ 20543us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.066390 (+ 2647us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.067886 (+ 1496us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2281035}
I20240502 08:16:57.068459 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 960) took 2306 ms. Trace:
I20240502 08:16:57.068609 24175 ts_tablet_manager.cc:1398] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
W20240502 08:16:57.071564 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c49f172372448c2be4909ae36371267 consensus state has no leader
I20240502 08:16:57.071974 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_891c4bb8dd8d4f1c8de1dc4eacb711a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.072530 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_891c4bb8dd8d4f1c8de1dc4eacb711a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.074769 24174 ts_tablet_manager.cc:1398] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 08:16:57.074998 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d3d08ffbf0a743a1ac0642fa87cf6f7e" } new_table_name: "default.Table_4621f8539a3d4b009ee5d89f54afd980" modify_external_catalogs: true
I20240502 08:16:57.068542 24181 rpcz_store.cc:276] 0502 08:16:54.761848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.762020 (+ 172us) service_pool.cc:225] Handling call
0502 08:16:57.068321 (+2306301us) ts_tablet_manager.cc:1120] Deleting tablet 2c28c47b4f814573971227b2208d1f83
0502 08:16:57.068333 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.068441 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2306072}
I20240502 08:16:57.082113 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 962) took 2314 ms. Trace:
I20240502 08:16:57.082204 24181 rpcz_store.cc:276] 0502 08:16:54.767264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.767328 (+ 64us) service_pool.cc:225] Handling call
0502 08:16:57.081965 (+2314637us) ts_tablet_manager.cc:1120] Deleting tablet 31775513f3a446b08c0aac25ae0cab20
0502 08:16:57.081978 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.082091 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2314466}
I20240502 08:16:57.082489 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 963) took 2315 ms. Trace:
I20240502 08:16:57.082547 24181 rpcz_store.cc:276] 0502 08:16:54.767367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.768032 (+ 665us) service_pool.cc:225] Handling call
0502 08:16:57.082399 (+2314367us) ts_tablet_manager.cc:1120] Deleting tablet 37b499184f31454c96f66cfdf9143d14
0502 08:16:57.082407 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.082475 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2314237}
I20240502 08:16:57.082798 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 970) took 2314 ms. Trace:
I20240502 08:16:57.082865 24181 rpcz_store.cc:276] 0502 08:16:54.767985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.777609 (+ 9624us) service_pool.cc:225] Handling call
0502 08:16:57.082712 (+2305103us) ts_tablet_manager.cc:1120] Deleting tablet 870edf582ab14ad1b57c4c4469b19954
0502 08:16:57.082720 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.082785 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2304921}
I20240502 08:16:57.083109 24181 tablet_replica.cc:330] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:57.083292 24181 raft_consensus.cc:2229] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:57.083633 24181 raft_consensus.cc:2258] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.088435 25494 raft_consensus.cc:482] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.088613 25494 raft_consensus.cc:504] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.088822 23603 master_service.cc:946] ReplaceTablet: received request to replace tablet af6efeebaebc495aba017a922e243c39 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:57.089277 25494 leader_election.cc:288] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:57.089670 24175 tablet_bootstrap.cc:492] T 9761df0bbdee44af90135bffb8b5828f P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:57.089823 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25c089f0c1b14e419f250d03d4eaad12" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:57.090005 24051 raft_consensus.cc:2454] T 25c089f0c1b14e419f250d03d4eaad12 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:57.090608 23861 leader_election.cc:302] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:57.091555 25494 raft_consensus.cc:2781] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:57.091658 25494 raft_consensus.cc:482] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:57.091769 25494 raft_consensus.cc:3037] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:57.093439 24181 ts_tablet_manager.cc:1875] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.093583 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec10ff7251a5442a8ecc1881b34db74d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:57.096855 23858 leader_election.cc:341] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 25c089f0c1b14e419f250d03d4eaad12
I20240502 08:16:57.097486 24175 tablet_bootstrap.cc:654] T 9761df0bbdee44af90135bffb8b5828f P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:57.097965 23996 leader_election.cc:341] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.098536 25423 consensus_queue.cc:1035] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:57.099617 23993 consensus_peers.cc:485] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fbef606ce7ac499d9bd84b49d804621c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:57.101680 25357 raft_consensus.cc:482] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.101805 25357 raft_consensus.cc:504] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.102147 25012 ts_tablet_manager.cc:1888] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:57.102262 25012 log.cc:1196] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/07344b78894c42809391d0fe2a9cd8f7
I20240502 08:16:57.102394 25357 leader_election.cc:288] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:57.102949 25012 ts_tablet_manager.cc:1909] T 07344b78894c42809391d0fe2a9cd8f7 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:57.103035 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090bec30f5664421b9bfe3f2810afd14" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:57.103307 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 898) took 3865 ms. Trace:
I20240502 08:16:57.103389 25012 rpcz_store.cc:276] 0502 08:16:53.238067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.238129 (+ 62us) service_pool.cc:225] Handling call
0502 08:16:57.031747 (+3793618us) ts_tablet_manager.cc:1120] Deleting tablet 07344b78894c42809391d0fe2a9cd8f7
0502 08:16:57.031757 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.053078 (+ 21321us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.102035 (+ 48957us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.103289 (+ 1254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3793458}
W20240502 08:16:57.103673 23861 leader_election.cc:341] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.104058 25012 tablet_replica.cc:330] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
W20240502 08:16:57.104146 23858 leader_election.cc:341] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 090bec30f5664421b9bfe3f2810afd14
I20240502 08:16:57.104262 23858 leader_election.cc:302] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:57.104310 25012 raft_consensus.cc:2229] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:57.104674 25423 raft_consensus.cc:482] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.104840 25357 raft_consensus.cc:2726] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:57.104815 25423 raft_consensus.cc:504] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.105432 25423 leader_election.cc:288] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:57.105597 25012 raft_consensus.cc:2258] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.106178 25012 ts_tablet_manager.cc:1875] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:57.106356 23861 consensus_peers.cc:485] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd3078cc1f6247609f8bd017ed076b45. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:16:57.109231 23993 leader_election.cc:341] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 18d284f71937459db0c0676ce4568175
I20240502 08:16:57.112241 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18d284f71937459db0c0676ce4568175" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:57.112697 23996 leader_election.cc:341] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.112788 23996 leader_election.cc:302] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:57.113153 25630 raft_consensus.cc:2726] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:57.113972 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 56d6373938824840934211f259e14d4f consensus state has no leader
I20240502 08:16:57.114614 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e8d89916bfe94d52af4f205db18f9e47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.115279 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8d89916bfe94d52af4f205db18f9e47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.115859 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 5d04312bd81f46ed8c49c42b1af1aec9 (table default.Table_9741459a96e649fb8151cc28b642523e [id=2b9169f285754bfda0f4f8dc7f5cb653]) successfully deleted
W20240502 08:16:57.116148 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet bb9b75d6be924d0291f94fc4a439dea6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb9b75d6be924d0291f94fc4a439dea6 already in progress: creating tablet
I20240502 08:16:57.116425 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet b59c2f93d6514c9cbf358ab96f1a06b5 (table default.Table_bbf9d7694fe942de8aa8bd4ed61336a5 [id=f1d6eb661b0143ee82e797cde61391c0]) successfully deleted
W20240502 08:16:57.117317 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 2c28c47b4f814573971227b2208d1f83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c28c47b4f814573971227b2208d1f83 already in progress: creating tablet
I20240502 08:16:57.118966 24181 ts_tablet_manager.cc:1888] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:57.119105 24181 log.cc:1196] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/955e7bd2216646c6aef5acce8d5b0d77
I20240502 08:16:57.119601 24181 ts_tablet_manager.cc:1909] T 955e7bd2216646c6aef5acce8d5b0d77 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:57.120586 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 975) took 2230 ms. Trace:
I20240502 08:16:57.120703 24181 rpcz_store.cc:276] 0502 08:16:54.889720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.889766 (+ 46us) service_pool.cc:225] Handling call
0502 08:16:57.083068 (+2193302us) ts_tablet_manager.cc:1120] Deleting tablet 955e7bd2216646c6aef5acce8d5b0d77
0502 08:16:57.083075 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.096458 (+ 13383us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.118861 (+ 22403us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.120560 (+ 1699us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2193064}
I20240502 08:16:57.121150 24181 tablet_replica.cc:330] T 78ddfc7b74644a0bbf8222d5f90e6658 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:57.121385 24181 raft_consensus.cc:2229] T 78ddfc7b74644a0bbf8222d5f90e6658 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:57.121598 24181 raft_consensus.cc:2258] T 78ddfc7b74644a0bbf8222d5f90e6658 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.122289 24181 ts_tablet_manager.cc:1875] T 78ddfc7b74644a0bbf8222d5f90e6658 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.124258 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 01143d14aca64adb846c8141ae41e6ac
I20240502 08:16:57.125149 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_3209c07b69e64b7a9f32f74fa58aef11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:57.126542 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 521581fe87e547ce816b599578bddeea
I20240502 08:16:57.127323 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a1b11bbb99a24f79b6d33fbf9cae4d5f" } new_table_name: "default.Table_129ded6f4dac43178e3f951a50366565" modify_external_catalogs: true
W20240502 08:16:57.131397 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3209c07b69e64b7a9f32f74fa58aef11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.134339 24173 tablet_bootstrap.cc:492] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:57.134531 24173 ts_tablet_manager.cc:1367] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.281s user 0.013s sys 0.029s
I20240502 08:16:57.135342 24173 raft_consensus.cc:348] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:16:57.135502 23434 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 122401us
I20240502 08:16:57.135571 24173 raft_consensus.cc:374] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:57.135663 24173 raft_consensus.cc:729] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:57.135865 24173 consensus_queue.cc:260] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.136555 24173 ts_tablet_manager.cc:1398] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:57.136868 24173 tablet_bootstrap.cc:492] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:57.138305 24173 tablet_bootstrap.cc:654] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:57.151177 25357 raft_consensus.cc:482] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.151324 25357 raft_consensus.cc:504] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.152009 25357 leader_election.cc:288] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:57.152129 25357 raft_consensus.cc:482] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.152244 25357 raft_consensus.cc:504] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:57.152789 25357 leader_election.cc:288] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:57.154109 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbef606ce7ac499d9bd84b49d804621c" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:57.157029 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df08f0d3af4c4c7e8abce656714b1b7f" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:57.158046 23858 leader_election.cc:341] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: fbef606ce7ac499d9bd84b49d804621c
I20240502 08:16:57.158159 23858 leader_election.cc:302] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:57.158389 25408 raft_consensus.cc:2726] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:16:57.158555 23858 leader_election.cc:341] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: df08f0d3af4c4c7e8abce656714b1b7f
I20240502 08:16:57.158641 23858 leader_election.cc:302] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:57.158926 25408 raft_consensus.cc:2726] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:57.159520 24181 ts_tablet_manager.cc:1888] T 78ddfc7b74644a0bbf8222d5f90e6658 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:57.159636 24181 log.cc:1196] T 78ddfc7b74644a0bbf8222d5f90e6658 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/78ddfc7b74644a0bbf8222d5f90e6658
I20240502 08:16:57.160058 24181 ts_tablet_manager.cc:1909] T 78ddfc7b74644a0bbf8222d5f90e6658 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:57.160379 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 974) took 2270 ms. Trace:
I20240502 08:16:57.160476 24181 rpcz_store.cc:276] 0502 08:16:54.889554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.890042 (+ 488us) service_pool.cc:225] Handling call
0502 08:16:57.121089 (+2231047us) ts_tablet_manager.cc:1120] Deleting tablet 78ddfc7b74644a0bbf8222d5f90e6658
0502 08:16:57.121102 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.156918 (+ 35816us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.159447 (+ 2529us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.160359 (+ 912us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2230813}
I20240502 08:16:57.160915 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 976) took 2017 ms. Trace:
I20240502 08:16:57.160997 24181 rpcz_store.cc:276] 0502 08:16:55.143334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.143535 (+ 201us) service_pool.cc:225] Handling call
0502 08:16:57.160784 (+2017249us) ts_tablet_manager.cc:1120] Deleting tablet e9941681b26d4c97b06a91155043b040
0502 08:16:57.160793 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.160887 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2017036}
I20240502 08:16:57.161311 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 977) took 1980 ms. Trace:
I20240502 08:16:57.161387 24181 rpcz_store.cc:276] 0502 08:16:55.180577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.180618 (+ 41us) service_pool.cc:225] Handling call
0502 08:16:57.161214 (+1980596us) ts_tablet_manager.cc:1120] Deleting tablet 89fefde55e514e5fabae3f4e21f44ca3
0502 08:16:57.161222 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.161295 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1980411}
W20240502 08:16:57.164238 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 31775513f3a446b08c0aac25ae0cab20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31775513f3a446b08c0aac25ae0cab20 already in progress: creating tablet
W20240502 08:16:57.164517 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 37b499184f31454c96f66cfdf9143d14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37b499184f31454c96f66cfdf9143d14 already in progress: creating tablet
W20240502 08:16:57.165710 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 870edf582ab14ad1b57c4c4469b19954 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 870edf582ab14ad1b57c4c4469b19954 already in progress: creating tablet
I20240502 08:16:57.165926 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 955e7bd2216646c6aef5acce8d5b0d77 (table default.Table_b3278c5d6d0b4593ac33bc9977f5c3b1 [id=e86031550cec46a793785c0b3859fa92]) successfully deleted
I20240502 08:16:57.166117 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 78ddfc7b74644a0bbf8222d5f90e6658 (table default.Table_0b7c860d457f421294ca1d37248e40b5 [id=c06501dd41a640279c769c9ab3926ad5]) successfully deleted
I20240502 08:16:57.166395 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_61b10a2ece3e45678e32119eea2f3c0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.166836 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e9941681b26d4c97b06a91155043b040 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9941681b26d4c97b06a91155043b040 already in progress: creating tablet
W20240502 08:16:57.167011 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 89fefde55e514e5fabae3f4e21f44ca3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89fefde55e514e5fabae3f4e21f44ca3 already in progress: creating tablet
I20240502 08:16:57.167187 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 77c49da1e0f0435ba241a6a66526e790 (table default.Table_8a9cdf1f53b0494694035cd549988bf0 [id=0de18f9c7e194e698b4f37603bfd4780]) successfully deleted
W20240502 08:16:57.167191 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61b10a2ece3e45678e32119eea2f3c0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.167992 24174 tablet_bootstrap.cc:492] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:57.100525 24180 ts_tablet_manager.cc:1888] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:57.168567 24180 log.cc:1196] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/edd7244b9f5741a8aa1c6835fe544676
I20240502 08:16:57.168881 24180 ts_tablet_manager.cc:1909] T edd7244b9f5741a8aa1c6835fe544676 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:57.126161 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet 01143d14aca64adb846c8141ae41e6ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.128933 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 521581fe87e547ce816b599578bddeea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.170572 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet edd7244b9f5741a8aa1c6835fe544676 (table default.Table_4c6989f7e6bb4e99b447a3c1188f5b92 [id=9775ed3c642a4d7e81d5abc48acca163]) successfully deleted
W20240502 08:16:57.170790 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 92b830e128ac439eba286f29dbb20ee4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92b830e128ac439eba286f29dbb20ee4 already in progress: creating tablet
W20240502 08:16:57.170976 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e5de31a3db014615989d9e018521bfc9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5de31a3db014615989d9e018521bfc9 already in progress: creating tablet
I20240502 08:16:57.135221 23835 heartbeater.cc:347] Connected to a master server at 127.22.225.125:36189
I20240502 08:16:57.143942 25531 raft_consensus.cc:482] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:16:57.171320 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 01143d14aca64adb846c8141ae41e6ac because the tablet was not found. No further retry: Not found: Tablet not found: 01143d14aca64adb846c8141ae41e6ac
I20240502 08:16:57.171317 25531 raft_consensus.cc:504] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.171940 25531 leader_election.cc:288] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:57.144577 23788 raft_consensus.cc:3037] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [term 2 FOLLOWER]: Advancing to term 3
W20240502 08:16:57.173012 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 521581fe87e547ce816b599578bddeea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 521581fe87e547ce816b599578bddeea already in progress: creating tablet
W20240502 08:16:57.173410 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d935b59497bc48a190061a6111e9ae1a consensus state has no leader
I20240502 08:16:57.176448 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a5a410b6314818b10927e35032f95d" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:57.176514 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a5a410b6314818b10927e35032f95d" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:57.181228 23723 leader_election.cc:341] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.181766 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_f3099e422df24ee593db8bca89aba612"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.182461 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3099e422df24ee593db8bca89aba612 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.182843 23788 raft_consensus.cc:1261] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [term 3 FOLLOWER]: Refusing update from remote peer 149ce744feab49b5aa5bcaa14bd6aba5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20240502 08:16:57.184079 23723 leader_election.cc:341] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.184193 23723 leader_election.cc:302] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:57.184284 25405 consensus_queue.cc:1035] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:16:57.184806 25531 raft_consensus.cc:2726] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:57.195000 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 4c395a29c7a1485eab39ca107dd91fca (table default.Table_1159b00d82f4483c8209c879e2d3c06a [id=272acbb9a6ca4f2dbcbfea6aacbe9cfc]). Expected version 4 got 0
I20240502 08:16:57.194649 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 979) took 1510 ms. Trace:
I20240502 08:16:57.195300 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 9761df0bbdee44af90135bffb8b5828f (table default.Table_8ba240c391da43978f93a27dce48e557 [id=1b5536950db244d1978b154da0be33e0]). Expected version 2 got 0
I20240502 08:16:57.195209 24181 rpcz_store.cc:276] 0502 08:16:55.686493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.686563 (+ 70us) service_pool.cc:225] Handling call
0502 08:16:57.193335 (+1506772us) ts_tablet_manager.cc:1120] Deleting tablet 0281c8123eab4c48bddc377f5695c6e0
0502 08:16:57.193508 (+ 173us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.194396 (+ 888us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1506550}
I20240502 08:16:57.199352 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 982) took 1506 ms. Trace:
W20240502 08:16:57.200932 23608 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.225.66:39077 (request call id 200) took 865 ms (client timeout 1000 ms). Trace:
W20240502 08:16:57.201023 23608 rpcz_store.cc:269] 0502 08:16:56.335583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.462252 (+126669us) service_pool.cc:225] Handling call
0502 08:16:57.200907 (+738655us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":8}
I20240502 08:16:57.201961 23606 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet a2da8b9d3f374d8a812336057c0ce7d2 (table default.Table_3edd27da6927478281e76361718425a3 [id=76b58ce209a24b9a95b405c1c50de3ca]). Expected version 3 got 0
W20240502 08:16:57.202972 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 336f0eb5085546dba763e0d5c5dd5324 consensus state has no leader
I20240502 08:16:57.201392 24181 rpcz_store.cc:276] 0502 08:16:55.692745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.693478 (+ 733us) service_pool.cc:225] Handling call
0502 08:16:57.197673 (+1504195us) ts_tablet_manager.cc:1120] Deleting tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6
0502 08:16:57.197684 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.199327 (+ 1643us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1503787}
W20240502 08:16:57.204185 23606 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.225.65:44121 (request call id 148) took 824 ms (client timeout 1000 ms). Trace:
W20240502 08:16:57.204278 23606 rpcz_store.cc:269] 0502 08:16:56.380057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.633405 (+253348us) service_pool.cc:225] Handling call
0502 08:16:57.204166 (+570761us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":5}
I20240502 08:16:57.206048 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 592ab8523b8d479d9caead321d5888c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.206305 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet ba3aa2621fdb43fd8299f81af486ef1e with delete_type TABLET_DATA_DELETED (replaced by tablet e20e126b1c9546e4a194bf9842f444a5) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.206569 24180 tablet_replica.cc:330] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:57.206791 24180 raft_consensus.cc:2229] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:57.207047 24180 pending_rounds.cc:62] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:16:57.207100 24180 pending_rounds.cc:68] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156497086758912 op_type: NO_OP noop_request { }
I20240502 08:16:57.207257 24180 raft_consensus.cc:2866] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:57.207343 24180 raft_consensus.cc:2258] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.207592 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet 592eeecbf5514215b6464bd030081e4b with delete_type TABLET_DATA_DELETED (replaced by tablet 28a947b7e7484120b3f4069f6eef5520) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.208531 24180 ts_tablet_manager.cc:1875] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.209142 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 983) took 1515 ms. Trace:
I20240502 08:16:57.210482 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_536141575d2c49fe807d39c1f38e3bb1" } modify_external_catalogs: true
I20240502 08:16:57.210686 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_536141575d2c49fe807d39c1f38e3bb1" } modify_external_catalogs: true
I20240502 08:16:57.212169 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0ca7df7eee144c08a8aa1c5cc0f008c4" } new_table_name: "default.Table_a32841ac077a46f99eaae2d807cc25a6" modify_external_catalogs: true
I20240502 08:16:57.210489 24181 rpcz_store.cc:276] 0502 08:16:55.692875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.693702 (+ 827us) service_pool.cc:225] Handling call
0502 08:16:57.205829 (+1512127us) ts_tablet_manager.cc:1120] Deleting tablet 39cc4ef1174a4358bad777b0471c5f76
0502 08:16:57.205840 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.208698 (+ 2858us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1511641}
I20240502 08:16:57.215087 23916 raft_consensus.cc:1261] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [term 3 FOLLOWER]: Refusing update from remote peer 149ce744feab49b5aa5bcaa14bd6aba5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 08:16:57.215687 25630 consensus_queue.cc:1035] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:16:57.218621 24180 ts_tablet_manager.cc:1888] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:57.218859 24180 log.cc:1196] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/592ab8523b8d479d9caead321d5888c3
I20240502 08:16:57.218698 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 980) took 1530 ms. Trace:
I20240502 08:16:57.221709 23917 raft_consensus.cc:3037] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:57.221972 23601 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.225.67:46497 (request call id 163) took 896 ms (client timeout 1000 ms). Trace:
W20240502 08:16:57.222050 23601 rpcz_store.cc:269] 0502 08:16:56.325373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.459571 (+134198us) service_pool.cc:225] Handling call
0502 08:16:57.221949 (+762378us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":10}
I20240502 08:16:57.220513 24181 rpcz_store.cc:276] 0502 08:16:55.688139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.688196 (+ 57us) service_pool.cc:225] Handling call
0502 08:16:57.215221 (+1527025us) ts_tablet_manager.cc:1120] Deleting tablet 558a3a4a8383439bb64af1d3eaa64572
0502 08:16:57.215232 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.218675 (+ 3443us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1518522}
I20240502 08:16:57.228472 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 988) took 1527 ms. Trace:
I20240502 08:16:57.231513 24181 rpcz_store.cc:276] 0502 08:16:55.701323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.701387 (+ 64us) service_pool.cc:225] Handling call
0502 08:16:57.226573 (+1525186us) ts_tablet_manager.cc:1120] Deleting tablet bb57cb00e884420bacc48f16739eb29c
0502 08:16:57.226910 (+ 337us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.228452 (+ 1542us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1524987}
I20240502 08:16:57.233952 25408 raft_consensus.cc:482] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.234107 25408 raft_consensus.cc:504] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.234779 25408 leader_election.cc:288] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:57.235424 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3aa2621fdb43fd8299f81af486ef1e" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:57.235671 24051 raft_consensus.cc:2379] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b89a1f0485a44b7ac1710e2369a809b in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:16:57.235951 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3aa2621fdb43fd8299f81af486ef1e" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:57.236670 23858 leader_election.cc:302] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:57.235450 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 989) took 1529 ms. Trace:
I20240502 08:16:57.237119 25408 raft_consensus.cc:3037] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:57.237000 24181 rpcz_store.cc:276] 0502 08:16:55.705223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.706309 (+ 1086us) service_pool.cc:225] Handling call
0502 08:16:57.234039 (+1527730us) ts_tablet_manager.cc:1120] Deleting tablet 36dece738afd42b99f3f38d7fd11a0af
0502 08:16:57.234343 (+ 304us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.235122 (+ 779us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1527486}
W20240502 08:16:57.238695 23858 consensus_peers.cc:485] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e7432f1df94431a83cf68356e3e78a3. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:57.240582 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 990) took 1533 ms. Trace:
I20240502 08:16:57.241137 24181 rpcz_store.cc:276] 0502 08:16:55.706863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.706910 (+ 47us) service_pool.cc:225] Handling call
0502 08:16:57.239574 (+1532664us) ts_tablet_manager.cc:1120] Deleting tablet dfd897ba97204b6788fd2d99cf2a0303
0502 08:16:57.239584 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.240270 (+ 686us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1532191}
I20240502 08:16:57.167395 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 07344b78894c42809391d0fe2a9cd8f7 (table default.Table_4c4a02d4a6eb4c87971a03d670f5fdaf [id=6b6ffe47b9534a618f0912d78e7bb92a]) successfully deleted
W20240502 08:16:57.242573 23597 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 75220.4us
I20240502 08:16:57.243376 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 01143d14aca64adb846c8141ae41e6ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.243489 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 995) took 1237 ms. Trace:
I20240502 08:16:57.244199 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 01143d14aca64adb846c8141ae41e6ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.243721 24181 rpcz_store.cc:276] 0502 08:16:56.006096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.006165 (+ 69us) service_pool.cc:225] Handling call
0502 08:16:57.242724 (+1236559us) ts_tablet_manager.cc:1120] Deleting tablet fc87f91093b94b209caa6f89691bcde3
0502 08:16:57.242899 (+ 175us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.243319 (+ 420us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1236340}
I20240502 08:16:57.244470 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7deeda2c4a754c3d8176faea44b99f1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.245118 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7deeda2c4a754c3d8176faea44b99f1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:57.247303 23723 consensus_peers.cc:485] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:57.249882 24180 ts_tablet_manager.cc:1909] T 592ab8523b8d479d9caead321d5888c3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:57.251272 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 521581fe87e547ce816b599578bddeea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.251708 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 521581fe87e547ce816b599578bddeea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.252439 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet df08f0d3af4c4c7e8abce656714b1b7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.253705 24181 tablet_replica.cc:330] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:57.253890 24181 raft_consensus.cc:2229] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:57.254112 24181 pending_rounds.cc:62] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5: Trying to abort 1 pending ops.
I20240502 08:16:57.254146 24181 pending_rounds.cc:68] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156493874683904 op_type: NO_OP noop_request { }
I20240502 08:16:57.254240 24181 raft_consensus.cc:2866] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:57.254304 24181 raft_consensus.cc:2258] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.254828 24181 ts_tablet_manager.cc:1875] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.258198 23601 catalog_manager.cc:5484] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67). New cstate: current_term: 1 leader_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:57.258958 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 4c395a29c7a1485eab39ca107dd91fca (table default.Table_1159b00d82f4483c8209c879e2d3c06a [id=272acbb9a6ca4f2dbcbfea6aacbe9cfc]). Expected version 4 got 0
I20240502 08:16:57.259506 23601 catalog_manager.cc:5484] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 reported cstate change: term changed from 1 to 3, leader changed from a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65) to 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67). New cstate: current_term: 3 leader_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:57.260767 23601 catalog_manager.cc:5484] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67). New cstate: current_term: 1 leader_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:57.261637 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet ec10ff7251a5442a8ecc1881b34db74d (table default.Table_236150d3903d437d95f67b0edec36091 [id=1d7a43116fe74c35b505eccd165f99cd]). Expected version 2 got 0
I20240502 08:16:57.262077 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 55f3f09fc1f94c00a47b9087b98e221d (table default.Table_7c74c34063104d54b3ab77c878892069 [id=106dd88de5eb49e89f0042beb7920736]). Expected version 2 got 0
I20240502 08:16:57.262403 25012 ts_tablet_manager.cc:1888] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:57.262466 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 001e3f4f72f64dc69dbd521724912f07 (table default.Table_d7cff78803d44e5aba3b50cd8ff11e92 [id=5c21117955d946418c656330984ae747]). Expected version 3 got 0
I20240502 08:16:57.262619 25012 log.cc:1196] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/1cd8338500f54604abcd6d42702c2eca
I20240502 08:16:57.262894 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet a2da8b9d3f374d8a812336057c0ce7d2 (table default.Table_3edd27da6927478281e76361718425a3 [id=76b58ce209a24b9a95b405c1c50de3ca]). Expected version 3 got 0
I20240502 08:16:57.263345 25012 ts_tablet_manager.cc:1909] T 1cd8338500f54604abcd6d42702c2eca P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:57.265094 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 899) took 4006 ms. Trace:
I20240502 08:16:57.266196 24174 tablet_bootstrap.cc:654] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:57.265254 25012 rpcz_store.cc:276] 0502 08:16:53.258734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.258984 (+ 250us) service_pool.cc:225] Handling call
0502 08:16:57.103887 (+3844903us) ts_tablet_manager.cc:1120] Deleting tablet 1cd8338500f54604abcd6d42702c2eca
0502 08:16:57.103897 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.255597 (+151700us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.262119 (+ 6522us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.264678 (+ 2559us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3844596}
I20240502 08:16:57.268435 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 900) took 4008 ms. Trace:
I20240502 08:16:57.268599 25012 rpcz_store.cc:276] 0502 08:16:53.260114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.260177 (+ 63us) service_pool.cc:225] Handling call
0502 08:16:57.267539 (+4007362us) ts_tablet_manager.cc:1120] Deleting tablet 2ebf55c8024c483f9ef0afe6945a02ff
0502 08:16:57.267717 (+ 178us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.267857 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4007203}
I20240502 08:16:57.269299 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 901) took 4008 ms. Trace:
I20240502 08:16:57.269500 25012 rpcz_store.cc:276] 0502 08:16:53.261166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.261266 (+ 100us) service_pool.cc:225] Handling call
0502 08:16:57.269120 (+4007854us) ts_tablet_manager.cc:1120] Deleting tablet 33bbebbca7464ea6ad59b81d7b7c560d
0502 08:16:57.269132 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.269282 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4007691}
I20240502 08:16:57.270717 25012 tablet_replica.cc:330] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:57.271194 25012 raft_consensus.cc:2229] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:57.271811 25012 raft_consensus.cc:2258] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.272956 25012 ts_tablet_manager.cc:1875] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.276819 24181 ts_tablet_manager.cc:1888] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:57.276984 24181 log.cc:1196] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/33c8131846dc41678827b86e87be18d3
I20240502 08:16:57.277302 24181 ts_tablet_manager.cc:1909] T 33c8131846dc41678827b86e87be18d3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:57.277607 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 997) took 1256 ms. Trace:
I20240502 08:16:57.277679 24181 rpcz_store.cc:276] 0502 08:16:56.020781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.020827 (+ 46us) service_pool.cc:225] Handling call
0502 08:16:57.253657 (+1232830us) ts_tablet_manager.cc:1120] Deleting tablet 33c8131846dc41678827b86e87be18d3
0502 08:16:57.253666 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.267061 (+ 13395us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.275490 (+ 8429us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.277569 (+ 2079us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1232665}
I20240502 08:16:57.277982 24181 tablet_replica.cc:330] T 6e7432f1df94431a83cf68356e3e78a3 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:57.278154 24181 raft_consensus.cc:2229] T 6e7432f1df94431a83cf68356e3e78a3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:57.278319 24181 raft_consensus.cc:2258] T 6e7432f1df94431a83cf68356e3e78a3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.278882 24181 ts_tablet_manager.cc:1875] T 6e7432f1df94431a83cf68356e3e78a3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.280761 25408 raft_consensus.cc:2726] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:57.287348 25413 consensus_queue.cc:1035] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:57.301406 24181 ts_tablet_manager.cc:1888] T 6e7432f1df94431a83cf68356e3e78a3 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:57.301527 24181 log.cc:1196] T 6e7432f1df94431a83cf68356e3e78a3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/6e7432f1df94431a83cf68356e3e78a3
I20240502 08:16:57.312336 24181 ts_tablet_manager.cc:1909] T 6e7432f1df94431a83cf68356e3e78a3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:57.313048 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 998) took 1292 ms. Trace:
I20240502 08:16:57.313124 24181 rpcz_store.cc:276] 0502 08:16:56.020864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.021022 (+ 158us) service_pool.cc:225] Handling call
0502 08:16:57.277939 (+1256917us) ts_tablet_manager.cc:1120] Deleting tablet 6e7432f1df94431a83cf68356e3e78a3
0502 08:16:57.277948 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.298452 (+ 20504us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.301314 (+ 2862us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.313029 (+ 11715us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1256740}
I20240502 08:16:57.313426 24181 tablet_replica.cc:330] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:57.313622 24181 raft_consensus.cc:2229] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:57.313897 24181 pending_rounds.cc:62] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5: Trying to abort 1 pending ops.
I20240502 08:16:57.313943 24181 pending_rounds.cc:68] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156494401957888 op_type: NO_OP noop_request { }
I20240502 08:16:57.314047 24181 raft_consensus.cc:2866] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:57.314119 24181 raft_consensus.cc:2258] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.314611 24181 ts_tablet_manager.cc:1875] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:57.304275 23993 consensus_peers.cc:485] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df08f0d3af4c4c7e8abce656714b1b7f. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:57.316582 25158 raft_consensus.cc:482] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.316731 25158 raft_consensus.cc:504] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.317337 25158 leader_election.cc:288] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:57.318233 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5f5597402374527bf0e5e50178df713" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:57.279007 23663 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 01143d14aca64adb846c8141ae41e6ac
I20240502 08:16:57.319047 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5f5597402374527bf0e5e50178df713" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:57.314705 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_44455df150c841b4b666c5279b4fe64d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:57.319684 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet 01143d14aca64adb846c8141ae41e6ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.316407 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 82c49bba3df548558123344a13c946ee
W20240502 08:16:57.319856 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44455df150c841b4b666c5279b4fe64d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.320540 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_474c796dbc574f53b4b9b2121710fb89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.320983 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_474c796dbc574f53b4b9b2121710fb89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:57.320971 23723 leader_election.cc:341] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:57.321332 23723 leader_election.cc:341] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.321431 23723 leader_election.cc:302] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:57.322173 25158 raft_consensus.cc:2726] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:57.322291 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet 82c49bba3df548558123344a13c946ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.324586 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 18d82e75f2624e6782f55987cea223cb
I20240502 08:16:57.325309 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d5691c9d792d49038cb518e00d54b2ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.325878 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5691c9d792d49038cb518e00d54b2ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.327404 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet 18d82e75f2624e6782f55987cea223cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:57.327853 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4f9e61686dc417bb5a88f96b4231f2e consensus state has no leader
I20240502 08:16:57.328233 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_23a2c8534b8b4671a4158faff46116a4" } new_table_name: "default.Table_a27bac2d73304d178179313a70c29d86" modify_external_catalogs: true
I20240502 08:16:57.334537 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 5f6ad372296c4204bda143a0a592643c (DEFAULT_TABLE table=default.Table_9e89353273c3429b9facd6433d847938 [id=12de5f5452674cb5a50110fdbac00ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.336205 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 5155ccf37d3e45bcb26740db910daec1 (DEFAULT_TABLE table=default.Table_a82f807957744746a06a4d90e90f0127 [id=a0c08ccaa36e4ebd80192a430d5b40c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.337249 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 638a4f142ff1461ab5462bdc18c505b4 (DEFAULT_TABLE table=default.Table_834d338053304767992670afbbddc67d [id=e309de7368894d7081cde447e888113f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.339097 23762 tablet_service.cc:1467] Processing CreateTablet for tablet d423a92750c94ee1a08688d60dd2168e (DEFAULT_TABLE table=default.Table_ca502551d3ba4c76868d9a79c06a8569 [id=71f7e0ccacc94451bf4cdc2ee7b6588f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.341421 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 25c089f0c1b14e419f250d03d4eaad12 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e56af7b42cb481a8363d60f49ee3f61) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.342329 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 955e7bd2216646c6aef5acce8d5b0d77 with delete_type TABLET_DATA_DELETED (replaced by tablet 4645143a00d440a7b6d56c5e30d249f1) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.343001 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 751808dfc1964b539f6f836d786156aa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.343830 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 577b149f52e44d5d9a9f6b9d85a6864d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.344485 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet e47459b5f5bb4ecb96f76240fefa1f5d with delete_type TABLET_DATA_DELETED (replaced by tablet 4917c1f0decf43c2aca74c67d8b45341) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.344982 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 18d284f71937459db0c0676ce4568175 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:57.345470 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 0281c8123eab4c48bddc377f5695c6e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0281c8123eab4c48bddc377f5695c6e0 already in progress: creating tablet
W20240502 08:16:57.345853 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6 already in progress: creating tablet
I20240502 08:16:57.346045 24181 ts_tablet_manager.cc:1888] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:57.346164 24181 log.cc:1196] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/fbef606ce7ac499d9bd84b49d804621c
W20240502 08:16:57.346280 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 39cc4ef1174a4358bad777b0471c5f76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39cc4ef1174a4358bad777b0471c5f76 already in progress: creating tablet
I20240502 08:16:57.346554 24173 tablet_bootstrap.cc:492] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
W20240502 08:16:57.346562 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 558a3a4a8383439bb64af1d3eaa64572 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 558a3a4a8383439bb64af1d3eaa64572 already in progress: creating tablet
I20240502 08:16:57.346822 24173 ts_tablet_manager.cc:1367] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.210s user 0.011s sys 0.044s
I20240502 08:16:57.347283 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b8d81f071d74b218fcfd4f4e8c20bab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.347573 23888 tablet_service.cc:1514] Processing DeleteTablet for tablet 25c089f0c1b14e419f250d03d4eaad12 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e56af7b42cb481a8363d60f49ee3f61) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.347625 24173 raft_consensus.cc:348] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.347893 24173 raft_consensus.cc:374] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:57.347977 24173 raft_consensus.cc:729] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:57.348253 24173 consensus_queue.cc:260] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.348820 24173 ts_tablet_manager.cc:1398] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:57.349007 25643 tablet.cc:1722] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:57.351907 24175 tablet_bootstrap.cc:492] T 9761df0bbdee44af90135bffb8b5828f P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:57.352051 24175 ts_tablet_manager.cc:1367] T 9761df0bbdee44af90135bffb8b5828f P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.262s user 0.008s sys 0.015s
I20240502 08:16:57.352042 25642 tablet.cc:1722] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:57.352946 24175 raft_consensus.cc:348] T 9761df0bbdee44af90135bffb8b5828f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:57.353153 24175 raft_consensus.cc:374] T 9761df0bbdee44af90135bffb8b5828f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:57.353211 24175 raft_consensus.cc:729] T 9761df0bbdee44af90135bffb8b5828f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:57.353417 24175 consensus_queue.cc:260] T 9761df0bbdee44af90135bffb8b5828f P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:57.353511 25645 tablet.cc:1722] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:57.354049 24175 ts_tablet_manager.cc:1398] T 9761df0bbdee44af90135bffb8b5828f P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:57.354300 24175 tablet_bootstrap.cc:492] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:57.355777 24175 tablet_bootstrap.cc:654] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:57.356670 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f6ad372296c4204bda143a0a592643c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.357013 24180 tablet_replica.cc:330] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:57.357197 24180 raft_consensus.cc:2229] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:57.357417 24180 pending_rounds.cc:62] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:16:57.357465 24180 pending_rounds.cc:68] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156498108870656 op_type: NO_OP noop_request { }
I20240502 08:16:57.357583 24180 raft_consensus.cc:2866] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:57.357663 24180 raft_consensus.cc:2258] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.361809 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5155ccf37d3e45bcb26740db910daec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.365785 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 638a4f142ff1461ab5462bdc18c505b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.346592 24181 ts_tablet_manager.cc:1909] T fbef606ce7ac499d9bd84b49d804621c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
W20240502 08:16:57.368820 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet bb57cb00e884420bacc48f16739eb29c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb57cb00e884420bacc48f16739eb29c already in progress: creating tablet
I20240502 08:16:57.368978 24180 ts_tablet_manager.cc:1875] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:57.369043 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 36dece738afd42b99f3f38d7fd11a0af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36dece738afd42b99f3f38d7fd11a0af already in progress: creating tablet
W20240502 08:16:57.369175 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet dfd897ba97204b6788fd2d99cf2a0303 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfd897ba97204b6788fd2d99cf2a0303 already in progress: creating tablet
W20240502 08:16:57.369307 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet fc87f91093b94b209caa6f89691bcde3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc87f91093b94b209caa6f89691bcde3 already in progress: creating tablet
I20240502 08:16:57.369441 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 33c8131846dc41678827b86e87be18d3 (table default.Table_bd9708a3da8240b4aa9444c8e9ed30d2 [id=5849a3c8b28e47439b73e2137187fac9]) successfully deleted
I20240502 08:16:57.369623 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 6e7432f1df94431a83cf68356e3e78a3 (table default.Table_a89a2cdb74684df5bcbe15389816c50b [id=04819651040e46c580d553290e3fa250]) successfully deleted
I20240502 08:16:57.369761 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d423a92750c94ee1a08688d60dd2168e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.371650 24173 tablet_bootstrap.cc:492] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:57.371948 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 999) took 1351 ms. Trace:
I20240502 08:16:57.372020 24181 rpcz_store.cc:276] 0502 08:16:56.020946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.021005 (+ 59us) service_pool.cc:225] Handling call
0502 08:16:57.313377 (+1292372us) ts_tablet_manager.cc:1120] Deleting tablet fbef606ce7ac499d9bd84b49d804621c
0502 08:16:57.313387 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.317548 (+ 4161us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.345930 (+ 28382us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.371929 (+ 25999us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1292090}
I20240502 08:16:57.372325 24181 tablet_replica.cc:330] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:57.372495 24181 raft_consensus.cc:2229] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:57.372633 24181 raft_consensus.cc:2258] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.373378 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 82c49bba3df548558123344a13c946ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.374125 24173 tablet_bootstrap.cc:654] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:57.374754 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 592ab8523b8d479d9caead321d5888c3 (table default.Table_af7c06dd8b3f4085b62eaef9ff9b57cb [id=11e47f3f89ff4bbbbb32ab105efe686c]) successfully deleted
I20240502 08:16:57.375056 24180 ts_tablet_manager.cc:1888] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:57.375414 24181 ts_tablet_manager.cc:1875] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.376600 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 18d82e75f2624e6782f55987cea223cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.377861 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 01143d14aca64adb846c8141ae41e6ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.378731 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 5f6ad372296c4204bda143a0a592643c (DEFAULT_TABLE table=default.Table_9e89353273c3429b9facd6433d847938 [id=12de5f5452674cb5a50110fdbac00ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.379076 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f6ad372296c4204bda143a0a592643c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.385488 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 5155ccf37d3e45bcb26740db910daec1 (DEFAULT_TABLE table=default.Table_a82f807957744746a06a4d90e90f0127 [id=a0c08ccaa36e4ebd80192a430d5b40c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.385963 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5155ccf37d3e45bcb26740db910daec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.391052 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 638a4f142ff1461ab5462bdc18c505b4 (DEFAULT_TABLE table=default.Table_834d338053304767992670afbbddc67d [id=e309de7368894d7081cde447e888113f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.391414 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 638a4f142ff1461ab5462bdc18c505b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.375356 24180 log.cc:1196] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/ba3aa2621fdb43fd8299f81af486ef1e
I20240502 08:16:57.397496 24180 ts_tablet_manager.cc:1909] T ba3aa2621fdb43fd8299f81af486ef1e P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:57.402354 24180 tablet_replica.cc:330] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:57.402513 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet ba3aa2621fdb43fd8299f81af486ef1e (table default.Table_4481e3bc8b3548b0a7ecef01d39f80e8 [id=f8b12b3bb4904c18be60b567cb05b739]) successfully deleted
I20240502 08:16:57.403155 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 1cd8338500f54604abcd6d42702c2eca (table default.Table_482a1b0b7a064e02805ac3b86fc62f99 [id=3c1d614616374739aeb928e797479df4]) successfully deleted
I20240502 08:16:57.405110 24180 raft_consensus.cc:2229] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240502 08:16:57.405171 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 2ebf55c8024c483f9ef0afe6945a02ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ebf55c8024c483f9ef0afe6945a02ff already in progress: creating tablet
I20240502 08:16:57.405714 24180 raft_consensus.cc:2258] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:57.408236 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 33bbebbca7464ea6ad59b81d7b7c560d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33bbebbca7464ea6ad59b81d7b7c560d already in progress: creating tablet
I20240502 08:16:57.407331 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 68d8126bbc1d4710904baeddaaf493f2 (DEFAULT_TABLE table=default.Table_96b57ae1050b4b6e819e085ac1d3c79c [id=fd74c0f02d3743178fcd5bc92edd5758]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.410099 24035 tablet_service.cc:1467] Processing CreateTablet for tablet d423a92750c94ee1a08688d60dd2168e (DEFAULT_TABLE table=default.Table_ca502551d3ba4c76868d9a79c06a8569 [id=71f7e0ccacc94451bf4cdc2ee7b6588f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.410543 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d423a92750c94ee1a08688d60dd2168e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.410676 24180 ts_tablet_manager.cc:1875] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.412611 24174 tablet_bootstrap.cc:492] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:57.412760 24174 ts_tablet_manager.cc:1367] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.245s user 0.001s sys 0.003s
I20240502 08:16:57.413427 24174 raft_consensus.cc:348] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:57.413666 24174 raft_consensus.cc:374] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:57.413748 24174 raft_consensus.cc:729] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:57.413969 24174 consensus_queue.cc:260] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:57.416435 24174 ts_tablet_manager.cc:1398] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:16:57.416740 24174 tablet_bootstrap.cc:492] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:57.419157 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet fbef606ce7ac499d9bd84b49d804621c (table default.Table_1e7f2516f34b483d913cf5080a8febec [id=d5c976b2debc472fa1d7c28ae73ca32a]) successfully deleted
I20240502 08:16:57.421588 24174 tablet_bootstrap.cc:654] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:57.422502 24181 ts_tablet_manager.cc:1888] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:57.422650 24181 log.cc:1196] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/c00f29fb76b44d6ea4f76963d2d0751c
I20240502 08:16:57.422969 25630 raft_consensus.cc:482] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.423074 24181 ts_tablet_manager.cc:1909] T c00f29fb76b44d6ea4f76963d2d0751c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:57.423117 25630 raft_consensus.cc:504] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.423767 25630 leader_election.cc:288] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:57.424319 23993 leader_election.cc:341] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 090bec30f5664421b9bfe3f2810afd14
I20240502 08:16:57.429278 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68d8126bbc1d4710904baeddaaf493f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.433092 24173 tablet_bootstrap.cc:492] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:57.433245 24173 ts_tablet_manager.cc:1367] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.002s
I20240502 08:16:57.434240 24180 ts_tablet_manager.cc:1888] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:57.434326 24180 log.cc:1196] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/592eeecbf5514215b6464bd030081e4b
I20240502 08:16:57.434649 24180 ts_tablet_manager.cc:1909] T 592eeecbf5514215b6464bd030081e4b P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:57.435626 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 592eeecbf5514215b6464bd030081e4b (table default.Table_4191e0f787d948bda654f8bbb6d8d24e [id=07050bd496e84a97a86fdc4f0b7811ac]) successfully deleted
I20240502 08:16:57.435608 24173 raft_consensus.cc:348] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:16:57.436087 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 01143d14aca64adb846c8141ae41e6ac because the tablet was not found. No further retry: Not found: Tablet not found: 01143d14aca64adb846c8141ae41e6ac
W20240502 08:16:57.436291 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 82c49bba3df548558123344a13c946ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82c49bba3df548558123344a13c946ee already in progress: creating tablet
I20240502 08:16:57.436569 24173 raft_consensus.cc:374] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 08:16:57.436690 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 18d82e75f2624e6782f55987cea223cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18d82e75f2624e6782f55987cea223cb already in progress: creating tablet
I20240502 08:16:57.436870 24173 raft_consensus.cc:729] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:57.436914 25114 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38208 (request call id 2) took 6424 ms. Trace:
I20240502 08:16:57.438576 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 68d8126bbc1d4710904baeddaaf493f2 (DEFAULT_TABLE table=default.Table_96b57ae1050b4b6e819e085ac1d3c79c [id=fd74c0f02d3743178fcd5bc92edd5758]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.439021 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68d8126bbc1d4710904baeddaaf493f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.438943 23759 tablet_service.cc:1467] Processing CreateTablet for tablet ba4f43b8caeb4d83a2ae45df0448597c (DEFAULT_TABLE table=default.Table_b8657ee1f37f41e6a9ac770b3c4fdf73 [id=dc1d2bc00e3c43449412ea5556c58845]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.439329 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba4f43b8caeb4d83a2ae45df0448597c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.440618 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1000) took 1419 ms. Trace:
I20240502 08:16:57.440733 24181 rpcz_store.cc:276] 0502 08:16:56.021031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.021300 (+ 269us) service_pool.cc:225] Handling call
0502 08:16:57.372283 (+1350983us) ts_tablet_manager.cc:1120] Deleting tablet c00f29fb76b44d6ea4f76963d2d0751c
0502 08:16:57.372292 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.407818 (+ 35526us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.422425 (+ 14607us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.440556 (+ 18131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1350846}
I20240502 08:16:57.441705 24173 consensus_queue.cc:260] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.443677 24173 ts_tablet_manager.cc:1398] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240502 08:16:57.444147 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet c00f29fb76b44d6ea4f76963d2d0751c (table default.Table_c2a25d0d51cd40769e5045ff80c9c3fc [id=a1004106109d47e7a1f9c6337480d3c0]) successfully deleted
I20240502 08:16:57.444079 24181 tablet_replica.cc:330] T 128c88af54d24ffd895e9de19cdb4c8a P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:57.446975 24181 raft_consensus.cc:2229] T 128c88af54d24ffd895e9de19cdb4c8a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:57.447588 24181 raft_consensus.cc:2258] T 128c88af54d24ffd895e9de19cdb4c8a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.437032 25584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f87df6a45ad94428a9aa6b2796a0fd25. Latest consensus state: current_term: 1 leader_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } } }
I20240502 08:16:57.447899 25584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [sys.catalog]: This master's current role is: FOLLOWER
I20240502 08:16:57.437342 25637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f87df6a45ad94428a9aa6b2796a0fd25)
I20240502 08:16:57.448493 25637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:57.454310 24181 ts_tablet_manager.cc:1875] T 128c88af54d24ffd895e9de19cdb4c8a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.456641 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1c914a8e055b422e8029a6a02b4a2925"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.457262 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c914a8e055b422e8029a6a02b4a2925 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.457810 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet af6efeebaebc495aba017a922e243c39
I20240502 08:16:57.458263 23603 catalog_manager.cc:6554] ReplaceTablet: tablet af6efeebaebc495aba017a922e243c39 deleted and replaced by tablet 9e13c0b81dc24b8ba1a88b31bf510efd
I20240502 08:16:57.458600 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ae314c4445fb416f97e233054cd2cb3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.459065 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae314c4445fb416f97e233054cd2cb3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.461866 25637 leader_election.cc:288] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcc8a323e6b9429eaaf4e0c1698ad077 (127.22.225.126:43265), f87df6a45ad94428a9aa6b2796a0fd25 (127.22.225.124:41745)
I20240502 08:16:57.467969 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet b054783710824dbc8f7bc3ea38f605b0
I20240502 08:16:57.468819 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_55a033a602c24586bc4d6d660749a0fe" } new_table_name: "default.Table_2ee0f7ddd41e41ea84e25359cd3ed0a4" modify_external_catalogs: true
I20240502 08:16:57.471115 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 66ea6743a3a44f9b95e1ca3d50a3dd40 (DEFAULT_TABLE table=default.Table_ac7f78ed7ac94104a09a418eed600e2d [id=5873e9784ac24716972b4fb9245e5331]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.471191 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet b054783710824dbc8f7bc3ea38f605b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.471498 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ea6743a3a44f9b95e1ca3d50a3dd40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.471767 23751 tablet_service.cc:1467] Processing CreateTablet for tablet f03473f7db3e411186f0db827a6630f7 (DEFAULT_TABLE table=default.Table_26889a40ce5a43bcbda607dcda6887bf [id=83e0466edd37473d9a9dfa8f0f8eed38]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.472100 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f03473f7db3e411186f0db827a6630f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.476464 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_52e9e35e99234042ae5e823f448c5cb7" } modify_external_catalogs: true
I20240502 08:16:57.476634 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_52e9e35e99234042ae5e823f448c5cb7" } modify_external_catalogs: true
I20240502 08:16:57.476773 23749 tablet_service.cc:1467] Processing CreateTablet for tablet aecc2a722985421da96565cb895875a2 (DEFAULT_TABLE table=default.Table_fdc1e83837dc40ca99542547bc5fa957 [id=9607b22df6a04fbc940db9bed2291a66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.476955 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_78883cd551c74a68b0e2a4dfe40e6955" } modify_external_catalogs: true
I20240502 08:16:57.477087 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_78883cd551c74a68b0e2a4dfe40e6955" } modify_external_catalogs: true
I20240502 08:16:57.477144 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecc2a722985421da96565cb895875a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.482427 23608 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet ec10ff7251a5442a8ecc1881b34db74d (table default.Table_236150d3903d437d95f67b0edec36091 [id=1d7a43116fe74c35b505eccd165f99cd]). Expected version 2 got 0
I20240502 08:16:57.482606 23608 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 74b4c563d6bd4a44bb3bab58a3fcdc07 (table default.Table_55a033a602c24586bc4d6d660749a0fe [id=a9c009feee0e486794a8500121fe5487]). Expected version 1 got 0
I20240502 08:16:57.482743 23608 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 4c395a29c7a1485eab39ca107dd91fca (table default.Table_1159b00d82f4483c8209c879e2d3c06a [id=272acbb9a6ca4f2dbcbfea6aacbe9cfc]). Expected version 4 got 0
I20240502 08:16:57.482976 23608 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 63afd6e4fee24edeaba1209105f1c3e6 (table default.Table_9f3ea2fdcdc2477b89d509e868ffe4de [id=84e36915119b4125a1b0d64b2d1647fa]). Expected version 1 got 0
I20240502 08:16:57.483083 23608 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 55f3f09fc1f94c00a47b9087b98e221d (table default.Table_7c74c34063104d54b3ab77c878892069 [id=106dd88de5eb49e89f0042beb7920736]). Expected version 2 got 0
I20240502 08:16:57.483196 23608 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 0501575a84ec454780f48574bf3155fe (table default.Table_791a010a6e484e10a6749eefcfdc293d [id=5cee878b3a31426c879a1bf95045bb09]). Expected version 3 got 0
I20240502 08:16:57.483579 23764 tablet_service.cc:1467] Processing CreateTablet for tablet a015eaae830549769bd1561684b04bca (DEFAULT_TABLE table=default.Table_2ec1dfc748994bfe8841412d61167608 [id=e0ea2320ed6347bca9de4d9615d3f053]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.483340 23608 catalog_manager.cc:5484] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:57.483960 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a015eaae830549769bd1561684b04bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.484021 23608 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 9761df0bbdee44af90135bffb8b5828f (table default.Table_8ba240c391da43978f93a27dce48e557 [id=1b5536950db244d1978b154da0be33e0]). Expected version 2 got 0
I20240502 08:16:57.484195 23608 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet a2da8b9d3f374d8a812336057c0ce7d2 (table default.Table_3edd27da6927478281e76361718425a3 [id=76b58ce209a24b9a95b405c1c50de3ca]). Expected version 3 got 0
I20240502 08:16:57.484153 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 3519b9c0ff2546c882ac9a92add72b40 (DEFAULT_TABLE table=default.Table_38d10d7d3ecf4f89accb1c4dbfebdbea [id=3834a986d4d844bf919e00778ec96047]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.483585 23752 tablet_service.cc:1467] Processing CreateTablet for tablet f244c446b2764ae89d5bc22d1e3ab456 (DEFAULT_TABLE table=default.Table_cc0830cedb8b4372bd149e16c0548861 [id=f8cdaacdac1b4752991090c9f88b81ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.484704 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 5abdb61c01124d1d8c82fff183640ffb (DEFAULT_TABLE table=default.Table_ae2ecda79c0245b59ef76809b5e3cdcc [id=f7401987176b428eb3355e3e29b5bebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.484994 23608 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 001e3f4f72f64dc69dbd521724912f07 (table default.Table_d7cff78803d44e5aba3b50cd8ff11e92 [id=5c21117955d946418c656330984ae747]). Expected version 3 got 0
I20240502 08:16:57.485188 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 37e04d50c60b4856a79217c00f63d0c7 (DEFAULT_TABLE table=default.Table_1b391fb1270447ed9134fcc3d288a9e3 [id=4321b452ff8c4646bef6c86930ecd313]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.485706 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 852486950b4144daa5cbd0644dbe1a41 (DEFAULT_TABLE table=default.Table_9cb7a43194434b49ad817e7876631133 [id=d79937be8924416094c969bf6e8ba27d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.485769 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 01143d14aca64adb846c8141ae41e6ac (DEFAULT_TABLE table=default.Table_825b08eccbb0430395675758faf7b2f8 [id=f586150dea1a4efeae35a1307ad51ef1]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:57.486063 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet b054783710824dbc8f7bc3ea38f605b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b054783710824dbc8f7bc3ea38f605b0 already in progress: creating tablet
I20240502 08:16:57.486119 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 0a1b534c89aa4d839c220a593a753c23 (DEFAULT_TABLE table=default.Table_2d4cb7ffff364fe6becc190a0dadfb25 [id=f3844c1a455e435a90e9e7e1f625739a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.484470 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3519b9c0ff2546c882ac9a92add72b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.486583 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 6e42383235df4f219aadc2d2e42045ac (DEFAULT_TABLE table=default.Table_cf7ced0046f647eb82829af1fb7c985b [id=187304972ddc45d49cd15953974c6f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.486855 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f244c446b2764ae89d5bc22d1e3ab456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.487026 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01143d14aca64adb846c8141ae41e6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.487087 23753 tablet_service.cc:1467] Processing CreateTablet for tablet eaac3bd5aa674422b7feb85ed9db1f70 (DEFAULT_TABLE table=default.Table_b0b35fed485c4223a4710de9abfd86fb [id=b863ddadc61c42d5a3fc5a1621cfb7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.487432 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaac3bd5aa674422b7feb85ed9db1f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.489665 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a1b534c89aa4d839c220a593a753c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.489898 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5abdb61c01124d1d8c82fff183640ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.493439 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e42383235df4f219aadc2d2e42045ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.495522 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e04d50c60b4856a79217c00f63d0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.498172 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet af6efeebaebc495aba017a922e243c39 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e13c0b81dc24b8ba1a88b31bf510efd) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.498415 24180 tablet_replica.cc:330] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:57.498597 24180 raft_consensus.cc:2229] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:57.498739 24180 raft_consensus.cc:2258] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.499265 24180 ts_tablet_manager.cc:1875] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.500782 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 852486950b4144daa5cbd0644dbe1a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.486583 23762 tablet_service.cc:1467] Processing CreateTablet for tablet f239cbee895542ed9fb5431ef13af0fa (DEFAULT_TABLE table=default.Table_b04d67567cc04ca49aeb83ae20849d4c [id=2716856a55f5460c971bc4ef1986da5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.505841 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f239cbee895542ed9fb5431ef13af0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.507586 25651 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20240502 08:16:57.513713 24180 ts_tablet_manager.cc:1888] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:57.518115 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_873744f861a94cc8af0fb5c3c80c944d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.518661 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_873744f861a94cc8af0fb5c3c80c944d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:57.519848 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 665bbee2d15b4112b651844431535d0e consensus state has no leader
I20240502 08:16:57.520185 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ff75818c766d43569443cacbfcb972c1" } new_table_name: "default.Table_fcef9af155ad4443b42d743e4046b1d8" modify_external_catalogs: true
I20240502 08:16:57.522995 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0d79aa9695cf4de7b5ff7249dcab2669"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.523382 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d79aa9695cf4de7b5ff7249dcab2669 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.365366 25012 ts_tablet_manager.cc:1888] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:57.367074 25408 raft_consensus.cc:482] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.529718 25408 raft_consensus.cc:504] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.530445 25408 leader_election.cc:288] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:57.368176 25357 raft_consensus.cc:482] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.530826 25357 raft_consensus.cc:504] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.531363 25357 leader_election.cc:288] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:57.372925 23888 tablet_service.cc:1514] Processing DeleteTablet for tablet 01143d14aca64adb846c8141ae41e6ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.375197 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 82c49bba3df548558123344a13c946ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.376296 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 18d82e75f2624e6782f55987cea223cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.378501 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 5f6ad372296c4204bda143a0a592643c (DEFAULT_TABLE table=default.Table_9e89353273c3429b9facd6433d847938 [id=12de5f5452674cb5a50110fdbac00ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.532384 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f6ad372296c4204bda143a0a592643c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.385236 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 5155ccf37d3e45bcb26740db910daec1 (DEFAULT_TABLE table=default.Table_a82f807957744746a06a4d90e90f0127 [id=a0c08ccaa36e4ebd80192a430d5b40c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.390313 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 638a4f142ff1461ab5462bdc18c505b4 (DEFAULT_TABLE table=default.Table_834d338053304767992670afbbddc67d [id=e309de7368894d7081cde447e888113f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.399801 23920 raft_consensus.cc:3037] T a2da8b9d3f374d8a812336057c0ce7d2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:57.424394 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090bec30f5664421b9bfe3f2810afd14" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:57.532698 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5155ccf37d3e45bcb26740db910daec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.532748 23917 raft_consensus.cc:2454] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:57.532907 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 638a4f142ff1461ab5462bdc18c505b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.540853 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet c84b27ed8cf7429ca720e4cb8d396342
I20240502 08:16:57.541687 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 9a9676602d1c4b37b3f4126a29f5456f
I20240502 08:16:57.533495 25012 log.cc:1196] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/6e7432f1df94431a83cf68356e3e78a3
I20240502 08:16:57.520145 25531 consensus_queue.cc:1035] T 9761df0bbdee44af90135bffb8b5828f P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:57.521759 24174 tablet_bootstrap.cc:492] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:57.545229 25012 ts_tablet_manager.cc:1909] T 6e7432f1df94431a83cf68356e3e78a3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:57.545359 24174 ts_tablet_manager.cc:1367] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.129s user 0.003s sys 0.000s
I20240502 08:16:57.545608 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 902) took 4283 ms. Trace:
I20240502 08:16:57.545779 25012 rpcz_store.cc:276] 0502 08:16:53.262113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.262336 (+ 223us) service_pool.cc:225] Handling call
0502 08:16:57.270635 (+4008299us) ts_tablet_manager.cc:1120] Deleting tablet 6e7432f1df94431a83cf68356e3e78a3
0502 08:16:57.270648 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.362664 (+ 92016us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.365277 (+ 2613us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.545588 (+180311us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4008114}
I20240502 08:16:57.546259 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 907) took 4024 ms. Trace:
I20240502 08:16:57.546135 24174 raft_consensus.cc:348] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:57.546468 24174 raft_consensus.cc:374] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:57.546391 25012 rpcz_store.cc:276] 0502 08:16:53.521841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.523840 (+ 1999us) service_pool.cc:225] Handling call
0502 08:16:57.546135 (+4022295us) ts_tablet_manager.cc:1120] Deleting tablet 418ab502915c4e4ea9998dd2d0180c48
0502 08:16:57.546145 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.546243 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4022093}
I20240502 08:16:57.546625 24174 raft_consensus.cc:729] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:57.546725 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 917) took 4015 ms. Trace:
I20240502 08:16:57.544508 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a9676602d1c4b37b3f4126a29f5456f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.546893 24174 consensus_queue.cc:260] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:57.547217 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet c84b27ed8cf7429ca720e4cb8d396342 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.549237 24174 ts_tablet_manager.cc:1398] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:57.549530 24174 tablet_bootstrap.cc:492] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:57.546511 24180 log.cc:1196] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/af6efeebaebc495aba017a922e243c39
I20240502 08:16:57.551410 24180 ts_tablet_manager.cc:1909] T af6efeebaebc495aba017a922e243c39 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:57.553360 24174 tablet_bootstrap.cc:654] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:57.553310 23861 consensus_peers.cc:485] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:57.553478 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet af6efeebaebc495aba017a922e243c39 (table default.Table_82f82cd6e702420283c288c10f905d90 [id=f293b22818344aea9eb6232141432183]) successfully deleted
W20240502 08:16:57.553843 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 9a9676602d1c4b37b3f4126a29f5456f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a9676602d1c4b37b3f4126a29f5456f already in progress: creating tablet
W20240502 08:16:57.554143 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet c84b27ed8cf7429ca720e4cb8d396342 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c84b27ed8cf7429ca720e4cb8d396342 already in progress: creating tablet
I20240502 08:16:57.556190 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet b054783710824dbc8f7bc3ea38f605b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.560684 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 37e04d50c60b4856a79217c00f63d0c7 (DEFAULT_TABLE table=default.Table_1b391fb1270447ed9134fcc3d288a9e3 [id=4321b452ff8c4646bef6c86930ecd313]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.562603 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 6e42383235df4f219aadc2d2e42045ac (DEFAULT_TABLE table=default.Table_cf7ced0046f647eb82829af1fb7c985b [id=187304972ddc45d49cd15953974c6f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.562532 24434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1055 } } ignore_live_leader: false dest_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" is_pre_election: true
I20240502 08:16:57.566658 25531 raft_consensus.cc:482] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.566797 25531 raft_consensus.cc:504] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:57.567958 25531 leader_election.cc:288] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:57.572229 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b0d4871dd24607854100a4ef762e25" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:57.572922 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 66ea6743a3a44f9b95e1ca3d50a3dd40 (DEFAULT_TABLE table=default.Table_ac7f78ed7ac94104a09a418eed600e2d [id=5873e9784ac24716972b4fb9245e5331]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.573266 24038 tablet_service.cc:1467] Processing CreateTablet for tablet aecc2a722985421da96565cb895875a2 (DEFAULT_TABLE table=default.Table_fdc1e83837dc40ca99542547bc5fa957 [id=9607b22df6a04fbc940db9bed2291a66]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:57.573858 23861 leader_election.cc:341] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.578779 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 0a1b534c89aa4d839c220a593a753c23 (DEFAULT_TABLE table=default.Table_2d4cb7ffff364fe6becc190a0dadfb25 [id=f3844c1a455e435a90e9e7e1f625739a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.579095 24021 tablet_service.cc:1467] Processing CreateTablet for tablet 852486950b4144daa5cbd0644dbe1a41 (DEFAULT_TABLE table=default.Table_9cb7a43194434b49ad817e7876631133 [id=d79937be8924416094c969bf6e8ba27d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:57.587697 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74b4c563d6bd4a44bb3bab58a3fcdc07 consensus state has no leader
I20240502 08:16:57.588866 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a82f807957744746a06a4d90e90f0127" } new_table_name: "default.Table_85e331a91f5644cca2bd22974f75dc18" modify_external_catalogs: true
I20240502 08:16:57.589095 23604 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 7ed919909289410e867881baab841f53 (table default.Table_2edad641920b44ce9a4d5b7503f0279a [id=ee3912407a934444ad2c62f7b83d93ac]). Expected version 2 got 0
I20240502 08:16:57.589387 23604 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet a2da8b9d3f374d8a812336057c0ce7d2 (table default.Table_3edd27da6927478281e76361718425a3 [id=76b58ce209a24b9a95b405c1c50de3ca]). Expected version 3 got 0
I20240502 08:16:57.589510 23604 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 001e3f4f72f64dc69dbd521724912f07 (table default.Table_d7cff78803d44e5aba3b50cd8ff11e92 [id=5c21117955d946418c656330984ae747]). Expected version 3 got 0
I20240502 08:16:57.590209 23604 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet e0b0d4871dd24607854100a4ef762e25 (table default.Table_5301ffccbac743618a78289c2c1415e1 [id=910c7cae36264e39bbfca0be919538e0]). Expected version 1 got 0
I20240502 08:16:57.590364 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_74a4ec7454a14f74ad2c1dc97fab2080" } modify_external_catalogs: true
I20240502 08:16:57.590550 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_74a4ec7454a14f74ad2c1dc97fab2080" } modify_external_catalogs: true
I20240502 08:16:57.590210 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_eccfd1a3ea3d4ab286b931f657c62607"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:57.589766 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1692280f84c44944b308c28c5918f6f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.592072 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eccfd1a3ea3d4ab286b931f657c62607 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:57.592255 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1692280f84c44944b308c28c5918f6f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.593088 23601 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_876663d4e8ac4320920b2d42fe658578 [id=8521bfc13ef040a78cfb2cc4501b7f04] alter complete (version 6)
I20240502 08:16:57.595326 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_86cb4c92da184779acc53787cbce4b4d" } modify_external_catalogs: true
I20240502 08:16:57.595489 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_86cb4c92da184779acc53787cbce4b4d" } modify_external_catalogs: true
I20240502 08:16:57.595767 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_adc39e4ac3924f079897f27ed4f1981f" } new_table_name: "default.Table_8c480b67753443e387de1afce385add6" modify_external_catalogs: true
I20240502 08:16:57.598531 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e645bb6e2948e091e08433944a52f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.598814 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet a4d151770ee44130adfd822518b9b017 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.599025 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 92adebc0c69e4a52b975253656132495 with delete_type TABLET_DATA_DELETED (replaced by tablet ae2b20ae3a014c85af33d5ee51b704e1) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.599229 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet 592ab8523b8d479d9caead321d5888c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.599462 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet e5f5597402374527bf0e5e50178df713 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.599648 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet ba3aa2621fdb43fd8299f81af486ef1e with delete_type TABLET_DATA_DELETED (replaced by tablet e20e126b1c9546e4a194bf9842f444a5) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.599828 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet a5a5a410b6314818b10927e35032f95d with delete_type TABLET_DATA_DELETED (replaced by tablet 92b830e128ac439eba286f29dbb20ee4) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.600139 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f1ab141585441db46d3027833932b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.546875 25012 rpcz_store.cc:276] 0502 08:16:53.531480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.531524 (+ 44us) service_pool.cc:225] Handling call
0502 08:16:57.546621 (+4015097us) ts_tablet_manager.cc:1120] Deleting tablet df87b8b1ff504031bf007880792192dd
0502 08:16:57.546633 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.546711 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4014933}
I20240502 08:16:57.600822 23608 master_service.cc:946] ReplaceTablet: received request to replace tablet 523603a31b5b49e0a18a5a64f1357d64 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:57.601198 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 916) took 4069 ms. Trace:
I20240502 08:16:57.601279 25012 rpcz_store.cc:276] 0502 08:16:53.531383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.532131 (+ 748us) service_pool.cc:225] Handling call
0502 08:16:57.601067 (+4068936us) ts_tablet_manager.cc:1120] Deleting tablet 40fab016ed4f489aa3ccdeafcfbe1477
0502 08:16:57.601078 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.601182 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4068774}
I20240502 08:16:57.602478 25357 raft_consensus.cc:482] T 9761df0bbdee44af90135bffb8b5828f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.602612 25357 raft_consensus.cc:504] T 9761df0bbdee44af90135bffb8b5828f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:57.603242 25357 leader_election.cc:288] T 9761df0bbdee44af90135bffb8b5828f P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:57.604600 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 921) took 4046 ms. Trace:
I20240502 08:16:57.604727 25012 rpcz_store.cc:276] 0502 08:16:53.558318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.558367 (+ 49us) service_pool.cc:225] Handling call
0502 08:16:57.601496 (+4043129us) ts_tablet_manager.cc:1120] Deleting tablet 711a38d7c47749f8a7f59e52b81b51ee
0502 08:16:57.601503 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.604564 (+ 3061us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4042951}
I20240502 08:16:57.605021 25012 tablet_replica.cc:330] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:57.605239 25012 raft_consensus.cc:2229] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:57.605607 25012 raft_consensus.cc:2258] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.606858 24173 tablet_bootstrap.cc:492] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:57.607414 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9761df0bbdee44af90135bffb8b5828f" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:57.607820 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0501575a84ec454780f48574bf3155fe" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:57.608793 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b0d4871dd24607854100a4ef762e25" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:57.608973 23788 raft_consensus.cc:2454] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
W20240502 08:16:57.609282 23723 leader_election.cc:341] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.609556 24181 ts_tablet_manager.cc:1888] T 128c88af54d24ffd895e9de19cdb4c8a P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:57.609651 25630 raft_consensus.cc:482] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.609982 25423 raft_consensus.cc:482] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.610122 25423 raft_consensus.cc:504] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.610751 25423 leader_election.cc:288] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:57.610899 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af6efeebaebc495aba017a922e243c39" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:57.611228 24181 log.cc:1196] T 128c88af54d24ffd895e9de19cdb4c8a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/128c88af54d24ffd895e9de19cdb4c8a
I20240502 08:16:57.611271 23996 leader_election.cc:302] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:57.611601 23996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 72402.2us
I20240502 08:16:57.611697 24181 ts_tablet_manager.cc:1909] T 128c88af54d24ffd895e9de19cdb4c8a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:57.612433 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 996) took 1591 ms. Trace:
I20240502 08:16:57.612493 24037 tablet_service.cc:1467] Processing CreateTablet for tablet ba4f43b8caeb4d83a2ae45df0448597c (DEFAULT_TABLE table=default.Table_b8657ee1f37f41e6a9ac770b3c4fdf73 [id=dc1d2bc00e3c43449412ea5556c58845]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.612558 24181 rpcz_store.cc:276] 0502 08:16:56.020676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.021085 (+ 409us) service_pool.cc:225] Handling call
0502 08:16:57.443996 (+1422911us) ts_tablet_manager.cc:1120] Deleting tablet 128c88af54d24ffd895e9de19cdb4c8a
0502 08:16:57.444006 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.457615 (+ 13609us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.460744 (+ 3129us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.612397 (+151653us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1422477}
I20240502 08:16:57.612866 24022 tablet_service.cc:1467] Processing CreateTablet for tablet f239cbee895542ed9fb5431ef13af0fa (DEFAULT_TABLE table=default.Table_b04d67567cc04ca49aeb83ae20849d4c [id=2716856a55f5460c971bc4ef1986da5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.612905 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 5abdb61c01124d1d8c82fff183640ffb (DEFAULT_TABLE table=default.Table_ae2ecda79c0245b59ef76809b5e3cdcc [id=f7401987176b428eb3355e3e29b5bebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.612978 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 994) took 1607 ms. Trace:
I20240502 08:16:57.613065 24181 rpcz_store.cc:276] 0502 08:16:56.005967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.023218 (+ 17251us) service_pool.cc:225] Handling call
0502 08:16:57.612834 (+1589616us) ts_tablet_manager.cc:1120] Deleting tablet 506644a1ead54a1faeaa8583ef96b449
0502 08:16:57.612847 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.612938 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1589300}
I20240502 08:16:57.613400 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1002) took 1561 ms. Trace:
I20240502 08:16:57.613494 24181 rpcz_store.cc:276] 0502 08:16:56.052087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.052396 (+ 309us) service_pool.cc:225] Handling call
0502 08:16:57.613281 (+1560885us) ts_tablet_manager.cc:1120] Deleting tablet 16a1c5276f014410b740c87b9971fb48
0502 08:16:57.613289 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.613365 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1560681}
I20240502 08:16:57.613790 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1020) took 1335 ms. Trace:
I20240502 08:16:57.613880 24181 rpcz_store.cc:276] 0502 08:16:56.278320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.279404 (+ 1084us) service_pool.cc:225] Handling call
0502 08:16:57.613680 (+1334276us) ts_tablet_manager.cc:1120] Deleting tablet 7fe583cbf4af493e82d3ec0508ea0f0b
0502 08:16:57.613689 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.613759 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1334124}
I20240502 08:16:57.614078 25423 raft_consensus.cc:2781] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:57.614148 24181 tablet_replica.cc:330] T 592ab8523b8d479d9caead321d5888c3 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:57.614182 25423 raft_consensus.cc:482] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:57.614262 25423 raft_consensus.cc:3037] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:57.614349 24181 raft_consensus.cc:2229] T 592ab8523b8d479d9caead321d5888c3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:57.614507 24181 raft_consensus.cc:2258] T 592ab8523b8d479d9caead321d5888c3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.614931 24024 tablet_service.cc:1467] Processing CreateTablet for tablet a015eaae830549769bd1561684b04bca (DEFAULT_TABLE table=default.Table_2ec1dfc748994bfe8841412d61167608 [id=e0ea2320ed6347bca9de4d9615d3f053]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.615731 24033 tablet_service.cc:1467] Processing CreateTablet for tablet f244c446b2764ae89d5bc22d1e3ab456 (DEFAULT_TABLE table=default.Table_cc0830cedb8b4372bd149e16c0548861 [id=f8cdaacdac1b4752991090c9f88b81ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.615952 24181 ts_tablet_manager.cc:1875] T 592ab8523b8d479d9caead321d5888c3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.616078 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 01143d14aca64adb846c8141ae41e6ac (DEFAULT_TABLE table=default.Table_825b08eccbb0430395675758faf7b2f8 [id=f586150dea1a4efeae35a1307ad51ef1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.616394 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e42383235df4f219aadc2d2e42045ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.616645 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecc2a722985421da96565cb895875a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.616698 25423 raft_consensus.cc:504] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.616874 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a015eaae830549769bd1561684b04bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.617086 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01143d14aca64adb846c8141ae41e6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.617883 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f239cbee895542ed9fb5431ef13af0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.619843 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 8c93c89e1341474eafb6030b95f1c68c (DEFAULT_TABLE table=default.Table_206aa937a259485a90fa3dd13e7bd838 [id=044ed1c60f534ea98ce735067e2ac68d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.620505 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f244c446b2764ae89d5bc22d1e3ab456. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.620954 23751 tablet_service.cc:1467] Processing CreateTablet for tablet fd3f59a2654a459e8bfd3be29c203e07 (DEFAULT_TABLE table=default.Table_97534a2075bc4b7297b36916b7a5f77c [id=5823995f324846088c24d9ffd5d742c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.622017 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 3519b9c0ff2546c882ac9a92add72b40 (DEFAULT_TABLE table=default.Table_38d10d7d3ecf4f89accb1c4dbfebdbea [id=3834a986d4d844bf919e00778ec96047]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.622272 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5abdb61c01124d1d8c82fff183640ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.626343 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 4b52e5c136d84cf1bfc8fadea62491f6 (DEFAULT_TABLE table=default.Table_5f11f4a4e19542c3bb6267e4e858d5e9 [id=bb7769d56b8f49cda34068306697cd56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.627173 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 82082b214d834186a185e0c65e9c1b41 (DEFAULT_TABLE table=default.Table_bbfae571fa824e51bf838a690198e9bb [id=e5eac9b0fb414b2fa4f2c5dea6daffcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.627856 23762 tablet_service.cc:1467] Processing CreateTablet for tablet f811d5771d4c45cf9c746b69705db095 (DEFAULT_TABLE table=default.Table_07b885519a2f4dcab31df7a12ddd2154 [id=dcc0e239e83a4a79af3524b5c2f8c393]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.628947 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 661f7748cf74413cbed6f6bb0d46fef4 (DEFAULT_TABLE table=default.Table_77228df00cdf4938acabcf03c8b4525e [id=7ad96431ea8c4fb3b3f537434e10792f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.630156 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd7b1a32207469393ff12f90a8043cb (DEFAULT_TABLE table=default.Table_d6b312d9972e4d8fa7dfb6fcad7314b9 [id=d9ba8333306d4852a612b83db8ab690c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.612179 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f57e3a39d54b7ab372d4b145b7a696" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:57.630622 23917 raft_consensus.cc:2454] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
W20240502 08:16:57.619086 23861 leader_election.cc:341] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.621287 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0501575a84ec454780f48574bf3155fe" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:57.631316 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "090bec30f5664421b9bfe3f2810afd14" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:57.631484 23917 raft_consensus.cc:3037] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:57.631721 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1b2aa20459484e4b98e0ae372418020f consensus state has no leader
W20240502 08:16:57.632674 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5155ccf37d3e45bcb26740db910daec1 consensus state has no leader
I20240502 08:16:57.635175 24173 tablet_bootstrap.cc:654] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:57.635216 24029 tablet_service.cc:1467] Processing CreateTablet for tablet f03473f7db3e411186f0db827a6630f7 (DEFAULT_TABLE table=default.Table_26889a40ce5a43bcbda607dcda6887bf [id=83e0466edd37473d9a9dfa8f0f8eed38]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.635524 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3519b9c0ff2546c882ac9a92add72b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.636533 23917 raft_consensus.cc:2454] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:57.637634 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9761df0bbdee44af90135bffb8b5828f" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:57.637112 25423 leader_election.cc:288] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:57.638808 25630 raft_consensus.cc:504] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.639353 25630 leader_election.cc:288] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:57.639912 23993 leader_election.cc:341] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 89f57e3a39d54b7ab372d4b145b7a696
W20240502 08:16:57.640127 23993 leader_election.cc:341] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 090bec30f5664421b9bfe3f2810afd14
I20240502 08:16:57.640597 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f3f09fc1f94c00a47b9087b98e221d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:57.640764 23788 raft_consensus.cc:2454] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:57.642248 23996 leader_election.cc:302] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:57.642902 23996 leader_election.cc:302] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:57.643833 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f3f09fc1f94c00a47b9087b98e221d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:57.650326 25630 raft_consensus.cc:2781] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:57.650635 23993 leader_election.cc:302] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:57.650839 23895 tablet_service.cc:1467] Processing CreateTablet for tablet d423a92750c94ee1a08688d60dd2168e (DEFAULT_TABLE table=default.Table_ca502551d3ba4c76868d9a79c06a8569 [id=71f7e0ccacc94451bf4cdc2ee7b6588f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.650940 25630 raft_consensus.cc:482] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:57.651229 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d423a92750c94ee1a08688d60dd2168e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.651289 25630 raft_consensus.cc:3037] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:57.651999 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f03473f7db3e411186f0db827a6630f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.653945 25630 raft_consensus.cc:504] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.655052 25413 raft_consensus.cc:2781] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240502 08:16:57.655135 23996 leader_election.cc:341] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.655140 25413 raft_consensus.cc:482] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:57.655323 25413 raft_consensus.cc:3037] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:57.655450 25405 raft_consensus.cc:2781] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:57.655673 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet af6efeebaebc495aba017a922e243c39 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e13c0b81dc24b8ba1a88b31bf510efd) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.657918 25630 leader_election.cc:288] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:57.658398 25405 raft_consensus.cc:686] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:57.658658 25405 consensus_queue.cc:237] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:16:57.659935 23993 leader_election.cc:341] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 89f57e3a39d54b7ab372d4b145b7a696
I20240502 08:16:57.660212 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba4f43b8caeb4d83a2ae45df0448597c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.661712 25413 raft_consensus.cc:504] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.662350 25413 leader_election.cc:288] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:57.662667 23788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f3f09fc1f94c00a47b9087b98e221d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:57.662739 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f3f09fc1f94c00a47b9087b98e221d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:57.662871 23788 raft_consensus.cc:3037] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:57.663286 23996 leader_election.cc:341] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.664319 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_15060d38700048cf9cd741f97742efe9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.664892 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15060d38700048cf9cd741f97742efe9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.665259 23788 raft_consensus.cc:2454] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:57.666503 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f57e3a39d54b7ab372d4b145b7a696" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:57.666647 23920 raft_consensus.cc:3037] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:57.667160 23993 leader_election.cc:302] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:57.667274 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a1b534c89aa4d839c220a593a753c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.667714 25413 raft_consensus.cc:2781] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:57.667964 25413 raft_consensus.cc:686] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:57.668223 25413 consensus_queue.cc:237] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.669957 25012 ts_tablet_manager.cc:1875] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.670771 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet c84b27ed8cf7429ca720e4cb8d396342 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.671157 24181 ts_tablet_manager.cc:1888] T 592ab8523b8d479d9caead321d5888c3 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:57.671260 24181 log.cc:1196] T 592ab8523b8d479d9caead321d5888c3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/592ab8523b8d479d9caead321d5888c3
I20240502 08:16:57.671553 24181 ts_tablet_manager.cc:1909] T 592ab8523b8d479d9caead321d5888c3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:57.671558 23889 tablet_service.cc:1467] Processing CreateTablet for tablet ba4f43b8caeb4d83a2ae45df0448597c (DEFAULT_TABLE table=default.Table_b8657ee1f37f41e6a9ac770b3c4fdf73 [id=dc1d2bc00e3c43449412ea5556c58845]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.672156 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ea6743a3a44f9b95e1ca3d50a3dd40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.673604 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a9676602d1c4b37b3f4126a29f5456f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.674110 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet af6efeebaebc495aba017a922e243c39 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e13c0b81dc24b8ba1a88b31bf510efd) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.674888 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_872fcb31253f47e1976fbed00a641287" } modify_external_catalogs: true
I20240502 08:16:57.674979 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet b054783710824dbc8f7bc3ea38f605b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.675100 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_872fcb31253f47e1976fbed00a641287" } modify_external_catalogs: true
I20240502 08:16:57.675701 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet 090bec30f5664421b9bfe3f2810afd14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.676195 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet ba3aa2621fdb43fd8299f81af486ef1e with delete_type TABLET_DATA_DELETED (replaced by tablet e20e126b1c9546e4a194bf9842f444a5) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.676761 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.677295 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 852486950b4144daa5cbd0644dbe1a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.678084 23887 tablet_service.cc:1467] Processing CreateTablet for tablet aecc2a722985421da96565cb895875a2 (DEFAULT_TABLE table=default.Table_fdc1e83837dc40ca99542547bc5fa957 [id=9607b22df6a04fbc940db9bed2291a66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.678520 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba4f43b8caeb4d83a2ae45df0448597c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.679051 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecc2a722985421da96565cb895875a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.681356 25158 raft_consensus.cc:482] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.681490 25158 raft_consensus.cc:504] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:57.682130 25158 leader_election.cc:288] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:57.682245 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6dd2e8e9898c4e51945829fcf6afe67c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.682811 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6dd2e8e9898c4e51945829fcf6afe67c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.682905 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb96ad2289754e428a1c109b2d59c53b" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:57.683465 23723 leader_election.cc:341] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.684237 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_aea78b7f89b04a068101eec9cff71223"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.684790 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aea78b7f89b04a068101eec9cff71223 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.688164 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e04d50c60b4856a79217c00f63d0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.690716 24173 tablet_bootstrap.cc:492] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:57.690836 24173 ts_tablet_manager.cc:1367] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.177s user 0.008s sys 0.002s
I20240502 08:16:57.691561 24173 raft_consensus.cc:348] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.691747 24173 raft_consensus.cc:374] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:57.691805 24173 raft_consensus.cc:729] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:57.692029 24173 consensus_queue.cc:260] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.692485 24173 ts_tablet_manager.cc:1398] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:57.694438 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 66ea6743a3a44f9b95e1ca3d50a3dd40 (DEFAULT_TABLE table=default.Table_ac7f78ed7ac94104a09a418eed600e2d [id=5873e9784ac24716972b4fb9245e5331]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.694766 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ea6743a3a44f9b95e1ca3d50a3dd40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.700124 23891 tablet_service.cc:1467] Processing CreateTablet for tablet f239cbee895542ed9fb5431ef13af0fa (DEFAULT_TABLE table=default.Table_b04d67567cc04ca49aeb83ae20849d4c [id=2716856a55f5460c971bc4ef1986da5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.700428 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f239cbee895542ed9fb5431ef13af0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.701963 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e4a87654d97249418e5df3a99b23c557
I20240502 08:16:57.702270 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 63a595654cff4565bcf21a7c8fabc2c4
I20240502 08:16:57.705518 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet e4a87654d97249418e5df3a99b23c557 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.706004 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a595654cff4565bcf21a7c8fabc2c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.706954 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1021) took 1387 ms. Trace:
I20240502 08:16:57.707033 24181 rpcz_store.cc:276] 0502 08:16:56.319778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.329760 (+ 9982us) service_pool.cc:225] Handling call
0502 08:16:57.614092 (+1284332us) ts_tablet_manager.cc:1120] Deleting tablet 592ab8523b8d479d9caead321d5888c3
0502 08:16:57.614101 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.653394 (+ 39293us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.671070 (+ 17676us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.706934 (+ 35864us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34615,"tablet-delete.queue_time_us":1283950}
I20240502 08:16:57.707433 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1022) took 1387 ms. Trace:
I20240502 08:16:57.707499 24181 rpcz_store.cc:276] 0502 08:16:56.320270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.337704 (+ 17434us) service_pool.cc:225] Handling call
0502 08:16:57.707342 (+1369638us) ts_tablet_manager.cc:1120] Deleting tablet 24a4104f4cb64889ad66810788ad6b29
0502 08:16:57.707351 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.707419 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1369325}
I20240502 08:16:57.707767 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1023) took 1387 ms. Trace:
I20240502 08:16:57.707823 24181 rpcz_store.cc:276] 0502 08:16:56.320660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.338136 (+ 17476us) service_pool.cc:225] Handling call
0502 08:16:57.707704 (+1369568us) ts_tablet_manager.cc:1120] Deleting tablet 654ce33d1e3d472eb6ae6a726eb803bb
0502 08:16:57.707712 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.707756 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1369336}
I20240502 08:16:57.708109 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1024) took 1053 ms. Trace:
I20240502 08:16:57.708169 24181 rpcz_store.cc:276] 0502 08:16:56.654775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.655138 (+ 363us) service_pool.cc:225] Handling call
0502 08:16:57.708024 (+1052886us) ts_tablet_manager.cc:1120] Deleting tablet 548ab3f9d0d84527b1092ae715ec2bdd
0502 08:16:57.708032 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.708097 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1052631}
I20240502 08:16:57.709249 24181 tablet_replica.cc:330] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:57.709414 24181 raft_consensus.cc:2229] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:57.709964 24181 raft_consensus.cc:2258] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.710680 24181 ts_tablet_manager.cc:1875] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.712911 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 4bf7892302444634885702720dd413c6 (DEFAULT_TABLE table=default.Table_10dce9ce73a64cc9a0bf79972f69aed9 [id=8c282b8fab7a40dd8b5de7c8ef9cee6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.715831 24181 ts_tablet_manager.cc:1888] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:57.715947 24181 log.cc:1196] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/df08f0d3af4c4c7e8abce656714b1b7f
I20240502 08:16:57.716296 24181 ts_tablet_manager.cc:1909] T df08f0d3af4c4c7e8abce656714b1b7f P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:57.716698 24181 tablet_replica.cc:330] T 25c089f0c1b14e419f250d03d4eaad12 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:57.716873 24181 raft_consensus.cc:2229] T 25c089f0c1b14e419f250d03d4eaad12 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:57.717036 24181 raft_consensus.cc:2258] T 25c089f0c1b14e419f250d03d4eaad12 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.717475 24181 ts_tablet_manager.cc:1875] T 25c089f0c1b14e419f250d03d4eaad12 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.721165 24173 tablet_bootstrap.cc:492] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:57.721096 23889 tablet_service.cc:1467] Processing CreateTablet for tablet f03473f7db3e411186f0db827a6630f7 (DEFAULT_TABLE table=default.Table_26889a40ce5a43bcbda607dcda6887bf [id=83e0466edd37473d9a9dfa8f0f8eed38]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.721449 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f03473f7db3e411186f0db827a6630f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.721377 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_4717a4024e6a4e4aadf5e4dd07a27d7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.722098 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4717a4024e6a4e4aadf5e4dd07a27d7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.094328 25494 raft_consensus.cc:504] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.723698 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0d79aa9695cf4de7b5ff7249dcab2669" } new_table_name: "default.Table_c15d8047d6fd438dbdb3f1961b549f4c" modify_external_catalogs: true
I20240502 08:16:57.726945 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_037c4757772643e5bf3a793f1ed95e38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.727464 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_037c4757772643e5bf3a793f1ed95e38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.732442 25034 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" instance_seqno: 1714637793363499) as {username='slave'} at 127.22.225.67:41539; Asking this server to re-register.
I20240502 08:16:57.734972 24107 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:57.741464 23920 raft_consensus.cc:2454] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:57.742338 24034 tablet_service.cc:1514] Processing DeleteTablet for tablet c00f29fb76b44d6ea4f76963d2d0751c with delete_type TABLET_DATA_DELETED (replaced by tablet a17fb4f2f02e43dc9f7cfdb7c73f7774) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.742413 23610 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet a2da8b9d3f374d8a812336057c0ce7d2 (table default.Table_3edd27da6927478281e76361718425a3 [id=76b58ce209a24b9a95b405c1c50de3ca]). Expected version 3 got 0
I20240502 08:16:57.742656 23610 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 9761df0bbdee44af90135bffb8b5828f (table default.Table_8ba240c391da43978f93a27dce48e557 [id=1b5536950db244d1978b154da0be33e0]). Expected version 2 got 0
I20240502 08:16:57.743083 24034 tablet_service.cc:1467] Processing CreateTablet for tablet eaac3bd5aa674422b7feb85ed9db1f70 (DEFAULT_TABLE table=default.Table_b0b35fed485c4223a4710de9abfd86fb [id=b863ddadc61c42d5a3fc5a1621cfb7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.743291 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 4bf7892302444634885702720dd413c6 (DEFAULT_TABLE table=default.Table_10dce9ce73a64cc9a0bf79972f69aed9 [id=8c282b8fab7a40dd8b5de7c8ef9cee6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.746773 23608 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_a394560ca3454cb88c3cbc2e87ab4fa7 [id=09f6a10417fb4639a9b97f5bb9efee6b] alter complete (version 1)
I20240502 08:16:57.749857 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_67d67d37c02a4642ad3d42d348ffbbac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.750335 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67d67d37c02a4642ad3d42d348ffbbac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.752655 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fcef9af155ad4443b42d743e4046b1d8" } modify_external_catalogs: true
I20240502 08:16:57.752833 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fcef9af155ad4443b42d743e4046b1d8" } modify_external_catalogs: true
I20240502 08:16:57.754031 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_79bacd8e921e4c1bafaf368c3e64bfaa" } modify_external_catalogs: true
I20240502 08:16:57.754226 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_79bacd8e921e4c1bafaf368c3e64bfaa" } modify_external_catalogs: true
I20240502 08:16:57.755522 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf7892302444634885702720dd413c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.756605 24173 tablet_bootstrap.cc:654] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:57.759668 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0001b9d68e0a4b21a9bfd584dcdf99ea has no consensus state
I20240502 08:16:57.763711 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 661f7748cf74413cbed6f6bb0d46fef4 (DEFAULT_TABLE table=default.Table_77228df00cdf4938acabcf03c8b4525e [id=7ad96431ea8c4fb3b3f537434e10792f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.763870 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd7b1a32207469393ff12f90a8043cb (DEFAULT_TABLE table=default.Table_d6b312d9972e4d8fa7dfb6fcad7314b9 [id=d9ba8333306d4852a612b83db8ab690c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.764041 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661f7748cf74413cbed6f6bb0d46fef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.764065 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 128c88af54d24ffd895e9de19cdb4c8a (table default.Table_d67be3894f494223a45d8c5a2c3e6113 [id=f7982b1d892942648cf3fca657c2668c]) successfully deleted
I20240502 08:16:57.764271 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd7b1a32207469393ff12f90a8043cb. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:57.764317 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 506644a1ead54a1faeaa8583ef96b449 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 506644a1ead54a1faeaa8583ef96b449 already in progress: creating tablet
W20240502 08:16:57.764546 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 16a1c5276f014410b740c87b9971fb48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16a1c5276f014410b740c87b9971fb48 already in progress: creating tablet
W20240502 08:16:57.764757 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 7fe583cbf4af493e82d3ec0508ea0f0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fe583cbf4af493e82d3ec0508ea0f0b already in progress: creating tablet
I20240502 08:16:57.765574 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 592ab8523b8d479d9caead321d5888c3 (table default.Table_af7c06dd8b3f4085b62eaef9ff9b57cb [id=11e47f3f89ff4bbbbb32ab105efe686c]) successfully deleted
W20240502 08:16:57.765806 23597 catalog_manager.cc:4797] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 24a4104f4cb64889ad66810788ad6b29 because the tablet was not found. No further retry: Not found: Tablet not found: 24a4104f4cb64889ad66810788ad6b29
W20240502 08:16:57.766023 23597 catalog_manager.cc:4797] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 654ce33d1e3d472eb6ae6a726eb803bb because the tablet was not found. No further retry: Not found: Tablet not found: 654ce33d1e3d472eb6ae6a726eb803bb
W20240502 08:16:57.766311 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 548ab3f9d0d84527b1092ae715ec2bdd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 548ab3f9d0d84527b1092ae715ec2bdd already in progress: creating tablet
W20240502 08:16:57.766490 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 96013fa1fa014b1891eb621cc3a81cdc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96013fa1fa014b1891eb621cc3a81cdc already in progress: creating tablet
W20240502 08:16:57.766644 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet edd7244b9f5741a8aa1c6835fe544676 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edd7244b9f5741a8aa1c6835fe544676 already in progress: creating tablet
W20240502 08:16:57.766811 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 92b830e128ac439eba286f29dbb20ee4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92b830e128ac439eba286f29dbb20ee4 already in progress: creating tablet
W20240502 08:16:57.767025 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e5de31a3db014615989d9e018521bfc9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5de31a3db014615989d9e018521bfc9 already in progress: creating tablet
W20240502 08:16:57.767262 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 01143d14aca64adb846c8141ae41e6ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01143d14aca64adb846c8141ae41e6ac already in progress: creating tablet
W20240502 08:16:57.767452 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 521581fe87e547ce816b599578bddeea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 521581fe87e547ce816b599578bddeea already in progress: creating tablet
I20240502 08:16:57.767714 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet df08f0d3af4c4c7e8abce656714b1b7f (table default.Table_59f82dbde55b406ba5271520c5bab104 [id=8e9acf4c0dbf4590b82aaee07243ed0c]) successfully deleted
I20240502 08:16:57.768250 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 6e7432f1df94431a83cf68356e3e78a3 (table default.Table_a89a2cdb74684df5bcbe15389816c50b [id=04819651040e46c580d553290e3fa250]) successfully deleted
I20240502 08:16:57.768954 24023 tablet_service.cc:1467] Processing CreateTablet for tablet f811d5771d4c45cf9c746b69705db095 (DEFAULT_TABLE table=default.Table_07b885519a2f4dcab31df7a12ddd2154 [id=dcc0e239e83a4a79af3524b5c2f8c393]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.769126 24037 tablet_service.cc:1467] Processing CreateTablet for tablet fd3f59a2654a459e8bfd3be29c203e07 (DEFAULT_TABLE table=default.Table_97534a2075bc4b7297b36916b7a5f77c [id=5823995f324846088c24d9ffd5d742c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.769258 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f811d5771d4c45cf9c746b69705db095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.769523 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd3f59a2654a459e8bfd3be29c203e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.769544 25531 raft_consensus.cc:482] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.769701 25531 raft_consensus.cc:504] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:57.770884 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8910bc67c6234a538c8160e7c020653c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:57.771631 23723 leader_election.cc:341] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.773305 25531 leader_election.cc:288] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:57.774006 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25c089f0c1b14e419f250d03d4eaad12" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:57.768961 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 4b52e5c136d84cf1bfc8fadea62491f6 (DEFAULT_TABLE table=default.Table_5f11f4a4e19542c3bb6267e4e858d5e9 [id=bb7769d56b8f49cda34068306697cd56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.776789 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b52e5c136d84cf1bfc8fadea62491f6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:57.781785 23723 leader_election.cc:341] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.781910 23723 leader_election.cc:302] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:57.782727 25531 raft_consensus.cc:2726] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:57.744403 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 5abdb61c01124d1d8c82fff183640ffb (DEFAULT_TABLE table=default.Table_ae2ecda79c0245b59ef76809b5e3cdcc [id=f7401987176b428eb3355e3e29b5bebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.783649 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5abdb61c01124d1d8c82fff183640ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.783831 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 82082b214d834186a185e0c65e9c1b41 (DEFAULT_TABLE table=default.Table_bbfae571fa824e51bf838a690198e9bb [id=e5eac9b0fb414b2fa4f2c5dea6daffcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.784222 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82082b214d834186a185e0c65e9c1b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.786103 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaac3bd5aa674422b7feb85ed9db1f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.788194 24021 tablet_service.cc:1467] Processing CreateTablet for tablet 8c93c89e1341474eafb6030b95f1c68c (DEFAULT_TABLE table=default.Table_206aa937a259485a90fa3dd13e7bd838 [id=044ed1c60f534ea98ce735067e2ac68d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.788590 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c93c89e1341474eafb6030b95f1c68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.789355 23894 tablet_service.cc:1467] Processing CreateTablet for tablet eaac3bd5aa674422b7feb85ed9db1f70 (DEFAULT_TABLE table=default.Table_b0b35fed485c4223a4710de9abfd86fb [id=b863ddadc61c42d5a3fc5a1621cfb7f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.744827 23888 tablet_service.cc:1467] Processing CreateTablet for tablet a015eaae830549769bd1561684b04bca (DEFAULT_TABLE table=default.Table_2ec1dfc748994bfe8841412d61167608 [id=e0ea2320ed6347bca9de4d9615d3f053]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.745258 23900 tablet_service.cc:1467] Processing CreateTablet for tablet f244c446b2764ae89d5bc22d1e3ab456 (DEFAULT_TABLE table=default.Table_cc0830cedb8b4372bd149e16c0548861 [id=f8cdaacdac1b4752991090c9f88b81ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.789786 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_340594e848c94913801c031a70a94575" } new_table_name: "default.Table_8c9b3ba9930a45ff99cdc242ebcd8a9c" modify_external_catalogs: true
I20240502 08:16:57.745579 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 3519b9c0ff2546c882ac9a92add72b40 (DEFAULT_TABLE table=default.Table_38d10d7d3ecf4f89accb1c4dbfebdbea [id=3834a986d4d844bf919e00778ec96047]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.745769 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 01143d14aca64adb846c8141ae41e6ac (DEFAULT_TABLE table=default.Table_825b08eccbb0430395675758faf7b2f8 [id=f586150dea1a4efeae35a1307ad51ef1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.746052 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 852486950b4144daa5cbd0644dbe1a41 (DEFAULT_TABLE table=default.Table_9cb7a43194434b49ad817e7876631133 [id=d79937be8924416094c969bf6e8ba27d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.745524 23886 tablet_service.cc:1467] Processing CreateTablet for tablet 37e04d50c60b4856a79217c00f63d0c7 (DEFAULT_TABLE table=default.Table_1b391fb1270447ed9134fcc3d288a9e3 [id=4321b452ff8c4646bef6c86930ecd313]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.753782 24175 tablet_bootstrap.cc:492] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:57.791050 24175 ts_tablet_manager.cc:1367] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.437s user 0.002s sys 0.004s
I20240502 08:16:57.791803 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaac3bd5aa674422b7feb85ed9db1f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.792055 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a015eaae830549769bd1561684b04bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.793208 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 852486950b4144daa5cbd0644dbe1a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.797295 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 4bf7892302444634885702720dd413c6 (DEFAULT_TABLE table=default.Table_10dce9ce73a64cc9a0bf79972f69aed9 [id=8c282b8fab7a40dd8b5de7c8ef9cee6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.797567 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf7892302444634885702720dd413c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.798257 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e04d50c60b4856a79217c00f63d0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.801667 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01143d14aca64adb846c8141ae41e6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.801944 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f244c446b2764ae89d5bc22d1e3ab456. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:57.802138 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c71f7398eab4c099cb45a55e50eee51 consensus state has no leader
W20240502 08:16:57.803145 23594 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet d935b59497bc48a190061a6111e9ae1a (table default.Table_129ded6f4dac43178e3f951a50366565 [id=de36429b0ca34ce592e4405254ea65c4], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d935b59497bc48a190061a6111e9ae1a consensus state has no leader
I20240502 08:16:57.803179 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet eb96ad2289754e428a1c109b2d59c53b with delete_type TABLET_DATA_DELETED (replaced by tablet 6ee764398b364d029ae7f0f575a30d64) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:57.803510 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a06fc827d204d02ab81d5eeb44f7421 consensus state has no leader
I20240502 08:16:57.810281 24174 tablet_bootstrap.cc:492] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:57.810431 24174 ts_tablet_manager.cc:1367] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.261s user 0.006s sys 0.000s
I20240502 08:16:57.810978 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 8c93c89e1341474eafb6030b95f1c68c (DEFAULT_TABLE table=default.Table_206aa937a259485a90fa3dd13e7bd838 [id=044ed1c60f534ea98ce735067e2ac68d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.811148 24174 raft_consensus.cc:348] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:57.811342 24174 raft_consensus.cc:374] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:57.811322 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c93c89e1341474eafb6030b95f1c68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.811403 24174 raft_consensus.cc:729] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:57.811594 24174 consensus_queue.cc:260] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:57.812810 25423 consensus_queue.cc:1035] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:57.813452 25649 raft_consensus.cc:482] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.813635 25649 raft_consensus.cc:504] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:57.814287 25649 leader_election.cc:288] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:57.815796 23888 tablet_service.cc:1467] Processing CreateTablet for tablet fd3f59a2654a459e8bfd3be29c203e07 (DEFAULT_TABLE table=default.Table_97534a2075bc4b7297b36916b7a5f77c [id=5823995f324846088c24d9ffd5d742c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.816170 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd3f59a2654a459e8bfd3be29c203e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.819101 23886 tablet_service.cc:1467] Processing CreateTablet for tablet f811d5771d4c45cf9c746b69705db095 (DEFAULT_TABLE table=default.Table_07b885519a2f4dcab31df7a12ddd2154 [id=dcc0e239e83a4a79af3524b5c2f8c393]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.819403 23886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f811d5771d4c45cf9c746b69705db095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.822710 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 82082b214d834186a185e0c65e9c1b41 (DEFAULT_TABLE table=default.Table_bbfae571fa824e51bf838a690198e9bb [id=e5eac9b0fb414b2fa4f2c5dea6daffcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.822997 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82082b214d834186a185e0c65e9c1b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.823325 23620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1055 } } ignore_live_leader: false dest_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" is_pre_election: true
I20240502 08:16:57.826189 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 4b52e5c136d84cf1bfc8fadea62491f6 (DEFAULT_TABLE table=default.Table_5f11f4a4e19542c3bb6267e4e858d5e9 [id=bb7769d56b8f49cda34068306697cd56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.826162 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_09bcb43a340f47248dc7fa91d2f73036"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:57.826575 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b52e5c136d84cf1bfc8fadea62491f6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:57.826722 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09bcb43a340f47248dc7fa91d2f73036 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.827009 24922 leader_election.cc:302] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84f3c07a31414c4f8fb61c44dc9f16e1; no voters: bcc8a323e6b9429eaaf4e0c1698ad077, f87df6a45ad94428a9aa6b2796a0fd25
I20240502 08:16:57.827491 25637 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:57.831298 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 661f7748cf74413cbed6f6bb0d46fef4 (DEFAULT_TABLE table=default.Table_77228df00cdf4938acabcf03c8b4525e [id=7ad96431ea8c4fb3b3f537434e10792f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.833246 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_04ee6fb74c7c4370a57d57b3b9aa3226"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:57.833341 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_effefe18d3cd42f7855fed709d87de8c" } new_table_name: "default.Table_6d585050acb748738a5019f610e1570e" modify_external_catalogs: true
W20240502 08:16:57.833776 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04ee6fb74c7c4370a57d57b3b9aa3226 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.834568 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d45052f775a8497fad354f27c3def079" } new_table_name: "default.Table_5b25dd22d0f24b7b9405582e1bc54ecb" modify_external_catalogs: true
I20240502 08:16:57.843334 25034 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" instance_seqno: 1714637792902555) as {username='slave'} at 127.22.225.66:60495; Asking this server to re-register.
I20240502 08:16:57.846093 24173 tablet_bootstrap.cc:492] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:57.846256 24173 ts_tablet_manager.cc:1367] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.125s user 0.008s sys 0.005s
I20240502 08:16:57.847838 24181 ts_tablet_manager.cc:1888] T 25c089f0c1b14e419f250d03d4eaad12 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:57.847968 24181 log.cc:1196] T 25c089f0c1b14e419f250d03d4eaad12 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/25c089f0c1b14e419f250d03d4eaad12
I20240502 08:16:57.848259 24181 ts_tablet_manager.cc:1909] T 25c089f0c1b14e419f250d03d4eaad12 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:57.849540 24181 tablet_replica.cc:330] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:57.849732 24181 raft_consensus.cc:2229] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:57.849877 24181 raft_consensus.cc:2258] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:57.774433 23861 leader_election.cc:341] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: Tablet not RUNNING: STOPPED
W20240502 08:16:57.857693 23861 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 83267.5us
W20240502 08:16:57.857730 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet df87b8b1ff504031bf007880792192dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df87b8b1ff504031bf007880792192dd already in progress: creating tablet
W20240502 08:16:57.857995 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 40fab016ed4f489aa3ccdeafcfbe1477 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40fab016ed4f489aa3ccdeafcfbe1477 already in progress: creating tablet
W20240502 08:16:57.858219 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 711a38d7c47749f8a7f59e52b81b51ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 711a38d7c47749f8a7f59e52b81b51ee already in progress: creating tablet
W20240502 08:16:57.858402 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 598cba0f8b9f4c8dae9e34872c4212fa consensus state has no leader
I20240502 08:16:57.858422 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47459b5f5bb4ecb96f76240fefa1f5d" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:57.858896 23996 leader_election.cc:302] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:57.859496 23861 leader_election.cc:341] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:57.860165 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce8dc203c36149cc91cc56176e929da6 consensus state has no leader
I20240502 08:16:57.777894 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 68d8126bbc1d4710904baeddaaf493f2 (DEFAULT_TABLE table=default.Table_96b57ae1050b4b6e819e085ac1d3c79c [id=fd74c0f02d3743178fcd5bc92edd5758]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.778250 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 0a1b534c89aa4d839c220a593a753c23 (DEFAULT_TABLE table=default.Table_2d4cb7ffff364fe6becc190a0dadfb25 [id=f3844c1a455e435a90e9e7e1f625739a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.778728 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet c84b27ed8cf7429ca720e4cb8d396342 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.779024 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a9676602d1c4b37b3f4126a29f5456f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.779114 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet e47459b5f5bb4ecb96f76240fefa1f5d with delete_type TABLET_DATA_DELETED (replaced by tablet 4917c1f0decf43c2aca74c67d8b45341) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.779353 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 090bec30f5664421b9bfe3f2810afd14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.779609 23897 tablet_service.cc:1514] Processing DeleteTablet for tablet ba3aa2621fdb43fd8299f81af486ef1e with delete_type TABLET_DATA_DELETED (replaced by tablet e20e126b1c9546e4a194bf9842f444a5) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.779902 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet fbef606ce7ac499d9bd84b49d804621c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:40 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.860793 25423 raft_consensus.cc:2781] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:57.780256 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet df08f0d3af4c4c7e8abce656714b1b7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.780524 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet 592ab8523b8d479d9caead321d5888c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.780601 23904 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f57e3a39d54b7ab372d4b145b7a696 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.860999 25423 raft_consensus.cc:686] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:57.783272 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 6e42383235df4f219aadc2d2e42045ac (DEFAULT_TABLE table=default.Table_cf7ced0046f647eb82829af1fb7c985b [id=187304972ddc45d49cd15953974c6f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.835163 24175 raft_consensus.cc:348] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.861227 24175 raft_consensus.cc:374] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:57.861297 24175 raft_consensus.cc:729] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:57.861492 24175 consensus_queue.cc:260] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.838666 25012 ts_tablet_manager.cc:1888] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:57.864392 24175 ts_tablet_manager.cc:1398] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.073s user 0.002s sys 0.000s
I20240502 08:16:57.864683 24175 tablet_bootstrap.cc:492] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:57.864724 25012 log.cc:1196] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/dd3078cc1f6247609f8bd017ed076b45
I20240502 08:16:57.865409 25012 ts_tablet_manager.cc:1909] T dd3078cc1f6247609f8bd017ed076b45 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:57.870846 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 922) took 4312 ms. Trace:
I20240502 08:16:57.871031 25012 rpcz_store.cc:276] 0502 08:16:53.558672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.558711 (+ 39us) service_pool.cc:225] Handling call
0502 08:16:57.604982 (+4046271us) ts_tablet_manager.cc:1120] Deleting tablet dd3078cc1f6247609f8bd017ed076b45
0502 08:16:57.604990 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.692733 (+ 87743us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.838601 (+145868us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:57.870826 (+ 32225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4046149}
I20240502 08:16:57.871665 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 936) took 4115 ms. Trace:
I20240502 08:16:57.871817 25012 rpcz_store.cc:276] 0502 08:16:53.755967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.848244 (+ 92277us) service_pool.cc:225] Handling call
0502 08:16:57.871481 (+4023237us) ts_tablet_manager.cc:1120] Deleting tablet 88acb3b0c2b7480e917ed68b7646b47d
0502 08:16:57.871490 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.871649 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4023005}
W20240502 08:16:57.846172 23858 leader_election.cc:341] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: af6efeebaebc495aba017a922e243c39
I20240502 08:16:57.834573 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd7b1a32207469393ff12f90a8043cb (DEFAULT_TABLE table=default.Table_d6b312d9972e4d8fa7dfb6fcad7314b9 [id=d9ba8333306d4852a612b83db8ab690c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.835637 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3519b9c0ff2546c882ac9a92add72b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.872467 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a1b534c89aa4d839c220a593a753c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.755226 25494 leader_election.cc:288] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:57.872643 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68d8126bbc1d4710904baeddaaf493f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.872345 23858 leader_election.cc:302] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:57.873032 24174 ts_tablet_manager.cc:1398] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.063s user 0.002s sys 0.000s
I20240502 08:16:57.878540 25494 raft_consensus.cc:2726] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:57.878648 23858 leader_election.cc:302] T 9761df0bbdee44af90135bffb8b5828f P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:57.878288 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e42383235df4f219aadc2d2e42045ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.861464 25423 consensus_queue.cc:237] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.879040 24175 tablet_bootstrap.cc:654] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:57.881803 25494 raft_consensus.cc:2726] T 9761df0bbdee44af90135bffb8b5828f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:57.881982 23858 leader_election.cc:302] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
W20240502 08:16:57.882061 23861 consensus_peers.cc:485] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1da01e2c2f054ccfb53383e70e16da84. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:57.882266 25494 raft_consensus.cc:2781] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240502 08:16:57.882308 23858 leader_election.cc:341] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 25c089f0c1b14e419f250d03d4eaad12
I20240502 08:16:57.882333 25494 raft_consensus.cc:482] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:57.882400 25494 raft_consensus.cc:3037] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:57.882382 23858 leader_election.cc:302] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:57.882575 25649 raft_consensus.cc:2726] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:16:57.884735 24173 raft_consensus.cc:348] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.884907 24173 raft_consensus.cc:374] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:57.884951 24173 raft_consensus.cc:729] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:57.884816 25494 raft_consensus.cc:504] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.885109 24173 consensus_queue.cc:260] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.885299 23972 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:57.886022 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b0d4871dd24607854100a4ef762e25" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:57.886220 23786 raft_consensus.cc:3037] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:57.886415 23858 leader_election.cc:341] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: e47459b5f5bb4ecb96f76240fefa1f5d
I20240502 08:16:57.886569 23858 leader_election.cc:302] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:57.886795 25649 raft_consensus.cc:2726] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:57.888298 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c93c89e1341474eafb6030b95f1c68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.888782 23786 raft_consensus.cc:2454] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:57.889061 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b0d4871dd24607854100a4ef762e25" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:57.889283 25494 leader_election.cc:288] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:57.890304 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661f7748cf74413cbed6f6bb0d46fef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.893788 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd3f59a2654a459e8bfd3be29c203e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.893740 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb96ad2289754e428a1c109b2d59c53b" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:57.894017 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8910bc67c6234a538c8160e7c020653c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:57.894371 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b52e5c136d84cf1bfc8fadea62491f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.894594 25664 raft_consensus.cc:482] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.894704 25664 raft_consensus.cc:504] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.897719 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82082b214d834186a185e0c65e9c1b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.900808 24173 ts_tablet_manager.cc:1398] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.054s user 0.000s sys 0.002s
I20240502 08:16:57.901098 24173 tablet_bootstrap.cc:492] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:57.901749 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f811d5771d4c45cf9c746b69705db095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.904284 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 3af208d6231a4cf18823e9fe367ca46d
I20240502 08:16:57.904767 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 1b2aa20459484e4b98e0ae372418020f
I20240502 08:16:57.905709 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661f7748cf74413cbed6f6bb0d46fef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.906566 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a32841ac077a46f99eaae2d807cc25a6" } modify_external_catalogs: true
I20240502 08:16:57.906639 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_201d8955331641b4afc23044b3706aa7" } modify_external_catalogs: true
I20240502 08:16:57.906775 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_201d8955331641b4afc23044b3706aa7" } modify_external_catalogs: true
I20240502 08:16:57.906772 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a32841ac077a46f99eaae2d807cc25a6" } modify_external_catalogs: true
I20240502 08:16:57.908138 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c1c64a390b0440cb9ff7c4d5ae9f9c5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.908670 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1c64a390b0440cb9ff7c4d5ae9f9c5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.909946 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd7b1a32207469393ff12f90a8043cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.912855 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8ffe5dd9e3154677a31e2eac5921f22d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.913331 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ffe5dd9e3154677a31e2eac5921f22d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.914840 23605 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 96b8f21c64e343d88d84e9acd0bc69b4 (table default.Table_9d372763eb03435f8b7074a802d86ffe [id=9ff8ea75a9bc4de194a70fe2088a94bc]). Expected version 1 got 0
I20240502 08:16:57.915094 23605 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet ec10ff7251a5442a8ecc1881b34db74d (table default.Table_236150d3903d437d95f67b0edec36091 [id=1d7a43116fe74c35b505eccd165f99cd]). Expected version 2 got 0
I20240502 08:16:57.915346 23605 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 9761df0bbdee44af90135bffb8b5828f (table default.Table_8ba240c391da43978f93a27dce48e557 [id=1b5536950db244d1978b154da0be33e0]). Expected version 2 got 0
I20240502 08:16:57.915398 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet d2886233097a4d19b45a4f139d82e68a
W20240502 08:16:57.916532 23605 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 96b8f21c64e343d88d84e9acd0bc69b4 (table default.Table_9d372763eb03435f8b7074a802d86ffe [id=9ff8ea75a9bc4de194a70fe2088a94bc], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 96b8f21c64e343d88d84e9acd0bc69b4 consensus state has no leader
I20240502 08:16:57.919070 25531 raft_consensus.cc:482] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:57.919209 25531 raft_consensus.cc:504] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.919813 25531 leader_election.cc:288] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:57.920444 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92adebc0c69e4a52b975253656132495" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:57.920929 23723 leader_election.cc:341] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.922616 24180 tablet_replica.cc:330] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:57.922812 24180 raft_consensus.cc:2229] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:57.922955 24180 raft_consensus.cc:2258] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:57.925730 24180 ts_tablet_manager.cc:1875] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.926378 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 876d7dc6291f47e4b0e671b50f2fa8c7
I20240502 08:16:57.926007 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf7892302444634885702720dd413c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.929170 24175 tablet_bootstrap.cc:492] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:57.929299 24175 ts_tablet_manager.cc:1367] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.006s
I20240502 08:16:57.929987 24175 raft_consensus.cc:348] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.930200 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet c4f9e61686dc417bb5a88f96b4231f2e
I20240502 08:16:57.930223 24175 raft_consensus.cc:374] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:57.930332 24175 raft_consensus.cc:729] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:57.930555 24175 consensus_queue.cc:260] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.932668 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd7b1a32207469393ff12f90a8043cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.933142 24175 ts_tablet_manager.cc:1398] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:16:57.933449 24175 tablet_bootstrap.cc:492] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:57.937981 24175 tablet_bootstrap.cc:654] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:57.941962 24174 tablet_bootstrap.cc:492] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:57.942517 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet 3af208d6231a4cf18823e9fe367ca46d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.941668 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 937) took 4185 ms. Trace:
I20240502 08:16:57.942724 25012 rpcz_store.cc:276] 0502 08:16:53.756312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.848498 (+ 92186us) service_pool.cc:225] Handling call
0502 08:16:57.872177 (+4023679us) ts_tablet_manager.cc:1120] Deleting tablet 765fdb66f12941f0ac16223e931e576e
0502 08:16:57.872184 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.941651 (+ 69467us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4023453}
I20240502 08:16:57.943024 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet d2886233097a4d19b45a4f139d82e68a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.943187 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 938) took 4186 ms. Trace:
I20240502 08:16:57.943254 23605 master_service.cc:946] ReplaceTablet: received request to replace tablet 4bc133adb9a44cc7b56a3746d1eb76af from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:57.943658 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 876d7dc6291f47e4b0e671b50f2fa8c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.943802 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2aa20459484e4b98e0ae372418020f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.944311 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f9e61686dc417bb5a88f96b4231f2e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.945102 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 990d0cc863ee4f49a2fc2fffb217c4d7 (DEFAULT_TABLE table=default.Table_1c914a8e055b422e8029a6a02b4a2925 [id=24f7d6e752a64e75b372d51f8e6f4ee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.945452 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef622696fe7435abe07808f5e8e918d (DEFAULT_TABLE table=default.Table_873744f861a94cc8af0fb5c3c80c944d [id=e929dd1d0abc40e8a34895656bd6f3b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.946138 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 1ded3cee64f9409583040ef4b6e1f803 (DEFAULT_TABLE table=default.Table_f3099e422df24ee593db8bca89aba612 [id=2b8a7a487bb441b596a3578786f77172]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.946328 25012 rpcz_store.cc:276] 0502 08:16:53.756547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.848822 (+ 92275us) service_pool.cc:225] Handling call
0502 08:16:57.943033 (+4094211us) ts_tablet_manager.cc:1120] Deleting tablet 552e46f69230474189278911cfeec6c7
0502 08:16:57.943043 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.943172 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4093987}
I20240502 08:16:57.946772 23762 tablet_service.cc:1467] Processing CreateTablet for tablet fe1bafedb8f04f7c95683fbb15008831 (DEFAULT_TABLE table=default.Table_61b10a2ece3e45678e32119eea2f3c0d [id=e4237f1908f646049c3b8983b3a828f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.947098 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 939) took 4190 ms. Trace:
I20240502 08:16:57.947441 23760 tablet_service.cc:1467] Processing CreateTablet for tablet c86680b5d8d24c55b9c57916c6d97e2d (DEFAULT_TABLE table=default.Table_3209c07b69e64b7a9f32f74fa58aef11 [id=dfb9dce29b174be7a541a1e6cb814efc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.948437 23751 tablet_service.cc:1467] Processing CreateTablet for tablet ff3afe539daa4c77902e2fc136d0ccab (DEFAULT_TABLE table=default.Table_e8d89916bfe94d52af4f205db18f9e47 [id=b49962d67a7f45fe95c093e6809a9c6c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:57.948877 23723 leader_election.cc:341] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.948918 25012 rpcz_store.cc:276] 0502 08:16:53.756870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:53.849071 (+ 92201us) service_pool.cc:225] Handling call
0502 08:16:57.947003 (+4097932us) ts_tablet_manager.cc:1120] Deleting tablet df266081ed8345dda989b3050f605685
0502 08:16:57.947010 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:57.947082 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4097746}
I20240502 08:16:57.949023 23723 leader_election.cc:302] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:57.949141 25012 tablet_replica.cc:330] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:57.949306 25012 raft_consensus.cc:2229] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:57.949297 25531 raft_consensus.cc:2726] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:57.949460 25012 raft_consensus.cc:2258] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:57.949759 23723 leader_election.cc:341] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.949849 23723 leader_election.cc:302] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:57.950037 25531 raft_consensus.cc:2726] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:57.950912 25012 ts_tablet_manager.cc:1875] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.952427 25034 ts_manager.cc:194] Registered new tserver with Master: 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:16:57.953909 23597 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 92207.9us
I20240502 08:16:57.954691 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6d600d6bd3814d73ad7a816b2ef4ee3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.955160 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d600d6bd3814d73ad7a816b2ef4ee3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.955680 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet dd3078cc1f6247609f8bd017ed076b45 (table default.Table_bac7f229ce9541b39c0a70951431b689 [id=f9f2bf985f124269982869050bbecd70]) successfully deleted
W20240502 08:16:57.956007 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 88acb3b0c2b7480e917ed68b7646b47d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88acb3b0c2b7480e917ed68b7646b47d already in progress: creating tablet
I20240502 08:16:57.956583 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet e4a87654d97249418e5df3a99b23c557 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:57.958154 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92adebc0c69e4a52b975253656132495" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:57.958194 25034 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" instance_seqno: 1714637792402425) as {username='slave'} at 127.22.225.65:38405; Asking this server to re-register.
I20240502 08:16:57.958573 23858 leader_election.cc:302] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:57.958840 25494 raft_consensus.cc:2781] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:57.958952 25494 raft_consensus.cc:686] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:16:57.959149 25494 consensus_queue.cc:237] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.959136 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_4e4cde0945344d63bd522607b883ed2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:57.959308 24174 tablet_bootstrap.cc:654] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:57.959393 23723 leader_election.cc:341] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:57.959481 23723 leader_election.cc:302] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
W20240502 08:16:57.959630 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e4cde0945344d63bd522607b883ed2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.959812 25531 raft_consensus.cc:2726] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:57.960480 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet d2886233097a4d19b45a4f139d82e68a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.960920 23895 tablet_service.cc:1514] Processing DeleteTablet for tablet e4a87654d97249418e5df3a99b23c557 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.961158 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a595654cff4565bcf21a7c8fabc2c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:57.962837 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e6458d60fe4646ce9109c151feeefacb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.963325 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6458d60fe4646ce9109c151feeefacb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.964534 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_18ad0085c7a0415c854a64868bf0735c" } new_table_name: "default.Table_a8abf61978714eef874cf73bb6eadbdc" modify_external_catalogs: true
I20240502 08:16:57.965238 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf7386fdb7649258b2e795aa46b501e (DEFAULT_TABLE table=default.Table_8eb9563cd4c74c64a4ba798132ddf385 [id=1e1704b3553c427daa67ad579814774a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.965775 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf7386fdb7649258b2e795aa46b501e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.966773 24181 ts_tablet_manager.cc:1875] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:57.967128 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ded3cee64f9409583040ef4b6e1f803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.967736 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe1bafedb8f04f7c95683fbb15008831. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.969348 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef622696fe7435abe07808f5e8e918d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.969779 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 990d0cc863ee4f49a2fc2fffb217c4d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.970100 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 1424b28b227f42c790d2e85e44f72ce3 (DEFAULT_TABLE table=default.Table_6e99bd5e0b2a4692a1400b64cb638f26 [id=1ec0052611154410a87adeae83c994a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.970218 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86680b5d8d24c55b9c57916c6d97e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.970723 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 4bc133adb9a44cc7b56a3746d1eb76af
I20240502 08:16:57.970789 23753 tablet_service.cc:1467] Processing CreateTablet for tablet fd32de329009465c89a4917e2716d3e9 (DEFAULT_TABLE table=default.Table_465db433d6a5443bac1914682bc40769 [id=54ecfc07bae64409aae7ba8effde4685]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:57.971122 23605 catalog_manager.cc:6554] ReplaceTablet: tablet 4bc133adb9a44cc7b56a3746d1eb76af deleted and replaced by tablet f2016fc145314ad788b8c8c2c53bf810
I20240502 08:16:57.971510 23754 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bc133adb9a44cc7b56a3746d1eb76af with delete_type TABLET_DATA_DELETED (replaced by tablet f2016fc145314ad788b8c8c2c53bf810) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:57.972505 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_91e1ada585af482cbe82348b4f4c2087"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:57.972806 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3afe539daa4c77902e2fc136d0ccab. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:57.972945 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91e1ada585af482cbe82348b4f4c2087 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.972990 23610 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 96b8f21c64e343d88d84e9acd0bc69b4 (table default.Table_9d372763eb03435f8b7074a802d86ffe [id=9ff8ea75a9bc4de194a70fe2088a94bc]). Expected version 1 got 0
I20240502 08:16:57.974994 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fa1b2e7c94124643a4e7e8a0d0a4939d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.975448 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa1b2e7c94124643a4e7e8a0d0a4939d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.976759 23610 catalog_manager.cc:5484] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66). New cstate: current_term: 1 leader_uuid: "5b89a1f0485a44b7ac1710e2369a809b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:57.977267 23610 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet e0b0d4871dd24607854100a4ef762e25 (table default.Table_5301ffccbac743618a78289c2c1415e1 [id=910c7cae36264e39bbfca0be919538e0]). Expected version 1 got 0
I20240502 08:16:57.984022 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d871e4eda37844d49e6042631c03b1c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:57.984506 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d871e4eda37844d49e6042631c03b1c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:57.989364 24180 ts_tablet_manager.cc:1888] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:57.989606 24175 tablet_bootstrap.cc:492] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:57.989884 24175 ts_tablet_manager.cc:1367] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.001s
I20240502 08:16:57.990681 24175 raft_consensus.cc:348] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:57.990937 24175 raft_consensus.cc:374] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:57.991041 24175 raft_consensus.cc:729] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:57.991273 24175 consensus_queue.cc:260] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:16:57.941839 23996 consensus_peers.cc:485] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:16:57.993235 23996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 51435us
I20240502 08:16:57.994033 24175 ts_tablet_manager.cc:1398] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20240502 08:16:57.994181 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f4a3c195bed430e896cf7cd4c6f55e6 consensus state has no leader
I20240502 08:16:57.994297 24175 tablet_bootstrap.cc:492] T 592eeecbf5514215b6464bd030081e4b P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:57.995712 24175 tablet_bootstrap.cc:654] T 592eeecbf5514215b6464bd030081e4b P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:58.004556 23861 leader_election.cc:341] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:58.005431 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 876d7dc6291f47e4b0e671b50f2fa8c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.005439 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 3af208d6231a4cf18823e9fe367ca46d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.006942 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet d2886233097a4d19b45a4f139d82e68a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.007395 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef622696fe7435abe07808f5e8e918d (DEFAULT_TABLE table=default.Table_873744f861a94cc8af0fb5c3c80c944d [id=e929dd1d0abc40e8a34895656bd6f3b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.007787 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef622696fe7435abe07808f5e8e918d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.008186 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_469c024c43d64e2fb98eff5ff16e988b" } new_table_name: "default.Table_97f00fbbeae84ee782f7028ca32b8556" modify_external_catalogs: true
I20240502 08:16:58.011117 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 990d0cc863ee4f49a2fc2fffb217c4d7 (DEFAULT_TABLE table=default.Table_1c914a8e055b422e8029a6a02b4a2925 [id=24f7d6e752a64e75b372d51f8e6f4ee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.012035 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b4c563d6bd4a44bb3bab58a3fcdc07" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:58.012534 23996 leader_election.cc:341] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:58.012741 25664 leader_election.cc:288] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:58.014261 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_64b18a844e8d44ba8bca05606e5b34e8" } new_table_name: "default.Table_be6bc028c42f4db7aff7192660c4f41c" modify_external_catalogs: true
I20240502 08:16:58.014397 24173 tablet_bootstrap.cc:654] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.015245 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 990d0cc863ee4f49a2fc2fffb217c4d7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:58.019335 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 49acc2af1a064dd2b0b60120beed69ba consensus state has no leader
I20240502 08:16:58.022955 23895 tablet_service.cc:1514] Processing DeleteTablet for tablet 876d7dc6291f47e4b0e671b50f2fa8c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:58.023479 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 26c8325099af4d72ad00afb2316df885 consensus state has no leader
I20240502 08:16:58.023550 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef622696fe7435abe07808f5e8e918d (DEFAULT_TABLE table=default.Table_873744f861a94cc8af0fb5c3c80c944d [id=e929dd1d0abc40e8a34895656bd6f3b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.023979 23895 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f9e61686dc417bb5a88f96b4231f2e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:58.025391 24173 tablet_bootstrap.cc:492] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:58.025558 24173 ts_tablet_manager.cc:1367] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.125s user 0.001s sys 0.003s
I20240502 08:16:58.026280 24173 raft_consensus.cc:348] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.026587 24173 raft_consensus.cc:374] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.026690 24173 raft_consensus.cc:729] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.027009 24173 consensus_queue.cc:260] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.027622 24173 ts_tablet_manager.cc:1398] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:58.027942 24173 tablet_bootstrap.cc:492] T ee022e8e86f8477a8fa1e58d52644b03 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:58.028092 24175 tablet_bootstrap.cc:492] T 592eeecbf5514215b6464bd030081e4b P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:58.028290 24175 ts_tablet_manager.cc:1367] T 592eeecbf5514215b6464bd030081e4b P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20240502 08:16:58.029081 24175 raft_consensus.cc:348] T 592eeecbf5514215b6464bd030081e4b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.029321 24175 raft_consensus.cc:374] T 592eeecbf5514215b6464bd030081e4b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.029459 24175 raft_consensus.cc:729] T 592eeecbf5514215b6464bd030081e4b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.029904 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2aa20459484e4b98e0ae372418020f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:58.029754 24175 consensus_queue.cc:260] T 592eeecbf5514215b6464bd030081e4b P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.033427 24175 ts_tablet_manager.cc:1398] T 592eeecbf5514215b6464bd030081e4b P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:16:58.033850 24175 tablet_bootstrap.cc:492] T 156c7356ece342aea0d8db667ebad89c P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:58.035547 24173 tablet_bootstrap.cc:654] T ee022e8e86f8477a8fa1e58d52644b03 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.037714 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f9e61686dc417bb5a88f96b4231f2e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.043301 23605 master_service.cc:946] ReplaceTablet: received request to replace tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9 from {username='slave'} at 127.0.0.1:36668
W20240502 08:16:58.044706 23996 consensus_peers.cc:485] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:58.045398 24175 tablet_bootstrap.cc:654] T 156c7356ece342aea0d8db667ebad89c P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.050603 24034 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2aa20459484e4b98e0ae372418020f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.052495 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9
I20240502 08:16:58.053136 23605 catalog_manager.cc:6554] ReplaceTablet: tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9 deleted and replaced by tablet 9578fd7ce1d545fca6524accc783e38b
I20240502 08:16:58.055321 24031 tablet_service.cc:1467] Processing CreateTablet for tablet c86680b5d8d24c55b9c57916c6d97e2d (DEFAULT_TABLE table=default.Table_3209c07b69e64b7a9f32f74fa58aef11 [id=dfb9dce29b174be7a541a1e6cb814efc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.055748 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86680b5d8d24c55b9c57916c6d97e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.057089 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 1ded3cee64f9409583040ef4b6e1f803 (DEFAULT_TABLE table=default.Table_f3099e422df24ee593db8bca89aba612 [id=2b8a7a487bb441b596a3578786f77172]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.057390 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ded3cee64f9409583040ef4b6e1f803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.061256 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6a6b2f29d9aa4a679e4c41863e3096bd" } new_table_name: "default.Table_f0aaafe0c6b64f8194b65bd904533211" modify_external_catalogs: true
I20240502 08:16:58.061517 24023 tablet_service.cc:1467] Processing CreateTablet for tablet fe1bafedb8f04f7c95683fbb15008831 (DEFAULT_TABLE table=default.Table_61b10a2ece3e45678e32119eea2f3c0d [id=e4237f1908f646049c3b8983b3a828f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.061908 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe1bafedb8f04f7c95683fbb15008831. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.063618 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef622696fe7435abe07808f5e8e918d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.065754 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ce5f27ca66f44a44af682629af53654f" } new_table_name: "default.Table_fa5057b6907545b7be4b9b4ed2026e15" modify_external_catalogs: true
I20240502 08:16:58.065840 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 990d0cc863ee4f49a2fc2fffb217c4d7 (DEFAULT_TABLE table=default.Table_1c914a8e055b422e8029a6a02b4a2925 [id=24f7d6e752a64e75b372d51f8e6f4ee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.066325 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 990d0cc863ee4f49a2fc2fffb217c4d7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:58.066671 24053 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.006s sys 0.006s
W20240502 08:16:58.066805 24053 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.521s user 0.006s sys 0.006s
I20240502 08:16:58.069978 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_249529c4082f44ccba05014ebef7543b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:58.070577 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_249529c4082f44ccba05014ebef7543b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:58.071134 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf7386fdb7649258b2e795aa46b501e (DEFAULT_TABLE table=default.Table_8eb9563cd4c74c64a4ba798132ddf385 [id=1e1704b3553c427daa67ad579814774a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.071475 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf7386fdb7649258b2e795aa46b501e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.077740 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet 3af208d6231a4cf18823e9fe367ca46d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:58.079123 24033 tablet_service.cc:1467] Processing CreateTablet for tablet fd32de329009465c89a4917e2716d3e9 (DEFAULT_TABLE table=default.Table_465db433d6a5443bac1914682bc40769 [id=54ecfc07bae64409aae7ba8effde4685]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.079479 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd32de329009465c89a4917e2716d3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.079823 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 1424b28b227f42c790d2e85e44f72ce3 (DEFAULT_TABLE table=default.Table_6e99bd5e0b2a4692a1400b64cb638f26 [id=1ec0052611154410a87adeae83c994a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.080215 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1424b28b227f42c790d2e85e44f72ce3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:58.083112 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1d8641acec9467fbc53b2ab90e04142 consensus state has no leader
W20240502 08:16:58.087993 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d68448a2033e450794440fa84de6a19a consensus state has no leader
I20240502 08:16:58.088826 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b79fbb283cbd4146aa1b162c998ab4e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:58.089542 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b79fbb283cbd4146aa1b162c998ab4e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:58.090189 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 25c089f0c1b14e419f250d03d4eaad12 (table default.Table_94efd28ada5b4024ab65e4d5feda0b38 [id=15b42ca7619645c8ac3cff50044f069b]) successfully deleted
I20240502 08:16:58.093344 24032 tablet_service.cc:1467] Processing CreateTablet for tablet ff3afe539daa4c77902e2fc136d0ccab (DEFAULT_TABLE table=default.Table_e8d89916bfe94d52af4f205db18f9e47 [id=b49962d67a7f45fe95c093e6809a9c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.096123 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3afe539daa4c77902e2fc136d0ccab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:57.448078 25114 rpcz_store.cc:276] 0502 08:16:51.012231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:51.012490 (+ 259us) service_pool.cc:225] Handling call
0502 08:16:56.630451 (+5617961us) raft_consensus.cc:1457] Updating replica for 517 ops
0502 08:16:56.676608 (+ 46157us) raft_consensus.cc:1520] Early marking committed up to index 538
0502 08:16:56.710191 (+ 33583us) raft_consensus.cc:1525] Triggering prepare for 517 ops
0502 08:16:56.710571 (+ 380us) write_op.cc:270] Start()
0502 08:16:56.710618 (+ 47us) write_op.cc:276] Timestamp: P: 1714637801324428 usec, L: 0
0502 08:16:56.710931 (+ 313us) write_op.cc:270] Start()
0502 08:16:56.710953 (+ 22us) write_op.cc:276] Timestamp: P: 1714637801496117 usec, L: 0
0502 08:16:56.711080 (+ 127us) write_op.cc:270] Start()
0502 08:16:56.711111 (+ 31us) write_op.cc:276] Timestamp: P: 1714637801496766 usec, L: 0
0502 08:16:56.711216 (+ 105us) write_op.cc:270] Start()
0502 08:16:56.711232 (+ 16us) write_op.cc:276] Timestamp: P: 1714637801497259 usec, L: 0
0502 08:16:56.711376 (+ 144us) write_op.cc:270] Start()
0502 08:16:56.711408 (+ 32us) write_op.cc:276] Timestamp: P: 1714637801506718 usec, L: 0
0502 08:16:56.711528 (+ 120us) write_op.cc:270] Start()
0502 08:16:56.711545 (+ 17us) write_op.cc:276] Timestamp: P: 1714637801507345 usec, L: 0
0502 08:16:56.711683 (+ 138us) write_op.cc:270] Start()
0502 08:16:56.711721 (+ 38us) write_op.cc:276] Timestamp: P: 1714637801507807 usec, L: 0
0502 08:16:56.711834 (+ 113us) write_op.cc:270] Start()
0502 08:16:56.711851 (+ 17us) write_op.cc:276] Timestamp: P: 1714637801516087 usec, L: 0
0502 08:16:56.711987 (+ 136us) write_op.cc:270] Start()
0502 08:16:56.712005 (+ 18us) write_op.cc:276] Timestamp: P: 1714637801544568 usec, L: 0
0502 08:16:56.712124 (+ 119us) write_op.cc:270] Start()
0502 08:16:56.712141 (+ 17us) write_op.cc:276] Timestamp: P: 1714637801545320 usec, L: 0
0502 08:16:56.712251 (+ 110us) write_op.cc:270] Start()
0502 08:16:56.714049 (+ 1798us) write_op.cc:276] Timestamp: P: 1714637801574939 usec, L: 0
0502 08:16:56.714208 (+ 159us) write_op.cc:270] Start()
0502 08:16:56.714228 (+ 20us) write_op.cc:276] Timestamp: P: 1714637801575617 usec, L: 0
0502 08:16:56.714347 (+ 119us) write_op.cc:270] Start()
0502 08:16:56.714364 (+ 17us) write_op.cc:276] Timestamp: P: 1714637801576205 usec, L: 0
0502 08:16:56.714483 (+ 119us) write_op.cc:270] Start()
0502 08:16:56.714500 (+ 17us) write_op.cc:276] Timestamp: P: 1714637801668666 usec, L: 0
0502 08:16:56.714613 (+ 113us) write_op.cc:270] Start()
0502 08:16:56.714636 (+ 23us) write_op.cc:276] Timestamp: P: 1714637801699963 usec, L: 0
0502 08:16:56.714747 (+ 111us) write_op.cc:270] Start()
0502 08:16:56.714763 (+ 16us) write_op.cc:276] Timestamp: P: 1714637801738545 usec, L: 0
0502 08:16:56.714869 (+ 106us) write_op.cc:270] Start()
0502 08:16:56.714885 (+ 16us) write_op.cc:276] Timestamp: P: 1714637801739416 usec, L: 0
0502 08:16:56.715020 (+ 135us) write_op.cc:270] Start()
0502 08:16:56.715037 (+ 17us) write_op.cc:276] Timestamp: P: 1714637801742903 usec, L: 0
0502 08:16:56.715153 (+ 116us) write_op.cc:270] Start()
0502 08:16:56.715170 (+ 17us) write_op.cc:276] Timestamp: P: 1714637801751193 usec, L: 0
0502 08:16:56.715273 (+ 103us) write_op.cc:270] Start()
0502 08:16:56.715291 (+ 18us) write_op.cc:276] Timestamp: P: 1714637801751803 usec, L: 0
0502 08:16:56.715416 (+ 125us) write_op.cc:270] Start()
0502 08:16:56.715433 (+ 17us) write_op.cc:276] Timestamp: P: 1714637801752335 usec, L: 0
0502 08:16:56.715538 (+ 105us) write_op.cc:270] Start()
0502 08:16:56.715555 (+ 17us) write_op.cc:276] Timestamp: P: 1714637801752849 usec, L: 0
0502 08:16:56.715729 (+ 174us) write_op.cc:270] Start()
0502 08:16:56.715749 (+ 20us) write_op.cc:276] Timestamp: P: 1714637801760088 usec, L: 0
0502 08:16:56.715879 (+ 130us) write_op.cc:270] Start()
0502 08:16:56.715900 (+ 21us) write_op.cc:276] Timestamp: P: 1714637801760643 usec, L: 0
0502 08:16:56.716117 (+ 217us) write_op.cc:270] Start()
0502 08:16:56.720334 (+ 4217us) write_op.cc:276] Timestamp: P: 1714637801762585 usec, L: 0
0502 08:16:56.720511 (+ 177us) write_op.cc:270] Start()
0502 08:16:56.720532 (+ 21us) write_op.cc:276] Timestamp: P: 1714637801831975 usec, L: 0
0502 08:16:56.720650 (+ 118us) write_op.cc:270] Start()
0502 08:16:56.720667 (+ 17us) write_op.cc:276] Timestamp: P: 1714637801832651 usec, L: 0
0502 08:16:56.720767 (+ 100us) write_op.cc:270] Start()
0502 08:16:56.720782 (+ 15us) write_op.cc:276] Timestamp: P: 1714637801833238 usec, L: 0
0502 08:16:56.720899 (+ 117us) write_op.cc:270] Start()
0502 08:16:56.720915 (+ 16us) write_op.cc:276] Timestamp: P: 1714637801833780 usec, L: 0
0502 08:16:56.721012 (+ 97us) write_op.cc:270] Start()
0502 08:16:56.721026 (+ 14us) write_op.cc:276] Timestamp: P: 1714637801835049 usec, L: 0
0502 08:16:56.721119 (+ 93us) write_op.cc:270] Start()
0502 08:16:56.721133 (+ 14us) write_op.cc:276] Timestamp: P: 1714637801835939 usec, L: 0
0502 08:16:56.721229 (+ 96us) write_op.cc:270] Start()
0502 08:16:56.721243 (+ 14us) write_op.cc:276] Timestamp: P: 1714637801837186 usec, L: 0
0502 08:16:56.721353 (+ 110us) write_op.cc:270] Start()
0502 08:16:56.721368 (+ 15us) write_op.cc:276] Timestamp: P: 1714637801874511 usec, L: 0
0502 08:16:56.721469 (+ 101us) write_op.cc:270] Start()
0502 08:16:56.721484 (+ 15us) write_op.cc:276] Timestamp: P: 1714637801880040 usec, L: 0
0502 08:16:56.729585 (+ 8101us) write_op.cc:270] Start()
0502 08:16:56.729624 (+ 39us) write_op.cc:276] Timestamp: P: 1714637801906369 usec, L: 0
0502 08:16:56.729808 (+ 184us) write_op.cc:270] Start()
0502 08:16:56.729830 (+ 22us) write_op.cc:276] Timestamp: P: 1714637801922428 usec, L: 0
0502 08:16:56.729959 (+ 129us) write_op.cc:270] Start()
0502 08:16:56.729994 (+ 35us) write_op.cc:276] Timestamp: P: 1714637801927373 usec, L: 0
0502 08:16:56.730168 (+ 174us) write_op.cc:270] Start()
0502 08:16:56.730188 (+ 20us) write_op.cc:276] Timestamp: P: 1714637802033163 usec, L: 0
0502 08:16:56.730333 (+ 145us) write_op.cc:270] Start()
0502 08:16:56.730353 (+ 20us) write_op.cc:276] Timestamp: P: 1714637802035014 usec, L: 0
0502 08:16:56.730483 (+ 130us) write_op.cc:270] Start()
0502 08:16:56.730512 (+ 29us) write_op.cc:276] Timestamp: P: 1714637802035896 usec, L: 0
0502 08:16:56.730627 (+ 115us) write_op.cc:270] Start()
0502 08:16:56.730648 (+ 21us) write_op.cc:276] Timestamp: P: 1714637802059967 usec, L: 0
0502 08:16:56.737704 (+ 7056us) write_op.cc:270] Start()
0502 08:16:56.737737 (+ 33us) write_op.cc:276] Timestamp: P: 1714637802062902 usec, L: 0
0502 08:16:56.739041 (+ 1304us) write_op.cc:270] Start()
0502 08:16:56.739070 (+ 29us) write_op.cc:276] Timestamp: P: 1714637802063642 usec, L: 0
0502 08:16:56.739212 (+ 142us) write_op.cc:270] Start()
0502 08:16:56.739227 (+ 15us) write_op.cc:276] Timestamp: P: 1714637802064299 usec, L: 0
0502 08:16:56.739322 (+ 95us) write_op.cc:270] Start()
0502 08:16:56.739336 (+ 14us) write_op.cc:276] Timestamp: P: 1714637802065005 usec, L: 0
0502 08:16:56.739434 (+ 98us) write_op.cc:270] Start()
0502 08:16:56.739449 (+ 15us) write_op.cc:276] Timestamp: P: 1714637802085667 usec, L: 0
0502 08:16:56.739681 (+ 232us) write_op.cc:270] Start()
0502 08:16:56.739702 (+ 21us) write_op.cc:276] Timestamp: P: 1714637802088376 usec, L: 0
0502 08:16:56.739854 (+ 152us) write_op.cc:270] Start()
0502 08:16:56.739874 (+ 20us) write_op.cc:276] Timestamp: P: 1714637802088988 usec, L: 0
0502 08:16:56.740011 (+ 137us) write_op.cc:270] Start()
0502 08:16:56.740033 (+ 22us) write_op.cc:276] Timestamp: P: 1714637802113541 usec, L: 0
0502 08:16:56.740181 (+ 148us) write_op.cc:270] Start()
0502 08:16:56.740200 (+ 19us) write_op.cc:276] Timestamp: P: 1714637802120577 usec, L: 0
0502 08:16:56.740333 (+ 133us) write_op.cc:270] Start()
0502 08:16:56.740356 (+ 23us) write_op.cc:276] Timestamp: P: 1714637802148053 usec, L: 0
0502 08:16:56.740496 (+ 140us) write_op.cc:270] Start()
0502 08:16:56.740515 (+ 19us) write_op.cc:276] Timestamp: P: 1714637802158489 usec, L: 0
0502 08:16:56.740644 (+ 129us) write_op.cc:270] Start()
0502 08:16:56.740661 (+ 17us) write_op.cc:276] Timestamp: P: 1714637802160803 usec, L: 0
0502 08:16:56.740772 (+ 111us) write_op.cc:270] Start()
0502 08:16:56.740789 (+ 17us) write_op.cc:276] Timestamp: P: 1714637802173858 usec, L: 0
0502 08:16:56.740901 (+ 112us) write_op.cc:270] Start()
0502 08:16:56.740919 (+ 18us) write_op.cc:276] Timestamp: P: 1714637802236643 usec, L: 0
0502 08:16:56.741058 (+ 139us) write_op.cc:270] Start()
0502 08:16:56.741107 (+ 49us) write_op.cc:276] Timestamp: P: 1714637802242989 usec, L: 0
0502 08:16:56.741238 (+ 131us) write_op.cc:270] Start()
0502 08:16:56.741255 (+ 17us) write_op.cc:276] Timestamp: P: 1714637802246201 usec, L: 0
0502 08:16:56.741369 (+ 114us) write_op.cc:270] Start()
0502 08:16:56.741386 (+ 17us) write_op.cc:276] Timestamp: P: 1714637802270204 usec, L: 0
0502 08:16:56.741522 (+ 136us) write_op.cc:270] Start()
0502 08:16:56.741539 (+ 17us) write_op.cc:276] Timestamp: P: 1714637802270959 usec, L: 0
0502 08:16:56.741672 (+ 133us) write_op.cc:270] Start()
0502 08:16:56.741721 (+ 49us) write_op.cc:276] Timestamp: P: 1714637802271512 usec, L: 0
0502 08:16:56.741844 (+ 123us) write_op.cc:270] Start()
0502 08:16:56.741862 (+ 18us) write_op.cc:276] Timestamp: P: 1714637802272067 usec, L: 0
0502 08:16:56.741979 (+ 117us) write_op.cc:270] Start()
0502 08:16:56.741998 (+ 19us) write_op.cc:276] Timestamp: P: 1714637802272598 usec, L: 0
0502 08:16:56.742107 (+ 109us) write_op.cc:270] Start()
0502 08:16:56.742125 (+ 18us) write_op.cc:276] Timestamp: P: 1714637802277500 usec, L: 0
0502 08:16:56.742272 (+ 147us) write_op.cc:270] Start()
0502 08:16:56.742292 (+ 20us) write_op.cc:276] Timestamp: P: 1714637802293356 usec, L: 0
0502 08:16:56.742438 (+ 146us) write_op.cc:270] Start()
0502 08:16:56.742456 (+ 18us) write_op.cc:276] Timestamp: P: 1714637802294084 usec, L: 0
0502 08:16:56.742593 (+ 137us) write_op.cc:270] Start()
0502 08:16:56.742611 (+ 18us) write_op.cc:276] Timestamp: P: 1714637802313298 usec, L: 0
0502 08:16:56.742713 (+ 102us) write_op.cc:270] Start()
0502 08:16:56.742728 (+ 15us) write_op.cc:276] Timestamp: P: 1714637802344762 usec, L: 0
0502 08:16:56.742812 (+ 84us) write_op.cc:270] Start()
0502 08:16:56.742827 (+ 15us) write_op.cc:276] Timestamp: P: 1714637802345630 usec, L: 0
0502 08:16:56.742911 (+ 84us) write_op.cc:270] Start()
0502 08:16:56.742925 (+ 14us) write_op.cc:276] Timestamp: P: 1714637802346801 usec, L: 0
0502 08:16:56.743011 (+ 86us) write_op.cc:270] Start()
0502 08:16:56.743025 (+ 14us) write_op.cc:276] Timestamp: P: 1714637802347443 usec, L: 0
0502 08:16:56.743128 (+ 103us) write_op.cc:270] Start()
0502 08:16:56.743143 (+ 15us) write_op.cc:276] Timestamp: P: 1714637802348018 usec, L: 0
0502 08:16:56.743234 (+ 91us) write_op.cc:270] Start()
0502 08:16:56.743254 (+ 20us) write_op.cc:276] Timestamp: P: 1714637802349176 usec, L: 0
0502 08:16:56.743345 (+ 91us) write_op.cc:270] Start()
0502 08:16:56.743360 (+ 15us) write_op.cc:276] Timestamp: P: 1714637802432247 usec, L: 0
0502 08:16:56.743477 (+ 117us) write_op.cc:270] Start()
0502 08:16:56.743492 (+ 15us) write_op.cc:276] Timestamp: P: 1714637802433966 usec, L: 0
0502 08:16:56.743589 (+ 97us) write_op.cc:270] Start()
0502 08:16:56.743602 (+ 13us) write_op.cc:276] Timestamp: P: 1714637802434622 usec, L: 0
0502 08:16:56.761716 (+ 18114us) write_op.cc:270] Start()
0502 08:16:56.761773 (+ 57us) write_op.cc:276] Timestamp: P: 1714637802467785 usec, L: 0
0502 08:16:56.762020 (+ 247us) write_op.cc:270] Start()
0502 08:16:56.762050 (+ 30us) write_op.cc:276] Timestamp: P: 1714637802468673 usec, L: 0
0502 08:16:56.762190 (+ 140us) write_op.cc:270] Start()
0502 08:16:56.762213 (+ 23us) write_op.cc:276] Timestamp: P: 1714637802491798 usec, L: 0
0502 08:16:56.762345 (+ 132us) write_op.cc:270] Start()
0502 08:16:56.762412 (+ 67us) write_op.cc:276] Timestamp: P: 1714637802527664 usec, L: 0
0502 08:16:56.762551 (+ 139us) write_op.cc:270] Start()
0502 08:16:56.762570 (+ 19us) write_op.cc:276] Timestamp: P: 1714637802578837 usec, L: 0
0502 08:16:56.762692 (+ 122us) write_op.cc:270] Start()
0502 08:16:56.762711 (+ 19us) write_op.cc:276] Timestamp: P: 1714637802586449 usec, L: 0
0502 08:16:56.762829 (+ 118us) write_op.cc:270] Start()
0502 08:16:56.762847 (+ 18us) write_op.cc:276] Timestamp: P: 1714637802590819 usec, L: 0
0502 08:16:56.762969 (+ 122us) write_op.cc:270] Start()
0502 08:16:56.762988 (+ 19us) write_op.cc:276] Timestamp: P: 1714637802596716 usec, L: 0
0502 08:16:56.763097 (+ 109us) write_op.cc:270] Start()
0502 08:16:56.763118 (+ 21us) write_op.cc:276] Timestamp: P: 1714637802614234 usec, L: 0
0502 08:16:56.763293 (+ 175us) write_op.cc:270] Start()
0502 08:16:56.763314 (+ 21us) write_op.cc:276] Timestamp: P: 1714637802636596 usec, L: 0
0502 08:16:56.763496 (+ 182us) write_op.cc:270] Start()
0502 08:16:56.763519 (+ 23us) write_op.cc:276] Timestamp: P: 1714637802672933 usec, L: 0
0502 08:16:56.763661 (+ 142us) write_op.cc:270] Start()
0502 08:16:56.763682 (+ 21us) write_op.cc:276] Timestamp: P: 1714637802693378 usec, L: 0
0502 08:16:56.763799 (+ 117us) write_op.cc:270] Start()
0502 08:16:56.763833 (+ 34us) write_op.cc:276] Timestamp: P: 1714637802709582 usec, L: 0
0502 08:16:56.764076 (+ 243us) write_op.cc:270] Start()
0502 08:16:56.764115 (+ 39us) write_op.cc:276] Timestamp: P: 1714637802720486 usec, L: 0
0502 08:16:56.764322 (+ 207us) write_op.cc:270] Start()
0502 08:16:56.764347 (+ 25us) write_op.cc:276] Timestamp: P: 1714637802734738 usec, L: 0
0502 08:16:56.764498 (+ 151us) write_op.cc:270] Start()
0502 08:16:56.764521 (+ 23us) write_op.cc:276] Timestamp: P: 1714637802747999 usec, L: 0
0502 08:16:56.764650 (+ 129us) write_op.cc:270] Start()
0502 08:16:56.764672 (+ 22us) write_op.cc:276] Timestamp: P: 1714637802807368 usec, L: 0
0502 08:16:56.764804 (+ 132us) write_op.cc:270] Start()
0502 08:16:56.764826 (+ 22us) write_op.cc:276] Timestamp: P: 1714637802836496 usec, L: 0
0502 08:16:56.765005 (+ 179us) write_op.cc:270] Start()
0502 08:16:56.765028 (+ 23us) write_op.cc:276] Timestamp: P: 1714637802837525 usec, L: 0
0502 08:16:56.765168 (+ 140us) write_op.cc:270] Start()
0502 08:16:56.765190 (+ 22us) write_op.cc:276] Timestamp: P: 1714637802838435 usec, L: 0
0502 08:16:56.765352 (+ 162us) write_op.cc:270] Start()
0502 08:16:56.765394 (+ 42us) write_op.cc:276] Timestamp: P: 1714637802839261 usec, L: 0
0502 08:16:56.765528 (+ 134us) write_op.cc:270] Start()
0502 08:16:56.765555 (+ 27us) write_op.cc:276] Timestamp: P: 1714637802853123 usec, L: 0
0502 08:16:56.765762 (+ 207us) write_op.cc:270] Start()
0502 08:16:56.765794 (+ 32us) write_op.cc:276] Timestamp: P: 1714637802853845 usec, L: 0
0502 08:16:56.766010 (+ 216us) write_op.cc:270] Start()
0502 08:16:56.766038 (+ 28us) write_op.cc:276] Timestamp: P: 1714637802858783 usec, L: 0
0502 08:16:56.766179 (+ 141us) write_op.cc:270] Start()
0502 08:16:56.766206 (+ 27us) write_op.cc:276] Timestamp: P: 1714637802862165 usec, L: 0
0502 08:16:56.769701 (+ 3495us) write_op.cc:270] Start()
0502 08:16:56.769746 (+ 45us) write_op.cc:276] Timestamp: P: 1714637802863939 usec, L: 0
0502 08:16:56.769954 (+ 208us) write_op.cc:270] Start()
0502 08:16:56.769972 (+ 18us) write_op.cc:276] Timestamp: P: 1714637802869938 usec, L: 0
0502 08:16:56.770173 (+ 201us) write_op.cc:270] Start()
0502 08:16:56.770201 (+ 28us) write_op.cc:276] Timestamp: P: 1714637802870673 usec, L: 0
0502 08:16:56.770365 (+ 164us) write_op.cc:270] Start()
0502 08:16:56.770391 (+ 26us) write_op.cc:276] Timestamp: P: 1714637802879494 usec, L: 0
0502 08:16:56.770543 (+ 152us) write_op.cc:270] Start()
0502 08:16:56.770567 (+ 24us) write_op.cc:276] Timestamp: P: 1714637802881439 usec, L: 0
0502 08:16:56.770733 (+ 166us) write_op.cc:270] Start()
0502 08:16:56.770758 (+ 25us) write_op.cc:276] Timestamp: P: 1714637802889899 usec, L: 0
0502 08:16:56.770894 (+ 136us) write_op.cc:270] Start()
0502 08:16:56.770925 (+ 31us) write_op.cc:276] Timestamp: P: 1714637802897978 usec, L: 0
0502 08:16:56.771051 (+ 126us) write_op.cc:270] Start()
0502 08:16:56.771070 (+ 19us) write_op.cc:276] Timestamp: P: 1714637802923773 usec, L: 0
0502 08:16:56.771207 (+ 137us) write_op.cc:270] Start()
0502 08:16:56.771227 (+ 20us) write_op.cc:276] Timestamp: P: 1714637802925590 usec, L: 0
0502 08:16:56.771418 (+ 191us) write_op.cc:270] Start()
0502 08:16:56.771449 (+ 31us) write_op.cc:276] Timestamp: P: 1714637802934800 usec, L: 0
0502 08:16:56.771643 (+ 194us) write_op.cc:270] Start()
0502 08:16:56.771669 (+ 26us) write_op.cc:276] Timestamp: P: 1714637802942827 usec, L: 0
0502 08:16:56.771847 (+ 178us) write_op.cc:270] Start()
0502 08:16:56.771875 (+ 28us) write_op.cc:276] Timestamp: P: 1714637802945115 usec, L: 0
0502 08:16:56.772039 (+ 164us) write_op.cc:270] Start()
0502 08:16:56.772067 (+ 28us) write_op.cc:276] Timestamp: P: 1714637802949365 usec, L: 0
0502 08:16:56.772225 (+ 158us) write_op.cc:270] Start()
0502 08:16:56.772252 (+ 27us) write_op.cc:276] Timestamp: P: 1714637802951930 usec, L: 0
0502 08:16:56.772470 (+ 218us) write_op.cc:270] Start()
0502 08:16:56.772497 (+ 27us) write_op.cc:276] Timestamp: P: 1714637802956700 usec, L: 0
0502 08:16:56.772674 (+ 177us) write_op.cc:270] Start()
0502 08:16:56.772701 (+ 27us) write_op.cc:276] Timestamp: P: 1714637802980609 usec, L: 0
0502 08:16:56.772872 (+ 171us) write_op.cc:270] Start()
0502 08:16:56.772892 (+ 20us) write_op.cc:276] Timestamp: P: 1714637803046364 usec, L: 0
0502 08:16:56.773044 (+ 152us) write_op.cc:270] Start()
0502 08:16:56.773071 (+ 27us) write_op.cc:276] Timestamp: P: 1714637803075362 usec, L: 0
0502 08:16:56.773269 (+ 198us) write_op.cc:270] Start()
0502 08:16:56.773292 (+ 23us) write_op.cc:276] Timestamp: P: 1714637803076552 usec, L: 0
0502 08:16:56.773449 (+ 157us) write_op.cc:270] Start()
0502 08:16:56.773481 (+ 32us) write_op.cc:276] Timestamp: P: 1714637803079093 usec, L: 0
0502 08:16:56.773657 (+ 176us) write_op.cc:270] Start()
0502 08:16:56.773682 (+ 25us) write_op.cc:276] Timestamp: P: 1714637803079621 usec, L: 0
0502 08:16:56.773831 (+ 149us) write_op.cc:270] Start()
0502 08:16:56.773854 (+ 23us) write_op.cc:276] Timestamp: P: 1714637803080826 usec, L: 0
0502 08:16:56.774031 (+ 177us) write_op.cc:270] Start()
0502 08:16:56.774056 (+ 25us) write_op.cc:276] Timestamp: P: 1714637803093323 usec, L: 0
0502 08:16:56.789696 (+ 15640us) write_op.cc:270] Start()
0502 08:16:56.789741 (+ 45us) write_op.cc:276] Timestamp: P: 1714637803107029 usec, L: 0
0502 08:16:56.790022 (+ 281us) write_op.cc:270] Start()
0502 08:16:56.790053 (+ 31us) write_op.cc:276] Timestamp: P: 1714637803109542 usec, L: 0
0502 08:16:56.790247 (+ 194us) write_op.cc:270] Start()
0502 08:16:56.790272 (+ 25us) write_op.cc:276] Timestamp: P: 1714637803111680 usec, L: 0
0502 08:16:56.790471 (+ 199us) write_op.cc:270] Start()
0502 08:16:56.790496 (+ 25us) write_op.cc:276] Timestamp: P: 1714637803117013 usec, L: 0
0502 08:16:56.790647 (+ 151us) write_op.cc:270] Start()
0502 08:16:56.790663 (+ 16us) write_op.cc:276] Timestamp: P: 1714637803117719 usec, L: 0
0502 08:16:56.790826 (+ 163us) write_op.cc:270] Start()
0502 08:16:56.790851 (+ 25us) write_op.cc:276] Timestamp: P: 1714637803118246 usec, L: 0
0502 08:16:56.791035 (+ 184us) write_op.cc:270] Start()
0502 08:16:56.791064 (+ 29us) write_op.cc:276] Timestamp: P: 1714637803149321 usec, L: 0
0502 08:16:56.791230 (+ 166us) write_op.cc:270] Start()
0502 08:16:56.791287 (+ 57us) write_op.cc:276] Timestamp: P: 1714637803153230 usec, L: 0
0502 08:16:56.791448 (+ 161us) write_op.cc:270] Start()
0502 08:16:56.791473 (+ 25us) write_op.cc:276] Timestamp: P: 1714637803178553 usec, L: 0
0502 08:16:56.791608 (+ 135us) write_op.cc:270] Start()
0502 08:16:56.791633 (+ 25us) write_op.cc:276] Timestamp: P: 1714637803195932 usec, L: 0
0502 08:16:56.791778 (+ 145us) write_op.cc:270] Start()
0502 08:16:56.791798 (+ 20us) write_op.cc:276] Timestamp: P: 1714637803196698 usec, L: 0
0502 08:16:56.791978 (+ 180us) write_op.cc:270] Start()
0502 08:16:56.792006 (+ 28us) write_op.cc:276] Timestamp: P: 1714637803197304 usec, L: 0
0502 08:16:56.792169 (+ 163us) write_op.cc:270] Start()
0502 08:16:56.792191 (+ 22us) write_op.cc:276] Timestamp: P: 1714637803243988 usec, L: 0
0502 08:16:56.792352 (+ 161us) write_op.cc:270] Start()
0502 08:16:56.792384 (+ 32us) write_op.cc:276] Timestamp: P: 1714637803259672 usec, L: 0
0502 08:16:56.792515 (+ 131us) write_op.cc:270] Start()
0502 08:16:56.792535 (+ 20us) write_op.cc:276] Timestamp: P: 1714637803277241 usec, L: 0
0502 08:16:56.792652 (+ 117us) write_op.cc:270] Start()
0502 08:16:56.792672 (+ 20us) write_op.cc:276] Timestamp: P: 1714637803303777 usec, L: 0
0502 08:16:56.792808 (+ 136us) write_op.cc:270] Start()
0502 08:16:56.792832 (+ 24us) write_op.cc:276] Timestamp: P: 1714637803305636 usec, L: 0
0502 08:16:56.792969 (+ 137us) write_op.cc:270] Start()
0502 08:16:56.792988 (+ 19us) write_op.cc:276] Timestamp: P: 1714637803309019 usec, L: 0
0502 08:16:56.793120 (+ 132us) write_op.cc:270] Start()
0502 08:16:56.793142 (+ 22us) write_op.cc:276] Timestamp: P: 1714637803327334 usec, L: 0
0502 08:16:56.793278 (+ 136us) write_op.cc:270] Start()
0502 08:16:56.793301 (+ 23us) write_op.cc:276] Timestamp: P: 1714637803342184 usec, L: 0
0502 08:16:56.793455 (+ 154us) write_op.cc:270] Start()
0502 08:16:56.793476 (+ 21us) write_op.cc:276] Timestamp: P: 1714637803362514 usec, L: 0
0502 08:16:56.793647 (+ 171us) write_op.cc:270] Start()
0502 08:16:56.793674 (+ 27us) write_op.cc:276] Timestamp: P: 1714637803364885 usec, L: 0
0502 08:16:56.793828 (+ 154us) write_op.cc:270] Start()
0502 08:16:56.793849 (+ 21us) write_op.cc:276] Timestamp: P: 1714637803382803 usec, L: 0
0502 08:16:56.794007 (+ 158us) write_op.cc:270] Start()
0502 08:16:56.794029 (+ 22us) write_op.cc:276] Timestamp: P: 1714637803383475 usec, L: 0
0502 08:16:56.794166 (+ 137us) write_op.cc:270] Start()
0502 08:16:56.794187 (+ 21us) write_op.cc:276] Timestamp: P: 1714637803384104 usec, L: 0
0502 08:16:56.794331 (+ 144us) write_op.cc:270] Start()
0502 08:16:56.794365 (+ 34us) write_op.cc:276] Timestamp: P: 1714637803390651 usec, L: 0
0502 08:16:56.794552 (+ 187us) write_op.cc:270] Start()
0502 08:16:56.794573 (+ 21us) write_op.cc:276] Timestamp: P: 1714637803391164 usec, L: 0
0502 08:16:56.794698 (+ 125us) write_op.cc:270] Start()
0502 08:16:56.794721 (+ 23us) write_op.cc:276] Timestamp: P: 1714637803391622 usec, L: 0
0502 08:16:56.794884 (+ 163us) write_op.cc:270] Start()
0502 08:16:56.794905 (+ 21us) write_op.cc:276] Timestamp: P: 1714637803405654 usec, L: 0
0502 08:16:56.795042 (+ 137us) write_op.cc:270] Start()
0502 08:16:56.795062 (+ 20us) write_op.cc:276] Timestamp: P: 1714637803408608 usec, L: 0
0502 08:16:56.795183 (+ 121us) write_op.cc:270] Start()
0502 08:16:56.795201 (+ 18us) write_op.cc:276] Timestamp: P: 1714637803413314 usec, L: 0
0502 08:16:56.795315 (+ 114us) write_op.cc:270] Start()
0502 08:16:56.795333 (+ 18us) write_op.cc:276] Timestamp: P: 1714637803417314 usec, L: 0
0502 08:16:56.795452 (+ 119us) write_op.cc:270] Start()
0502 08:16:56.795470 (+ 18us) write_op.cc:276] Timestamp: P: 1714637803422016 usec, L: 0
0502 08:16:56.795587 (+ 117us) write_op.cc:270] Start()
0502 08:16:56.795606 (+ 19us) write_op.cc:276] Timestamp: P: 1714637803472435 usec, L: 0
0502 08:16:56.795743 (+ 137us) write_op.cc:270] Start()
0502 08:16:56.795764 (+ 21us) write_op.cc:276] Timestamp: P: 1714637803491805 usec, L: 0
0502 08:16:56.795882 (+ 118us) write_op.cc:270] Start()
0502 08:16:56.795901 (+ 19us) write_op.cc:276] Timestamp: P: 1714637803548168 usec, L: 0
0502 08:16:56.796016 (+ 115us) write_op.cc:270] Start()
0502 08:16:56.796034 (+ 18us) write_op.cc:276] Timestamp: P: 1714637803555126 usec, L: 0
0502 08:16:56.796147 (+ 113us) write_op.cc:270] Start()
0502 08:16:56.796165 (+ 18us) write_op.cc:276] Timestamp: P: 1714637803561132 usec, L: 0
0502 08:16:56.796270 (+ 105us) write_op.cc:270] Start()
0502 08:16:56.796287 (+ 17us) write_op.cc:276] Timestamp: P: 1714637803608376 usec, L: 0
0502 08:16:56.796446 (+ 159us) write_op.cc:270] Start()
0502 08:16:56.796466 (+ 20us) write_op.cc:276] Timestamp: P: 1714637803609072 usec, L: 0
0502 08:16:56.796619 (+ 153us) write_op.cc:270] Start()
0502 08:16:56.796641 (+ 22us) write_op.cc:276] Timestamp: P: 1714637803620220 usec, L: 0
0502 08:16:56.796756 (+ 115us) write_op.cc:270] Start()
0502 08:16:56.796776 (+ 20us) write_op.cc:276] Timestamp: P: 1714637803620846 usec, L: 0
0502 08:16:56.796923 (+ 147us) write_op.cc:270] Start()
0502 08:16:56.796940 (+ 17us) write_op.cc:276] Timestamp: P: 1714637803621442 usec, L: 0
0502 08:16:56.797065 (+ 125us) write_op.cc:270] Start()
0502 08:16:56.797117 (+ 52us) write_op.cc:276] Timestamp: P: 1714637803629274 usec, L: 0
0502 08:16:56.797268 (+ 151us) write_op.cc:270] Start()
0502 08:16:56.797289 (+ 21us) write_op.cc:276] Timestamp: P: 1714637803644220 usec, L: 0
0502 08:16:56.797461 (+ 172us) write_op.cc:270] Start()
0502 08:16:56.797486 (+ 25us) write_op.cc:276] Timestamp: P: 1714637803654412 usec, L: 0
0502 08:16:56.805642 (+ 8156us) write_op.cc:270] Start()
0502 08:16:56.805680 (+ 38us) write_op.cc:276] Timestamp: P: 1714637803680933 usec, L: 0
0502 08:16:56.805867 (+ 187us) write_op.cc:270] Start()
0502 08:16:56.805891 (+ 24us) write_op.cc:276] Timestamp: P: 1714637803681510 usec, L: 0
0502 08:16:56.806053 (+ 162us) write_op.cc:270] Start()
0502 08:16:56.806085 (+ 32us) write_op.cc:276] Timestamp: P: 1714637803684188 usec, L: 0
0502 08:16:56.806247 (+ 162us) write_op.cc:270] Start()
0502 08:16:56.806267 (+ 20us) write_op.cc:276] Timestamp: P: 1714637803684621 usec, L: 0
0502 08:16:56.806441 (+ 174us) write_op.cc:270] Start()
0502 08:16:56.806464 (+ 23us) write_op.cc:276] Timestamp: P: 1714637803685088 usec, L: 0
0502 08:16:56.806603 (+ 139us) write_op.cc:270] Start()
0502 08:16:56.806625 (+ 22us) write_op.cc:276] Timestamp: P: 1714637803685617 usec, L: 0
0502 08:16:56.806780 (+ 155us) write_op.cc:270] Start()
0502 08:16:56.806805 (+ 25us) write_op.cc:276] Timestamp: P: 1714637803686112 usec, L: 0
0502 08:16:56.806956 (+ 151us) write_op.cc:270] Start()
0502 08:16:56.806976 (+ 20us) write_op.cc:276] Timestamp: P: 1714637803686577 usec, L: 0
0502 08:16:56.809245 (+ 2269us) write_op.cc:270] Start()
0502 08:16:56.809269 (+ 24us) write_op.cc:276] Timestamp: P: 1714637803716964 usec, L: 0
0502 08:16:56.809441 (+ 172us) write_op.cc:270] Start()
0502 08:16:56.809463 (+ 22us) write_op.cc:276] Timestamp: P: 1714637803724893 usec, L: 0
0502 08:16:56.809624 (+ 161us) write_op.cc:270] Start()
0502 08:16:56.809649 (+ 25us) write_op.cc:276] Timestamp: P: 1714637803766493 usec, L: 0
0502 08:16:56.809791 (+ 142us) write_op.cc:270] Start()
0502 08:16:56.809812 (+ 21us) write_op.cc:276] Timestamp: P: 1714637803774091 usec, L: 0
0502 08:16:56.809975 (+ 163us) write_op.cc:270] Start()
0502 08:16:56.809996 (+ 21us) write_op.cc:276] Timestamp: P: 1714637803776065 usec, L: 0
0502 08:16:56.810109 (+ 113us) write_op.cc:270] Start()
0502 08:16:56.810127 (+ 18us) write_op.cc:276] Timestamp: P: 1714637803841059 usec, L: 0
0502 08:16:56.810258 (+ 131us) write_op.cc:270] Start()
0502 08:16:56.810280 (+ 22us) write_op.cc:276] Timestamp: P: 1714637803882283 usec, L: 0
0502 08:16:56.810420 (+ 140us) write_op.cc:270] Start()
0502 08:16:56.810440 (+ 20us) write_op.cc:276] Timestamp: P: 1714637803882948 usec, L: 0
0502 08:16:56.810563 (+ 123us) write_op.cc:270] Start()
0502 08:16:56.810581 (+ 18us) write_op.cc:276] Timestamp: P: 1714637803883407 usec, L: 0
0502 08:16:56.814053 (+ 3472us) write_op.cc:270] Start()
0502 08:16:56.814094 (+ 41us) write_op.cc:276] Timestamp: P: 1714637803883895 usec, L: 0
0502 08:16:56.814302 (+ 208us) write_op.cc:270] Start()
0502 08:16:56.814328 (+ 26us) write_op.cc:276] Timestamp: P: 1714637803884438 usec, L: 0
0502 08:16:56.814502 (+ 174us) write_op.cc:270] Start()
0502 08:16:56.814527 (+ 25us) write_op.cc:276] Timestamp: P: 1714637803902071 usec, L: 0
0502 08:16:56.814679 (+ 152us) write_op.cc:270] Start()
0502 08:16:56.814703 (+ 24us) write_op.cc:276] Timestamp: P: 1714637803902717 usec, L: 0
0502 08:16:56.814856 (+ 153us) write_op.cc:270] Start()
0502 08:16:56.814874 (+ 18us) write_op.cc:276] Timestamp: P: 1714637803903207 usec, L: 0
0502 08:16:56.815003 (+ 129us) write_op.cc:270] Start()
0502 08:16:56.815024 (+ 21us) write_op.cc:276] Timestamp: P: 1714637803930451 usec, L: 0
0502 08:16:56.815151 (+ 127us) write_op.cc:270] Start()
0502 08:16:56.815192 (+ 41us) write_op.cc:276] Timestamp: P: 1714637803931462 usec, L: 0
0502 08:16:56.815380 (+ 188us) write_op.cc:270] Start()
0502 08:16:56.815407 (+ 27us) write_op.cc:276] Timestamp: P: 1714637803931983 usec, L: 0
0502 08:16:56.815581 (+ 174us) write_op.cc:270] Start()
0502 08:16:56.815622 (+ 41us) write_op.cc:276] Timestamp: P: 1714637803943054 usec, L: 0
0502 08:16:56.815793 (+ 171us) write_op.cc:270] Start()
0502 08:16:56.815821 (+ 28us) write_op.cc:276] Timestamp: P: 1714637803944643 usec, L: 0
0502 08:16:56.815974 (+ 153us) write_op.cc:270] Start()
0502 08:16:56.816000 (+ 26us) write_op.cc:276] Timestamp: P: 17146
W20240502 08:16:58.100469 23597 catalog_manager.cc:4797] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 955e7bd2216646c6aef5acce8d5b0d77 because the tablet was not found. No further retry: Not found: Tablet not found: 955e7bd2216646c6aef5acce8d5b0d77
W20240502 08:16:58.100816 23597 catalog_manager.cc:4797] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 751808dfc1964b539f6f836d786156aa because the tablet was not found. No further retry: Not found: Tablet not found: 751808dfc1964b539f6f836d786156aa
W20240502 08:16:58.101768 23597 catalog_manager.cc:4797] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 577b149f52e44d5d9a9f6b9d85a6864d because the tablet was not found. No further retry: Not found: Tablet not found: 577b149f52e44d5d9a9f6b9d85a6864d
I20240502 08:16:58.104477 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bc133adb9a44cc7b56a3746d1eb76af with delete_type TABLET_DATA_DELETED (replaced by tablet f2016fc145314ad788b8c8c2c53bf810) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.110122 24023 tablet_service.cc:1467] Processing CreateTablet for tablet b2ff583f31f945cc9ce7cd8b0b3ad4a9 (DEFAULT_TABLE table=default.Table_ecb54a9c637e45e6ac9bbc1012d1d78f [id=41b2bd3fb1a445c3821994763a7d5dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.115367 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ff583f31f945cc9ce7cd8b0b3ad4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.116967 24181 ts_tablet_manager.cc:1888] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:58.120110 24181 log.cc:1196] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/e47459b5f5bb4ecb96f76240fefa1f5d
I20240502 08:16:58.125873 24181 ts_tablet_manager.cc:1909] T e47459b5f5bb4ecb96f76240fefa1f5d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:58.133978 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9 with delete_type TABLET_DATA_DELETED (replaced by tablet 9578fd7ce1d545fca6524accc783e38b) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.134855 24029 tablet_service.cc:1467] Processing CreateTablet for tablet c315472dabe5419a85a35bbf82d09004 (DEFAULT_TABLE table=default.Table_9975eed13bd34c0e8e61352b573d680a [id=7ec95bf0867e4c35a5e60704a66123a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.135972 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c315472dabe5419a85a35bbf82d09004. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.127029 24021 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a595654cff4565bcf21a7c8fabc2c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:57 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.131830 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 9df28a4bcf954a5f9f5ca18cc54be43e (DEFAULT_TABLE table=default.Table_191144ffbcfe41bfaef1158d4a866ec6 [id=a6ed1fdfb038492289e0cd3c4a0fd332]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.132544 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea48a733d9f488097cf99a8d91947c6 (DEFAULT_TABLE table=default.Table_d2ed4128fb3845cf9d0372ba1bb5bd61 [id=7180eb46a178479eaab9e8428783cc62]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.142998 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df28a4bcf954a5f9f5ca18cc54be43e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.144997 25387 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } } }
I20240502 08:16:58.145341 25387 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:58.145795 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ea48a733d9f488097cf99a8d91947c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.151626 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 1ded3cee64f9409583040ef4b6e1f803 (DEFAULT_TABLE table=default.Table_f3099e422df24ee593db8bca89aba612 [id=2b8a7a487bb441b596a3578786f77172]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.152019 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ded3cee64f9409583040ef4b6e1f803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.152379 23891 tablet_service.cc:1467] Processing CreateTablet for tablet c86680b5d8d24c55b9c57916c6d97e2d (DEFAULT_TABLE table=default.Table_3209c07b69e64b7a9f32f74fa58aef11 [id=dfb9dce29b174be7a541a1e6cb814efc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.152467 23895 tablet_service.cc:1467] Processing CreateTablet for tablet fe1bafedb8f04f7c95683fbb15008831 (DEFAULT_TABLE table=default.Table_61b10a2ece3e45678e32119eea2f3c0d [id=e4237f1908f646049c3b8983b3a828f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.152761 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86680b5d8d24c55b9c57916c6d97e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.153024 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe1bafedb8f04f7c95683fbb15008831. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.153909 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c456616beb454b6ea33638f85512f97e" } modify_external_catalogs: true
I20240502 08:16:58.154136 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c456616beb454b6ea33638f85512f97e" } modify_external_catalogs: true
I20240502 08:16:58.155189 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf7386fdb7649258b2e795aa46b501e (DEFAULT_TABLE table=default.Table_8eb9563cd4c74c64a4ba798132ddf385 [id=1e1704b3553c427daa67ad579814774a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.155731 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a33cd662ce1043e3853792e92624c9be" } modify_external_catalogs: true
I20240502 08:16:58.155943 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a33cd662ce1043e3853792e92624c9be" } modify_external_catalogs: true
I20240502 08:16:58.132843 24036 tablet_service.cc:1467] Processing CreateTablet for tablet d5c5f1890d19434280a07cc2fa936195 (DEFAULT_TABLE table=default.Table_7d9efd3ee8d94c048fb1ce4498a4167b [id=928b35ea47df408885572962f2ff00b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.157929 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c5f1890d19434280a07cc2fa936195. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.158870 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_19538f34f25a4fd79dd6f076de5c021e" } modify_external_catalogs: true
I20240502 08:16:58.159089 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_19538f34f25a4fd79dd6f076de5c021e" } modify_external_catalogs: true
I20240502 08:16:58.161228 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_aea78b7f89b04a068101eec9cff71223" } modify_external_catalogs: true
I20240502 08:16:58.161427 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_aea78b7f89b04a068101eec9cff71223" } modify_external_catalogs: true
I20240502 08:16:58.167685 24181 tablet_replica.cc:330] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:58.167927 24181 raft_consensus.cc:2229] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:58.168083 24181 raft_consensus.cc:2258] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:58.168588 24181 ts_tablet_manager.cc:1875] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:58.169394 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ff3ffdf15eba464b8f3d89be9f35a38f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:58.170013 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff3ffdf15eba464b8f3d89be9f35a38f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:58.171900 23607 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 74b4c563d6bd4a44bb3bab58a3fcdc07 (table default.Table_2ee0f7ddd41e41ea84e25359cd3ed0a4 [id=a9c009feee0e486794a8500121fe5487]). Expected version 2 got 0
I20240502 08:16:58.172165 23607 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 9761df0bbdee44af90135bffb8b5828f (table default.Table_8ba240c391da43978f93a27dce48e557 [id=1b5536950db244d1978b154da0be33e0]). Expected version 2 got 0
I20240502 08:16:58.172513 23607 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 001e3f4f72f64dc69dbd521724912f07 (table default.Table_d7cff78803d44e5aba3b50cd8ff11e92 [id=5c21117955d946418c656330984ae747]). Expected version 3 got 0
I20240502 08:16:58.173210 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3dd46e0e1b4e2aacd4b2f77a92d378 (DEFAULT_TABLE table=default.Table_3a88ce68a50f4471b706d37631957df0 [id=0dbc109d8f3f4bc18d321ae8f3c4ae9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.173651 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3dd46e0e1b4e2aacd4b2f77a92d378. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.174037 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d18d476de2064b649fe32153819d1176"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:58.174391 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 752d9448f5214a70a2971920e54f4103 (DEFAULT_TABLE table=default.Table_474c796dbc574f53b4b9b2121710fb89 [id=da9e30fa3fe64b55a3178a7b1368a163]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:58.174539 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d18d476de2064b649fe32153819d1176 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:58.174830 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752d9448f5214a70a2971920e54f4103. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.168541 24173 tablet_bootstrap.cc:492] T ee022e8e86f8477a8fa1e58d52644b03 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:58.178326 23903 tablet_service.cc:1467] Processing CreateTablet for tablet ff3afe539daa4c77902e2fc136d0ccab (DEFAULT_TABLE table=default.Table_e8d89916bfe94d52af4f205db18f9e47 [id=b49962d67a7f45fe95c093e6809a9c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.178439 23899 tablet_service.cc:1467] Processing CreateTablet for tablet fd32de329009465c89a4917e2716d3e9 (DEFAULT_TABLE table=default.Table_465db433d6a5443bac1914682bc40769 [id=54ecfc07bae64409aae7ba8effde4685]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.180065 25012 ts_tablet_manager.cc:1888] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:58.180606 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0d515c0c1c334feb9d4252e90ba9ad4e" } modify_external_catalogs: true
I20240502 08:16:58.181807 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 1424b28b227f42c790d2e85e44f72ce3 (DEFAULT_TABLE table=default.Table_6e99bd5e0b2a4692a1400b64cb638f26 [id=1ec0052611154410a87adeae83c994a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.183553 23905 tablet_service.cc:1467] Processing CreateTablet for tablet fdb8f870d6e4469ba8313c7f7e86444c (DEFAULT_TABLE table=default.Table_dfb2d9793c694bc1845e757adebff4e6 [id=1e48595500584cdea2e138aa3bee6373]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.184044 23893 tablet_service.cc:1467] Processing CreateTablet for tablet d5c5f1890d19434280a07cc2fa936195 (DEFAULT_TABLE table=default.Table_7d9efd3ee8d94c048fb1ce4498a4167b [id=928b35ea47df408885572962f2ff00b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.184510 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bc133adb9a44cc7b56a3746d1eb76af with delete_type TABLET_DATA_DELETED (replaced by tablet f2016fc145314ad788b8c8c2c53bf810) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:58.185735 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 18d284f71937459db0c0676ce4568175 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:38 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:58.185905 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0d515c0c1c334feb9d4252e90ba9ad4e" } modify_external_catalogs: true
I20240502 08:16:58.186700 24032 tablet_service.cc:1467] Processing CreateTablet for tablet fdb8f870d6e4469ba8313c7f7e86444c (DEFAULT_TABLE table=default.Table_dfb2d9793c694bc1845e757adebff4e6 [id=1e48595500584cdea2e138aa3bee6373]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.187819 24173 ts_tablet_manager.cc:1367] T ee022e8e86f8477a8fa1e58d52644b03 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.160s user 0.004s sys 0.002s
I20240502 08:16:58.189612 25012 log.cc:1196] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/24a4104f4cb64889ad66810788ad6b29
I20240502 08:16:58.189651 23897 tablet_service.cc:1467] Processing CreateTablet for tablet b2ff583f31f945cc9ce7cd8b0b3ad4a9 (DEFAULT_TABLE table=default.Table_ecb54a9c637e45e6ac9bbc1012d1d78f [id=41b2bd3fb1a445c3821994763a7d5dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.190073 25012 ts_tablet_manager.cc:1909] T 24a4104f4cb64889ad66810788ad6b29 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:58.190729 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 940) took 4031 ms. Trace:
I20240502 08:16:58.190837 25012 rpcz_store.cc:276] 0502 08:16:54.159017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.160973 (+ 1956us) service_pool.cc:225] Handling call
0502 08:16:57.949107 (+3788134us) ts_tablet_manager.cc:1120] Deleting tablet 24a4104f4cb64889ad66810788ad6b29
0502 08:16:57.949114 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:58.138371 (+189257us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:58.179915 (+ 41544us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:58.190453 (+ 10538us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3787915}
I20240502 08:16:58.201886 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdb8f870d6e4469ba8313c7f7e86444c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.202057 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 948) took 3978 ms. Trace:
I20240502 08:16:58.202314 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf7386fdb7649258b2e795aa46b501e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.202607 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ff583f31f945cc9ce7cd8b0b3ad4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.203240 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd32de329009465c89a4917e2716d3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.202284 25012 rpcz_store.cc:276] 0502 08:16:54.212545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.212707 (+ 162us) service_pool.cc:225] Handling call
0502 08:16:58.191144 (+3978437us) ts_tablet_manager.cc:1120] Deleting tablet 17d6f4f6e9ee46c28b7e7513f791c81f
0502 08:16:58.191152 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:58.191282 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3978206}
I20240502 08:16:58.206904 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1424b28b227f42c790d2e85e44f72ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.206995 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 949) took 3994 ms. Trace:
I20240502 08:16:58.207180 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdb8f870d6e4469ba8313c7f7e86444c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.207391 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3afe539daa4c77902e2fc136d0ccab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.209232 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet ee022e8e86f8477a8fa1e58d52644b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:58.210044 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c5f1890d19434280a07cc2fa936195. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.209987 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 9df28a4bcf954a5f9f5ca18cc54be43e (DEFAULT_TABLE table=default.Table_191144ffbcfe41bfaef1158d4a866ec6 [id=a6ed1fdfb038492289e0cd3c4a0fd332]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.210355 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df28a4bcf954a5f9f5ca18cc54be43e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.210335 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9 with delete_type TABLET_DATA_DELETED (replaced by tablet 9578fd7ce1d545fca6524accc783e38b) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:58.212450 23604 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet ec10ff7251a5442a8ecc1881b34db74d (table default.Table_236150d3903d437d95f67b0edec36091 [id=1d7a43116fe74c35b505eccd165f99cd]). Expected version 2 got 0
I20240502 08:16:58.214001 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea48a733d9f488097cf99a8d91947c6 (DEFAULT_TABLE table=default.Table_d2ed4128fb3845cf9d0372ba1bb5bd61 [id=7180eb46a178479eaab9e8428783cc62]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.214316 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ea48a733d9f488097cf99a8d91947c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.218291 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 752d9448f5214a70a2971920e54f4103 (DEFAULT_TABLE table=default.Table_474c796dbc574f53b4b9b2121710fb89 [id=da9e30fa3fe64b55a3178a7b1368a163]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.218608 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752d9448f5214a70a2971920e54f4103. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.207119 25012 rpcz_store.cc:276] 0502 08:16:54.212655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.226841 (+ 14186us) service_pool.cc:225] Handling call
0502 08:16:58.206807 (+3979966us) ts_tablet_manager.cc:1120] Deleting tablet 7a0ff1c3818a49d2b13f57ea6a81c3d6
0502 08:16:58.206816 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:58.206921 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3979728}
I20240502 08:16:58.223105 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 950) took 3995 ms. Trace:
I20240502 08:16:58.223197 25012 rpcz_store.cc:276] 0502 08:16:54.227297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.232259 (+ 4962us) service_pool.cc:225] Handling call
0502 08:16:58.222985 (+3990726us) ts_tablet_manager.cc:1120] Deleting tablet 518cb4b86ea34e95bef51c5c2a64d89b
0502 08:16:58.222997 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:58.223087 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3990511}
I20240502 08:16:58.223476 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 951) took 3988 ms. Trace:
I20240502 08:16:58.223543 25012 rpcz_store.cc:276] 0502 08:16:54.234878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.237701 (+ 2823us) service_pool.cc:225] Handling call
0502 08:16:58.223388 (+3985687us) ts_tablet_manager.cc:1120] Deleting tablet ac7a7cec2e154d6fa095d68d1ba3308d
0502 08:16:58.223396 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:58.223464 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3985443}
I20240502 08:16:58.223881 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 952) took 3985 ms. Trace:
I20240502 08:16:58.223954 25012 rpcz_store.cc:276] 0502 08:16:54.238092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.238254 (+ 162us) service_pool.cc:225] Handling call
0502 08:16:58.223757 (+3985503us) ts_tablet_manager.cc:1120] Deleting tablet e4ac2b5073e34cd98e0246620fc1515d
0502 08:16:58.223764 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:58.223856 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3985314}
I20240502 08:16:58.224246 25012 tablet_replica.cc:330] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:58.225646 25012 raft_consensus.cc:2229] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:58.226099 25012 raft_consensus.cc:2258] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:58.226349 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0911b0f21cf34f8a80774273c164a5d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:58.226665 25012 ts_tablet_manager.cc:1875] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:58.227043 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0911b0f21cf34f8a80774273c164a5d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:58.227608 24173 raft_consensus.cc:348] T ee022e8e86f8477a8fa1e58d52644b03 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.227838 24173 raft_consensus.cc:374] T ee022e8e86f8477a8fa1e58d52644b03 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.227911 24173 raft_consensus.cc:729] T ee022e8e86f8477a8fa1e58d52644b03 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.228827 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3dd46e0e1b4e2aacd4b2f77a92d378 (DEFAULT_TABLE table=default.Table_3a88ce68a50f4471b706d37631957df0 [id=0dbc109d8f3f4bc18d321ae8f3c4ae9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.229491 23893 tablet_service.cc:1467] Processing CreateTablet for tablet c315472dabe5419a85a35bbf82d09004 (DEFAULT_TABLE table=default.Table_9975eed13bd34c0e8e61352b573d680a [id=7ec95bf0867e4c35a5e60704a66123a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.230818 23895 tablet_service.cc:1514] Processing DeleteTablet for tablet 592eeecbf5514215b6464bd030081e4b with delete_type TABLET_DATA_DELETED (replaced by tablet 28a947b7e7484120b3f4069f6eef5520) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:58.231527 25494 raft_consensus.cc:482] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:58.231670 25494 raft_consensus.cc:504] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.232036 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d9e8b22b519c433ab85326949d23aeb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:58.232254 25494 leader_election.cc:288] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:58.232704 23917 raft_consensus.cc:3037] T 9761df0bbdee44af90135bffb8b5828f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:58.233026 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee022e8e86f8477a8fa1e58d52644b03" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:58.236794 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9e8b22b519c433ab85326949d23aeb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:58.239018 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 765fdb66f12941f0ac16223e931e576e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 765fdb66f12941f0ac16223e931e576e already in progress: creating tablet
W20240502 08:16:58.239389 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 552e46f69230474189278911cfeec6c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 552e46f69230474189278911cfeec6c7 already in progress: creating tablet
W20240502 08:16:58.239686 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet df266081ed8345dda989b3050f605685 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df266081ed8345dda989b3050f605685 already in progress: creating tablet
I20240502 08:16:58.245720 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 24a4104f4cb64889ad66810788ad6b29 (table default.Table_307041e730f248a5bd4d265f08aad5e4 [id=f2316137abe74f5c8e79faf0c4638540]) successfully deleted
W20240502 08:16:58.249248 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 17d6f4f6e9ee46c28b7e7513f791c81f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17d6f4f6e9ee46c28b7e7513f791c81f already in progress: creating tablet
I20240502 08:16:58.240391 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6c97a1a9206b45f39095fef784e77788"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:58.241961 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e0bd774640464d2a92b2e71126c83883
I20240502 08:16:58.253605 23601 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 9761df0bbdee44af90135bffb8b5828f (table default.Table_8ba240c391da43978f93a27dce48e557 [id=1b5536950db244d1978b154da0be33e0]). Expected version 2 got 0
I20240502 08:16:58.254263 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5b25dd22d0f24b7b9405582e1bc54ecb" } new_table_name: "default.Table_5a3f969f790640fe8d43a5cafc46b809" modify_external_catalogs: true
I20240502 08:16:58.242265 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 11ebacd4c41941cd8a6ec634e3aeeaf5
I20240502 08:16:58.256137 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_225db6e818fc4f8d85229eef53824c79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:58.256659 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_225db6e818fc4f8d85229eef53824c79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:58.242766 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 73d7633fd23541f3a553af008dd6a354
I20240502 08:16:58.258255 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d4e98fc3f0334516b7074a56d74d8332" } new_table_name: "default.Table_a52ae049081249eeb49eeb8d8620da1e" modify_external_catalogs: true
W20240502 08:16:58.258853 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 7a0ff1c3818a49d2b13f57ea6a81c3d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a0ff1c3818a49d2b13f57ea6a81c3d6 already in progress: creating tablet
I20240502 08:16:58.260733 23609 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 0141d4e966b844e59ce2144a0d570b02
I20240502 08:16:58.261476 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6e99bd5e0b2a4692a1400b64cb638f26" } new_table_name: "default.Table_6214e3b0246945ccaa4e060cf61ae4de" modify_external_catalogs: true
W20240502 08:16:58.263105 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 518cb4b86ea34e95bef51c5c2a64d89b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 518cb4b86ea34e95bef51c5c2a64d89b already in progress: creating tablet
W20240502 08:16:58.260876 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c97a1a9206b45f39095fef784e77788 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:58.268399 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_98f610e5d20b467e8e69ed5915b862ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:58.268429 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_bef474d64fe543c28023e61872f4b725"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:58.268849 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98f610e5d20b467e8e69ed5915b862ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:58.268891 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bef474d64fe543c28023e61872f4b725 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:58.270606 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 55fb41b8f8f9487a81e98bae9cddaae1
W20240502 08:16:58.271467 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet ac7a7cec2e154d6fa095d68d1ba3308d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac7a7cec2e154d6fa095d68d1ba3308d already in progress: creating tablet
I20240502 08:16:57.989769 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1424b28b227f42c790d2e85e44f72ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.273329 24180 log.cc:1196] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/63e645bb6e2948e091e08433944a52f4
I20240502 08:16:58.273499 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd32de329009465c89a4917e2716d3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.273711 24180 ts_tablet_manager.cc:1909] T 63e645bb6e2948e091e08433944a52f4 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:58.274606 24180 tablet_replica.cc:330] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:58.274704 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 63e645bb6e2948e091e08433944a52f4 (table default.Table_92a3c7cbfc16461ba5b08f262bc016d3 [id=47b70f46b4d345618054c69267d33c95]) successfully deleted
I20240502 08:16:58.274789 24180 raft_consensus.cc:2229] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:58.274945 24180 raft_consensus.cc:2258] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:58.275406 24180 ts_tablet_manager.cc:1875] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:58.005443 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b4c563d6bd4a44bb3bab58a3fcdc07" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:58.036938 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 9df28a4bcf954a5f9f5ca18cc54be43e (DEFAULT_TABLE table=default.Table_191144ffbcfe41bfaef1158d4a866ec6 [id=a6ed1fdfb038492289e0cd3c4a0fd332]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.277773 23783 raft_consensus.cc:2454] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:58.037226 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea48a733d9f488097cf99a8d91947c6 (DEFAULT_TABLE table=default.Table_d2ed4128fb3845cf9d0372ba1bb5bd61 [id=7180eb46a178479eaab9e8428783cc62]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.046015 23759 tablet_service.cc:1467] Processing CreateTablet for tablet d5c5f1890d19434280a07cc2fa936195 (DEFAULT_TABLE table=default.Table_7d9efd3ee8d94c048fb1ce4498a4167b [id=928b35ea47df408885572962f2ff00b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.074750 23763 tablet_service.cc:1467] Processing CreateTablet for tablet fdb8f870d6e4469ba8313c7f7e86444c (DEFAULT_TABLE table=default.Table_dfb2d9793c694bc1845e757adebff4e6 [id=1e48595500584cdea2e138aa3bee6373]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.088678 23760 tablet_service.cc:1467] Processing CreateTablet for tablet b2ff583f31f945cc9ce7cd8b0b3ad4a9 (DEFAULT_TABLE table=default.Table_ecb54a9c637e45e6ac9bbc1012d1d78f [id=41b2bd3fb1a445c3821994763a7d5dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.092181 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9 with delete_type TABLET_DATA_DELETED (replaced by tablet 9578fd7ce1d545fca6524accc783e38b) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:58.278383 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3dd46e0e1b4e2aacd4b2f77a92d378. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.278501 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 11ebacd4c41941cd8a6ec634e3aeeaf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:58.278625 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 73d7633fd23541f3a553af008dd6a354 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:58.278717 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 55fb41b8f8f9487a81e98bae9cddaae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:58.094588 23765 tablet_service.cc:1467] Processing CreateTablet for tablet c315472dabe5419a85a35bbf82d09004 (DEFAULT_TABLE table=default.Table_9975eed13bd34c0e8e61352b573d680a [id=7ec95bf0867e4c35a5e60704a66123a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.109381 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 752d9448f5214a70a2971920e54f4103 (DEFAULT_TABLE table=default.Table_474c796dbc574f53b4b9b2121710fb89 [id=da9e30fa3fe64b55a3178a7b1368a163]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.110242 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3dd46e0e1b4e2aacd4b2f77a92d378 (DEFAULT_TABLE table=default.Table_3a88ce68a50f4471b706d37631957df0 [id=0dbc109d8f3f4bc18d321ae8f3c4ae9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.110913 23752 tablet_service.cc:1467] Processing CreateTablet for tablet 64c3a2a82f004a088c959df50e664473 (DEFAULT_TABLE table=default.Table_df3f96e9bc0e43ef9c4d791737052252 [id=658f618912b940ad9df72ab1d357088e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.111670 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 0743a0f31b5441728536b818372d239e (DEFAULT_TABLE table=default.Table_bdb591e6a3e6408692d56430f64e36b7 [id=26262ec0ae254b93a5bd16d8060c4892]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.113389 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 8c860f268d284b0bb8cb05e4aa04baa5 (DEFAULT_TABLE table=default.Table_63fdac96c4d7418a9c60737466b8b0a9 [id=2af081997621493aa56be89b1f5cb097]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.113713 23767 tablet_service.cc:1467] Processing CreateTablet for tablet ff4a333c6dd44a13bc267e885bb9e487 (DEFAULT_TABLE table=default.Table_b49e692bc51f437281bdebe78679a3ff [id=3724d5ac2e8e4a9aba21ead1879a880e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.114404 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 0000f1984ab64164868de1ada6638007 (DEFAULT_TABLE table=default.Table_891c4bb8dd8d4f1c8de1dc4eacb711a1 [id=dc9b170f24484f73ae3be4dba368892d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.115149 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 59f335d067384fcf9f968aa7be53ac9e (DEFAULT_TABLE table=default.Table_eb6b182c8aff4600af282bf9dd035542 [id=da4b07a528b34fa5bf62e06889e05438]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.117066 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 448a78fb8079482baed5f75cddd746be (DEFAULT_TABLE table=default.Table_74c703e708b34d03891d74e02e877c53 [id=080df455dd864e5382c34baa9e02a2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.121073 23835 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:58.121330 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 43f9164589084269b9ecf3c7b9750e17 (DEFAULT_TABLE table=default.Table_7deeda2c4a754c3d8176faea44b99f1a [id=dc6c1432f15c4d8c8f293470fb9fed72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.189145 24174 tablet_bootstrap.cc:492] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:58.197746 25691 tablet.cc:1722] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:58.281715 24174 ts_tablet_manager.cc:1367] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.340s user 0.003s sys 0.000s
I20240502 08:16:58.260223 25531 raft_consensus.cc:482] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:58.283002 25531 raft_consensus.cc:504] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:58.284847 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c315472dabe5419a85a35bbf82d09004. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.287912 23993 leader_election.cc:302] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:58.266399 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet e0bd774640464d2a92b2e71126c83883 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:58.282491 24174 raft_consensus.cc:348] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.288712 25423 raft_consensus.cc:2781] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:58.289117 25423 raft_consensus.cc:482] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:58.289283 25423 raft_consensus.cc:3037] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:58.289222 24174 raft_consensus.cc:374] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.289608 24174 raft_consensus.cc:729] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.290226 24174 consensus_queue.cc:260] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.292110 24174 ts_tablet_manager.cc:1398] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240502 08:16:58.295614 25423 raft_consensus.cc:504] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.299679 25405 raft_consensus.cc:482] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:58.299947 25405 raft_consensus.cc:504] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.302419 25405 leader_election.cc:288] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:58.303156 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_382a0f031cf1457c8b008f9123f8af74" } new_table_name: "default.Table_467cf1c70b744bae9e7c09ec700c4335" modify_external_catalogs: true
I20240502 08:16:58.305483 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdb8f870d6e4469ba8313c7f7e86444c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.305737 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752d9448f5214a70a2971920e54f4103. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.305943 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 448a78fb8079482baed5f75cddd746be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.306129 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f9164589084269b9ecf3c7b9750e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.306299 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4a333c6dd44a13bc267e885bb9e487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.306514 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ff583f31f945cc9ce7cd8b0b3ad4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.306716 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f335d067384fcf9f968aa7be53ac9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.307076 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ea48a733d9f488097cf99a8d91947c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.309662 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df28a4bcf954a5f9f5ca18cc54be43e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.311762 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c860f268d284b0bb8cb05e4aa04baa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.313679 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c5f1890d19434280a07cc2fa936195. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.316258 24181 ts_tablet_manager.cc:1888] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:58.316371 24181 log.cc:1196] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/18d284f71937459db0c0676ce4568175
I20240502 08:16:58.316668 24181 ts_tablet_manager.cc:1909] T 18d284f71937459db0c0676ce4568175 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:58.317437 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0000f1984ab64164868de1ada6638007. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:58.273078 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e4ac2b5073e34cd98e0246620fc1515d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4ac2b5073e34cd98e0246620fc1515d already in progress: creating tablet
I20240502 08:16:58.321708 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3dd46e0e1b4e2aacd4b2f77a92d378. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.325678 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0743a0f31b5441728536b818372d239e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:58.327548 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 598cba0f8b9f4c8dae9e34872c4212fa consensus state has no leader
I20240502 08:16:58.328629 24173 consensus_queue.cc:260] T ee022e8e86f8477a8fa1e58d52644b03 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.328925 24053 raft_consensus.cc:2454] T ee022e8e86f8477a8fa1e58d52644b03 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:58.329465 24173 ts_tablet_manager.cc:1398] T ee022e8e86f8477a8fa1e58d52644b03 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.141s user 0.001s sys 0.001s
I20240502 08:16:58.329674 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c3a2a82f004a088c959df50e664473. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:58.321484 23597 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 48629.2us
I20240502 08:16:58.331050 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 0743a0f31b5441728536b818372d239e (DEFAULT_TABLE table=default.Table_bdb591e6a3e6408692d56430f64e36b7 [id=26262ec0ae254b93a5bd16d8060c4892]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.331483 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0743a0f31b5441728536b818372d239e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:58.332604 23993 leader_election.cc:341] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 592eeecbf5514215b6464bd030081e4b
I20240502 08:16:58.334134 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 64c3a2a82f004a088c959df50e664473 (DEFAULT_TABLE table=default.Table_df3f96e9bc0e43ef9c4d791737052252 [id=658f618912b940ad9df72ab1d357088e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.339643 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c315472dabe5419a85a35bbf82d09004. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.340554 24173 tablet_bootstrap.cc:492] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:58.342103 24174 tablet_bootstrap.cc:492] T b0e8982fe35f47cda376d9646b52e961 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:58.342973 24037 tablet_service.cc:1467] Processing CreateTablet for tablet ff4a333c6dd44a13bc267e885bb9e487 (DEFAULT_TABLE table=default.Table_b49e692bc51f437281bdebe78679a3ff [id=3724d5ac2e8e4a9aba21ead1879a880e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.343330 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4a333c6dd44a13bc267e885bb9e487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.343442 24174 tablet_bootstrap.cc:654] T b0e8982fe35f47cda376d9646b52e961 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.346249 25630 consensus_queue.cc:1035] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.052s
I20240502 08:16:58.349833 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c3a2a82f004a088c959df50e664473. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.350459 25423 leader_election.cc:288] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:58.356016 24021 tablet_service.cc:1467] Processing CreateTablet for tablet 59f335d067384fcf9f968aa7be53ac9e (DEFAULT_TABLE table=default.Table_eb6b182c8aff4600af282bf9dd035542 [id=da4b07a528b34fa5bf62e06889e05438]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.356370 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f335d067384fcf9f968aa7be53ac9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.356990 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b4c563d6bd4a44bb3bab58a3fcdc07" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:58.357291 23786 raft_consensus.cc:3037] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:58.360348 23786 raft_consensus.cc:2454] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:58.361006 23993 leader_election.cc:302] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:58.361814 23861 leader_election.cc:302] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:58.362113 25494 raft_consensus.cc:2781] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:58.362185 25494 raft_consensus.cc:482] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:58.362234 25494 raft_consensus.cc:3037] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:58.363082 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592eeecbf5514215b6464bd030081e4b" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:58.363258 23920 raft_consensus.cc:2454] T 592eeecbf5514215b6464bd030081e4b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:58.363639 25405 raft_consensus.cc:2781] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:58.363806 25405 raft_consensus.cc:686] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:58.364550 25494 raft_consensus.cc:504] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.354256 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 8c860f268d284b0bb8cb05e4aa04baa5 (DEFAULT_TABLE table=default.Table_63fdac96c4d7418a9c60737466b8b0a9 [id=2af081997621493aa56be89b1f5cb097]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.355139 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 0000f1984ab64164868de1ada6638007 (DEFAULT_TABLE table=default.Table_891c4bb8dd8d4f1c8de1dc4eacb711a1 [id=dc9b170f24484f73ae3be4dba368892d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.365088 25494 leader_election.cc:288] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:58.365130 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0000f1984ab64164868de1ada6638007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.366971 25531 leader_election.cc:288] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:58.372066 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c860f268d284b0bb8cb05e4aa04baa5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:58.373087 23858 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 121998us
I20240502 08:16:58.373944 25494 raft_consensus.cc:482] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:58.374060 25494 raft_consensus.cc:504] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.374617 25494 leader_election.cc:288] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:58.374647 23897 tablet_service.cc:1467] Processing CreateTablet for tablet ff4a333c6dd44a13bc267e885bb9e487 (DEFAULT_TABLE table=default.Table_b49e692bc51f437281bdebe78679a3ff [id=3724d5ac2e8e4a9aba21ead1879a880e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.375013 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 59f335d067384fcf9f968aa7be53ac9e (DEFAULT_TABLE table=default.Table_eb6b182c8aff4600af282bf9dd035542 [id=da4b07a528b34fa5bf62e06889e05438]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.375169 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 8c860f268d284b0bb8cb05e4aa04baa5 (DEFAULT_TABLE table=default.Table_63fdac96c4d7418a9c60737466b8b0a9 [id=2af081997621493aa56be89b1f5cb097]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.374708 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 0000f1984ab64164868de1ada6638007 (DEFAULT_TABLE table=default.Table_891c4bb8dd8d4f1c8de1dc4eacb711a1 [id=dc9b170f24484f73ae3be4dba368892d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:58.375566 23861 leader_election.cc:341] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 18d284f71937459db0c0676ce4568175
I20240502 08:16:58.375816 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 64c3a2a82f004a088c959df50e664473 (DEFAULT_TABLE table=default.Table_df3f96e9bc0e43ef9c4d791737052252 [id=658f618912b940ad9df72ab1d357088e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.377017 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 0743a0f31b5441728536b818372d239e (DEFAULT_TABLE table=default.Table_bdb591e6a3e6408692d56430f64e36b7 [id=26262ec0ae254b93a5bd16d8060c4892]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.378046 24174 tablet_bootstrap.cc:492] T b0e8982fe35f47cda376d9646b52e961 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:58.378207 24174 ts_tablet_manager.cc:1367] T b0e8982fe35f47cda376d9646b52e961 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20240502 08:16:58.379127 24174 raft_consensus.cc:348] T b0e8982fe35f47cda376d9646b52e961 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.379351 24174 raft_consensus.cc:374] T b0e8982fe35f47cda376d9646b52e961 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.379413 24174 raft_consensus.cc:729] T b0e8982fe35f47cda376d9646b52e961 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.379642 24174 consensus_queue.cc:260] T b0e8982fe35f47cda376d9646b52e961 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.380210 24174 ts_tablet_manager.cc:1398] T b0e8982fe35f47cda376d9646b52e961 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:58.380491 24174 tablet_bootstrap.cc:492] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:58.383956 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_195e6d7dde6049f188b2b301fac87a03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:58.384441 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_195e6d7dde6049f188b2b301fac87a03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:58.388844 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c99c7506b13949cf9c5534c4c5b9d3b4" } new_table_name: "default.Table_660621104fba4545bc2b8c30dc7d035a" modify_external_catalogs: true
I20240502 08:16:58.390844 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_42e74a4247614e8d9cafd1adf4935783"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:58.391927 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42e74a4247614e8d9cafd1adf4935783 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:58.394524 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 5e02e460f6ea4b269c178e80b4e0a3f4 (DEFAULT_TABLE table=default.Table_456a3cba3b8e4f09b405cd14e3785a23 [id=25b8092a8da04b04b8e7e7f2ae2f94d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.394932 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e02e460f6ea4b269c178e80b4e0a3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.395007 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b4c563d6bd4a44bb3bab58a3fcdc07" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:58.372895 23996 leader_election.cc:302] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:58.398607 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 448a78fb8079482baed5f75cddd746be (DEFAULT_TABLE table=default.Table_74c703e708b34d03891d74e02e877c53 [id=080df455dd864e5382c34baa9e02a2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.372901 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee022e8e86f8477a8fa1e58d52644b03" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:58.364087 25405 consensus_queue.cc:237] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.380474 24022 tablet_service.cc:1514] Processing DeleteTablet for tablet e0bd774640464d2a92b2e71126c83883 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.380590 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f83bcf640f465bb95d87a991cc5ff6" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:58.380760 24028 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f57e3a39d54b7ab372d4b145b7a696 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.380873 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 11ebacd4c41941cd8a6ec634e3aeeaf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.380970 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet df08f0d3af4c4c7e8abce656714b1b7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.381250 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 448a78fb8079482baed5f75cddd746be (DEFAULT_TABLE table=default.Table_74c703e708b34d03891d74e02e877c53 [id=080df455dd864e5382c34baa9e02a2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.385435 25687 tablet.cc:1722] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:58.403738 23752 tablet_service.cc:1467] Processing CreateTablet for tablet a6958c1ad27c4ed4ba95b0553b5e5f3f (DEFAULT_TABLE table=default.Table_c4e6c4c8a4d54368aa8547df0627bfee [id=82a2922537e24f1c91d0b4e12bda24a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.404325 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6958c1ad27c4ed4ba95b0553b5e5f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.404291 25423 raft_consensus.cc:2781] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:58.410823 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb9750041a44138bcf463da241e5e3a (DEFAULT_TABLE table=default.Table_ae314c4445fb416f97e233054cd2cb3a [id=975a8ceabb83442db7f86fc17b87ad84]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.411234 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb9750041a44138bcf463da241e5e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.412292 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet e0bd774640464d2a92b2e71126c83883 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:58.413156 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 11ebacd4c41941cd8a6ec634e3aeeaf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:58.414193 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 9eaf8d65f0b4472eb8ed0ef6fc722577 (DEFAULT_TABLE table=default.Table_4c6989f7e6bb4e99b447a3c1188f5b92 [id=9775ed3c642a4d7e81d5abc48acca163]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.414827 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eaf8d65f0b4472eb8ed0ef6fc722577. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:58.415280 23996 leader_election.cc:341] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:58.425025 24174 tablet_bootstrap.cc:654] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.432433 24053 raft_consensus.cc:3037] T ee022e8e86f8477a8fa1e58d52644b03 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:58.435060 24053 raft_consensus.cc:2454] T ee022e8e86f8477a8fa1e58d52644b03 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:58.436184 23861 leader_election.cc:302] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:16:58.436800 25494 raft_consensus.cc:2781] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:58.436935 25494 raft_consensus.cc:686] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:16:58.437165 25494 consensus_queue.cc:237] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.440460 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 73d7633fd23541f3a553af008dd6a354 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.440467 24051 raft_consensus.cc:2454] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:58.435606 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f83bcf640f465bb95d87a991cc5ff6" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:58.441092 23723 leader_election.cc:302] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:58.441506 24173 tablet_bootstrap.cc:654] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.442274 25531 raft_consensus.cc:2781] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:16:58.442476 25531 raft_consensus.cc:482] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:58.442700 25531 raft_consensus.cc:3037] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:16:58.446794 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 448a78fb8079482baed5f75cddd746be. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:58.443585 23858 leader_election.cc:341] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: ee022e8e86f8477a8fa1e58d52644b03
I20240502 08:16:58.442986 23922 raft_consensus.cc:2454] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
W20240502 08:16:58.459728 23996 consensus_peers.cc:485] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:58.459915 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 43f9164589084269b9ecf3c7b9750e17 (DEFAULT_TABLE table=default.Table_7deeda2c4a754c3d8176faea44b99f1a [id=dc6c1432f15c4d8c8f293470fb9fed72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.459754 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 55fb41b8f8f9487a81e98bae9cddaae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:58.465507 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3449d7d1e8044b6cbf20766a5597e5ab consensus state has no leader
I20240502 08:16:58.469609 23602 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 9761df0bbdee44af90135bffb8b5828f (table default.Table_8ba240c391da43978f93a27dce48e557 [id=1b5536950db244d1978b154da0be33e0]). Expected version 2 got 0
I20240502 08:16:58.470036 23602 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet e0b0d4871dd24607854100a4ef762e25 (table default.Table_5301ffccbac743618a78289c2c1415e1 [id=910c7cae36264e39bbfca0be919538e0]). Expected version 1 got 0
I20240502 08:16:58.470304 23602 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 7ed919909289410e867881baab841f53 (table default.Table_2edad641920b44ce9a4d5b7503f0279a [id=ee3912407a934444ad2c62f7b83d93ac]). Expected version 2 got 0
I20240502 08:16:58.471807 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet e47459b5f5bb4ecb96f76240fefa1f5d (table default.Table_922974ba60ab43e5af71756d9209fd0d [id=51cd52edc7ae46738d9ffa2bd8083fdb]) successfully deleted
I20240502 08:16:58.472749 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 18d284f71937459db0c0676ce4568175 (table default.Table_7af51f5f2feb42ca8dbe067300bbbfd9 [id=7eff9f0e4b304a01b8d8c521bfdd1563]) successfully deleted
I20240502 08:16:58.379657 24180 ts_tablet_manager.cc:1888] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:58.474833 24180 log.cc:1196] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/a4d151770ee44130adfd822518b9b017
I20240502 08:16:58.475548 24180 ts_tablet_manager.cc:1909] T a4d151770ee44130adfd822518b9b017 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:58.477438 23602 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet a2da8b9d3f374d8a812336057c0ce7d2 (table default.Table_3edd27da6927478281e76361718425a3 [id=76b58ce209a24b9a95b405c1c50de3ca]). Expected version 3 got 0
I20240502 08:16:58.477800 23602 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 9761df0bbdee44af90135bffb8b5828f (table default.Table_8ba240c391da43978f93a27dce48e557 [id=1b5536950db244d1978b154da0be33e0]). Expected version 2 got 0
I20240502 08:16:58.478170 23602 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet ec10ff7251a5442a8ecc1881b34db74d (table default.Table_236150d3903d437d95f67b0edec36091 [id=1d7a43116fe74c35b505eccd165f99cd]). Expected version 2 got 0
I20240502 08:16:58.478482 23602 catalog_manager.cc:5484] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67). New cstate: current_term: 1 leader_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:58.479061 23602 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 55f3f09fc1f94c00a47b9087b98e221d (table default.Table_7c74c34063104d54b3ab77c878892069 [id=106dd88de5eb49e89f0042beb7920736]). Expected version 2 got 0
W20240502 08:16:58.483104 23858 leader_election.cc:341] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: ee022e8e86f8477a8fa1e58d52644b03
W20240502 08:16:58.486394 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1424b28b227f42c790d2e85e44f72ce3 consensus state has no leader
I20240502 08:16:58.487128 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d7cff78803d44e5aba3b50cd8ff11e92" } modify_external_catalogs: true
I20240502 08:16:58.487339 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d7cff78803d44e5aba3b50cd8ff11e92" } modify_external_catalogs: true
I20240502 08:16:58.488122 23765 tablet_service.cc:1467] Processing CreateTablet for tablet cc2c7c90bb8649cb8fe79e0f6957db36 (DEFAULT_TABLE table=default.Table_cccac0aeb15d4f568676f29b4d73a49f [id=7fb6d15fd04442b7832e80f1bf45ae11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.488593 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc2c7c90bb8649cb8fe79e0f6957db36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.492266 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4a333c6dd44a13bc267e885bb9e487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.492471 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c860f268d284b0bb8cb05e4aa04baa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.492663 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c3a2a82f004a088c959df50e664473. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.492888 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0000f1984ab64164868de1ada6638007. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.493099 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f335d067384fcf9f968aa7be53ac9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.493256 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 9e13c0b81dc24b8ba1a88b31bf510efd (DEFAULT_TABLE table=default.Table_82f82cd6e702420283c288c10f905d90 [id=f293b22818344aea9eb6232141432183]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.493842 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 448a78fb8079482baed5f75cddd746be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.494597 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0743a0f31b5441728536b818372d239e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.495499 25012 ts_tablet_manager.cc:1888] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:58.495651 25012 log.cc:1196] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/1da01e2c2f054ccfb53383e70e16da84
I20240502 08:16:58.495962 25357 raft_consensus.cc:482] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:58.496593 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f9164589084269b9ecf3c7b9750e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.497654 24175 tablet_bootstrap.cc:492] T 156c7356ece342aea0d8db667ebad89c P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:58.497829 24175 ts_tablet_manager.cc:1367] T 156c7356ece342aea0d8db667ebad89c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.464s user 0.014s sys 0.000s
I20240502 08:16:58.498512 24175 raft_consensus.cc:348] T 156c7356ece342aea0d8db667ebad89c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.498756 24175 raft_consensus.cc:374] T 156c7356ece342aea0d8db667ebad89c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.499233 24175 raft_consensus.cc:729] T 156c7356ece342aea0d8db667ebad89c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.499622 24175 consensus_queue.cc:260] T 156c7356ece342aea0d8db667ebad89c P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.501068 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_02f448a8a9844def8ecda8716fdc352e" } modify_external_catalogs: true
I20240502 08:16:58.501348 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_02f448a8a9844def8ecda8716fdc352e" } modify_external_catalogs: true
I20240502 08:16:58.496084 25357 raft_consensus.cc:504] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.503320 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet 55fb41b8f8f9487a81e98bae9cddaae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:58.503345 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1c914a8e055b422e8029a6a02b4a2925" } modify_external_catalogs: true
I20240502 08:16:58.503511 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1c914a8e055b422e8029a6a02b4a2925" } modify_external_catalogs: true
I20240502 08:16:58.504271 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 43f9164589084269b9ecf3c7b9750e17 (DEFAULT_TABLE table=default.Table_7deeda2c4a754c3d8176faea44b99f1a [id=dc6c1432f15c4d8c8f293470fb9fed72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.504673 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f9164589084269b9ecf3c7b9750e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.505995 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4c264de039c8469284d9921dc3bdf875" } modify_external_catalogs: true
I20240502 08:16:58.506187 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4c264de039c8469284d9921dc3bdf875" } modify_external_catalogs: true
W20240502 08:16:58.507305 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1fc2270a5461448ca0245bd875f4245a consensus state has no leader
I20240502 08:16:58.507989 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_465187f6217440139197568c3a016f33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:58.508648 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_465187f6217440139197568c3a016f33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:58.503845 23858 leader_election.cc:341] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 18d284f71937459db0c0676ce4568175
I20240502 08:16:58.509424 23858 leader_election.cc:302] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:58.509744 25649 raft_consensus.cc:2726] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:58.511933 25357 leader_election.cc:288] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:58.514055 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet a198c33a3e314b5b81a08244862b281a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:58.515875 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_cbac61918e774f599192caeeaef51577"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:58.516618 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cbac61918e774f599192caeeaef51577 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:58.517927 25423 raft_consensus.cc:482] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:58.518059 25423 raft_consensus.cc:3037] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:16:58.518541 23858 leader_election.cc:341] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: edd7244b9f5741a8aa1c6835fe544676
I20240502 08:16:58.519973 25698 tablet.cc:1722] T a2da8b9d3f374d8a812336057c0ce7d2 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:16:58.520694 25012 ts_tablet_manager.cc:1909] T 1da01e2c2f054ccfb53383e70e16da84 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:58.521406 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 953) took 4282 ms. Trace:
I20240502 08:16:58.521484 25012 rpcz_store.cc:276] 0502 08:16:54.238553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.238692 (+ 139us) service_pool.cc:225] Handling call
0502 08:16:58.224206 (+3985514us) ts_tablet_manager.cc:1120] Deleting tablet 1da01e2c2f054ccfb53383e70e16da84
0502 08:16:58.224214 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:58.371427 (+147213us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:58.495388 (+123961us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:58.521388 (+ 26000us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3981980}
I20240502 08:16:58.527124 25531 raft_consensus.cc:504] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:58.529757 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd7244b9f5741a8aa1c6835fe544676" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:58.530053 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1038) took 1147 ms. Trace:
I20240502 08:16:58.530148 24181 rpcz_store.cc:276] 0502 08:16:57.373254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.373306 (+ 52us) service_pool.cc:225] Handling call
0502 08:16:58.454421 (+1081115us) ts_tablet_manager.cc:1120] Deleting tablet 82c49bba3df548558123344a13c946ee
0502 08:16:58.519756 (+ 65335us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:58.520203 (+ 447us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1077428}
I20240502 08:16:58.531033 25531 leader_election.cc:288] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 2 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:58.537753 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f83bcf640f465bb95d87a991cc5ff6" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:58.538048 24053 raft_consensus.cc:3037] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:16:58.538070 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1039) took 1154 ms. Trace:
I20240502 08:16:58.538149 24181 rpcz_store.cc:276] 0502 08:16:57.376499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.376553 (+ 54us) service_pool.cc:225] Handling call
0502 08:16:58.530614 (+1154061us) ts_tablet_manager.cc:1120] Deleting tablet 18d82e75f2624e6782f55987cea223cb
0502 08:16:58.530623 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:58.530700 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1153900}
I20240502 08:16:58.538555 25423 raft_consensus.cc:504] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.540632 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1040) took 1160 ms. Trace:
I20240502 08:16:58.545224 24181 rpcz_store.cc:276] 0502 08:16:57.377766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.377824 (+ 58us) service_pool.cc:225] Handling call
0502 08:16:58.538479 (+1160655us) ts_tablet_manager.cc:1120] Deleting tablet 01143d14aca64adb846c8141ae41e6ac
0502 08:16:58.538486 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:58.538605 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1160511}
W20240502 08:16:58.545243 23993 leader_election.cc:341] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 592eeecbf5514215b6464bd030081e4b
I20240502 08:16:58.547305 24181 tablet_replica.cc:330] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:58.547528 24181 raft_consensus.cc:2229] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:58.547591 25423 leader_election.cc:288] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:58.551437 24181 pending_rounds.cc:62] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5: Trying to abort 1 pending ops.
I20240502 08:16:58.551517 24181 pending_rounds.cc:68] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156501132173312 op_type: NO_OP noop_request { }
I20240502 08:16:58.552894 24181 raft_consensus.cc:2866] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:58.553022 24181 raft_consensus.cc:2258] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:58.553997 24181 ts_tablet_manager.cc:1875] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:58.557668 23597 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 83791.1us
W20240502 08:16:58.558756 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 82c49bba3df548558123344a13c946ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82c49bba3df548558123344a13c946ee already in progress: creating tablet
W20240502 08:16:58.559049 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 18d82e75f2624e6782f55987cea223cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18d82e75f2624e6782f55987cea223cb already in progress: creating tablet
W20240502 08:16:58.559341 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 01143d14aca64adb846c8141ae41e6ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01143d14aca64adb846c8141ae41e6ac already in progress: creating tablet
W20240502 08:16:58.559597 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet b054783710824dbc8f7bc3ea38f605b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b054783710824dbc8f7bc3ea38f605b0 already in progress: creating tablet
W20240502 08:16:58.559827 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet af6efeebaebc495aba017a922e243c39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af6efeebaebc495aba017a922e243c39 already in progress: creating tablet
W20240502 08:16:58.560102 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet c84b27ed8cf7429ca720e4cb8d396342 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c84b27ed8cf7429ca720e4cb8d396342 already in progress: creating tablet
W20240502 08:16:58.560345 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 9a9676602d1c4b37b3f4126a29f5456f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a9676602d1c4b37b3f4126a29f5456f already in progress: creating tablet
I20240502 08:16:58.562120 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 5e02e460f6ea4b269c178e80b4e0a3f4 (DEFAULT_TABLE table=default.Table_456a3cba3b8e4f09b405cd14e3785a23 [id=25b8092a8da04b04b8e7e7f2ae2f94d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.562240 25531 raft_consensus.cc:482] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:58.562366 25531 raft_consensus.cc:504] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:58.562745 24053 raft_consensus.cc:2454] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 2.
I20240502 08:16:58.562987 25531 leader_election.cc:288] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:58.530259 23861 leader_election.cc:341] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:58.563117 23861 leader_election.cc:302] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:58.563395 23723 leader_election.cc:302] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:58.563637 25357 raft_consensus.cc:2726] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:58.563728 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e02e460f6ea4b269c178e80b4e0a3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.563824 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 9eaf8d65f0b4472eb8ed0ef6fc722577 (DEFAULT_TABLE table=default.Table_4c6989f7e6bb4e99b447a3c1188f5b92 [id=9775ed3c642a4d7e81d5abc48acca163]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.563849 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592eeecbf5514215b6464bd030081e4b" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:58.564028 23922 raft_consensus.cc:3037] T 592eeecbf5514215b6464bd030081e4b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:58.566157 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eaf8d65f0b4472eb8ed0ef6fc722577. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.566370 23922 raft_consensus.cc:2454] T 592eeecbf5514215b6464bd030081e4b P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:58.566915 24024 tablet_service.cc:1467] Processing CreateTablet for tablet a6958c1ad27c4ed4ba95b0553b5e5f3f (DEFAULT_TABLE table=default.Table_c4e6c4c8a4d54368aa8547df0627bfee [id=82a2922537e24f1c91d0b4e12bda24a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.555001 24175 ts_tablet_manager.cc:1398] T 156c7356ece342aea0d8db667ebad89c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.057s user 0.002s sys 0.000s
I20240502 08:16:58.567258 25531 raft_consensus.cc:2781] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:16:58.567391 25531 raft_consensus.cc:686] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 2 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:58.567673 25531 consensus_queue.cc:237] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:58.567341 24175 tablet_bootstrap.cc:492] T 4c395a29c7a1485eab39ca107dd91fca P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:58.567713 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f83bcf640f465bb95d87a991cc5ff6" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:58.568135 23917 raft_consensus.cc:3037] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:16:58.569147 24175 tablet_bootstrap.cc:654] T 4c395a29c7a1485eab39ca107dd91fca P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.562943 23904 tablet_service.cc:1467] Processing CreateTablet for tablet a6958c1ad27c4ed4ba95b0553b5e5f3f (DEFAULT_TABLE table=default.Table_c4e6c4c8a4d54368aa8547df0627bfee [id=82a2922537e24f1c91d0b4e12bda24a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.570080 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6958c1ad27c4ed4ba95b0553b5e5f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.570950 23996 leader_election.cc:302] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:58.573122 25423 raft_consensus.cc:2781] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:58.573320 25423 raft_consensus.cc:686] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:58.521816 25012 tablet_replica.cc:330] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:58.574661 25012 raft_consensus.cc:2229] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:58.574872 25012 raft_consensus.cc:2258] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:58.575407 25012 ts_tablet_manager.cc:1875] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:58.577068 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 73d7633fd23541f3a553af008dd6a354 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:58.577613 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b8f21c64e343d88d84e9acd0bc69b4" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:58.578120 23723 leader_election.cc:341] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:58.578626 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 5e02e460f6ea4b269c178e80b4e0a3f4 (DEFAULT_TABLE table=default.Table_456a3cba3b8e4f09b405cd14e3785a23 [id=25b8092a8da04b04b8e7e7f2ae2f94d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.578572 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b8f21c64e343d88d84e9acd0bc69b4" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:58.579041 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e02e460f6ea4b269c178e80b4e0a3f4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:58.579241 23723 leader_election.cc:341] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:58.579344 23723 leader_election.cc:302] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:58.579691 25158 raft_consensus.cc:2726] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:58.590662 25423 consensus_queue.cc:237] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.593372 25691 tablet.cc:1722] T 4c395a29c7a1485eab39ca107dd91fca P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240502 08:16:58.595764 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6958c1ad27c4ed4ba95b0553b5e5f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.597136 24174 tablet_bootstrap.cc:492] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:58.598008 23917 raft_consensus.cc:2454] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 2.
I20240502 08:16:58.607673 24174 ts_tablet_manager.cc:1367] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.227s user 0.004s sys 0.001s
I20240502 08:16:58.610903 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 1da01e2c2f054ccfb53383e70e16da84 (table default.Table_3ef466d7a87644e99070934d65e5c851 [id=f596bcb8d6f04ac680be38e763a33750]) successfully deleted
I20240502 08:16:58.610929 25687 tablet.cc:1722] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240502 08:16:58.611968 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 9e13c0b81dc24b8ba1a88b31bf510efd (DEFAULT_TABLE table=default.Table_82f82cd6e702420283c288c10f905d90 [id=f293b22818344aea9eb6232141432183]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.612303 24032 tablet_service.cc:1467] Processing CreateTablet for tablet cc2c7c90bb8649cb8fe79e0f6957db36 (DEFAULT_TABLE table=default.Table_cccac0aeb15d4f568676f29b4d73a49f [id=7fb6d15fd04442b7832e80f1bf45ae11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.612529 25413 consensus_queue.cc:1035] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
I20240502 08:16:58.606400 23768 tablet_service.cc:1467] Processing CreateTablet for tablet dcc025d20e93460e882f666d71d6caa4 (DEFAULT_TABLE table=default.Table_d5691c9d792d49038cb518e00d54b2ac [id=53a60582a1004422a917a692217129c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.613366 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e13c0b81dc24b8ba1a88b31bf510efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.618046 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc2c7c90bb8649cb8fe79e0f6957db36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.623474 23786 raft_consensus.cc:1261] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Refusing update from remote peer 5b89a1f0485a44b7ac1710e2369a809b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:58.623848 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb9750041a44138bcf463da241e5e3a (DEFAULT_TABLE table=default.Table_ae314c4445fb416f97e233054cd2cb3a [id=975a8ceabb83442db7f86fc17b87ad84]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.624944 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb9750041a44138bcf463da241e5e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.625320 23904 tablet_service.cc:1467] Processing CreateTablet for tablet cc2c7c90bb8649cb8fe79e0f6957db36 (DEFAULT_TABLE table=default.Table_cccac0aeb15d4f568676f29b4d73a49f [id=7fb6d15fd04442b7832e80f1bf45ae11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.626267 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc2c7c90bb8649cb8fe79e0f6957db36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.608306 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1129) took 1009 ms. Trace:
I20240502 08:16:58.636812 24175 tablet_bootstrap.cc:492] T 4c395a29c7a1485eab39ca107dd91fca P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:58.637035 24175 ts_tablet_manager.cc:1367] T 4c395a29c7a1485eab39ca107dd91fca P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.000s
I20240502 08:16:58.637737 24175 raft_consensus.cc:348] T 4c395a29c7a1485eab39ca107dd91fca P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.638057 24175 raft_consensus.cc:374] T 4c395a29c7a1485eab39ca107dd91fca P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.638159 24175 raft_consensus.cc:729] T 4c395a29c7a1485eab39ca107dd91fca P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.638413 24175 consensus_queue.cc:260] T 4c395a29c7a1485eab39ca107dd91fca P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.639029 24175 ts_tablet_manager.cc:1398] T 4c395a29c7a1485eab39ca107dd91fca P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240502 08:16:58.638427 23434 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 139832us
I20240502 08:16:58.640445 24180 rpcz_store.cc:276] 0502 08:16:57.598334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.598766 (+ 432us) service_pool.cc:225] Handling call
0502 08:16:58.274561 (+675795us) ts_tablet_manager.cc:1120] Deleting tablet a4d151770ee44130adfd822518b9b017
0502 08:16:58.274569 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:58.319092 (+ 44523us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:58.379496 (+ 60404us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:58.608270 (+228774us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":675598}
I20240502 08:16:58.604421 25012 ts_tablet_manager.cc:1888] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:58.641479 25012 log.cc:1196] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/577b149f52e44d5d9a9f6b9d85a6864d
I20240502 08:16:58.614271 23897 tablet_service.cc:1467] Processing CreateTablet for tablet 9e13c0b81dc24b8ba1a88b31bf510efd (DEFAULT_TABLE table=default.Table_82f82cd6e702420283c288c10f905d90 [id=f293b22818344aea9eb6232141432183]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.644743 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet a4d151770ee44130adfd822518b9b017 (table default.Table_fb39fa7949f3412c83974132184b5d9f [id=8b62b02f324d4b6cbc7095de109955b5]) successfully deleted
I20240502 08:16:58.644893 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e13c0b81dc24b8ba1a88b31bf510efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.601938 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e13c0b81dc24b8ba1a88b31bf510efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.643607 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 514fcfdc980f4c91a6415daea767d2ca (DEFAULT_TABLE table=default.Table_22d85074cecf41dbbe8ca2743891e567 [id=51321444e3094fef9ce7772e952cb9e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.645192 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc025d20e93460e882f666d71d6caa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.646026 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514fcfdc980f4c91a6415daea767d2ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.648998 24031 tablet_service.cc:1467] Processing CreateTablet for tablet dcc025d20e93460e882f666d71d6caa4 (DEFAULT_TABLE table=default.Table_d5691c9d792d49038cb518e00d54b2ac [id=53a60582a1004422a917a692217129c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.649617 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc025d20e93460e882f666d71d6caa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.614511 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 9eaf8d65f0b4472eb8ed0ef6fc722577 (DEFAULT_TABLE table=default.Table_4c6989f7e6bb4e99b447a3c1188f5b92 [id=9775ed3c642a4d7e81d5abc48acca163]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:58.617805 23861 consensus_peers.cc:485] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:58.624181 25649 consensus_queue.cc:1035] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:58.658854 25012 ts_tablet_manager.cc:1909] T 577b149f52e44d5d9a9f6b9d85a6864d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:58.662649 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9522d368c4c84f25bce3eb822a9bcad8" } modify_external_catalogs: true
I20240502 08:16:58.662849 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9522d368c4c84f25bce3eb822a9bcad8" } modify_external_catalogs: true
I20240502 08:16:58.676306 25034 ts_manager.cc:194] Registered new tserver with Master: 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:58.679087 25030 ts_manager.cc:194] Registered new tserver with Master: a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:16:58.688436 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 954) took 4251 ms. Trace:
I20240502 08:16:58.688863 25012 rpcz_store.cc:276] 0502 08:16:54.437264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.437723 (+ 459us) service_pool.cc:225] Handling call
0502 08:16:58.521768 (+4084045us) ts_tablet_manager.cc:1120] Deleting tablet 577b149f52e44d5d9a9f6b9d85a6864d
0502 08:16:58.521778 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:58.601690 (+ 79912us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:58.604346 (+ 2656us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:58.688415 (+ 84069us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4083803}
I20240502 08:16:58.659427 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eaf8d65f0b4472eb8ed0ef6fc722577. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.685655 24175 tablet_bootstrap.cc:492] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:58.613865 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb9750041a44138bcf463da241e5e3a (DEFAULT_TABLE table=default.Table_ae314c4445fb416f97e233054cd2cb3a [id=975a8ceabb83442db7f86fc17b87ad84]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.653054 24180 tablet_replica.cc:330] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:58.696952 24180 raft_consensus.cc:2229] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:58.697142 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb9750041a44138bcf463da241e5e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.697196 24180 raft_consensus.cc:2258] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:58.697715 24180 ts_tablet_manager.cc:1875] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:58.698056 24175 tablet_bootstrap.cc:654] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.699020 25012 tablet_replica.cc:330] T 78ddfc7b74644a0bbf8222d5f90e6658 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:58.699615 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 577b149f52e44d5d9a9f6b9d85a6864d (table default.Table_1600e0e09856405ea0d90ac82aa47fda [id=89eea7c9830f47d2bd1c95044d5af4d6]) successfully deleted
I20240502 08:16:58.700157 25012 raft_consensus.cc:2229] T 78ddfc7b74644a0bbf8222d5f90e6658 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:58.701328 25012 raft_consensus.cc:2258] T 78ddfc7b74644a0bbf8222d5f90e6658 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:58.646098 24174 raft_consensus.cc:348] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.673496 23752 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3e66ef4a914f009cca6d418f825d4e (DEFAULT_TABLE table=default.Table_1345a4fd2d3e46deaecb136b548f5b3d [id=9da81562ad6b4ebdb75b4d4c5f2685c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.701680 24174 raft_consensus.cc:374] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.701774 24174 raft_consensus.cc:729] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.702095 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3e66ef4a914f009cca6d418f825d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.709030 25012 ts_tablet_manager.cc:1875] T 78ddfc7b74644a0bbf8222d5f90e6658 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:58.701001 24175 tablet_bootstrap.cc:492] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:58.711035 24175 ts_tablet_manager.cc:1367] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240502 08:16:58.711814 24175 raft_consensus.cc:348] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.715476 24181 ts_tablet_manager.cc:1888] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:58.717077 24175 raft_consensus.cc:374] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.717283 24175 raft_consensus.cc:729] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.717382 24181 log.cc:1196] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/090bec30f5664421b9bfe3f2810afd14
I20240502 08:16:58.717790 24181 ts_tablet_manager.cc:1909] T 090bec30f5664421b9bfe3f2810afd14 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:58.718008 25698 tablet.cc:1722] T e0b0d4871dd24607854100a4ef762e25 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:58.701984 24174 consensus_queue.cc:260] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.719473 24174 ts_tablet_manager.cc:1398] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.091s user 0.002s sys 0.000s
I20240502 08:16:58.720355 24175 consensus_queue.cc:260] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.727962 23890 tablet_service.cc:1467] Processing CreateTablet for tablet dcc025d20e93460e882f666d71d6caa4 (DEFAULT_TABLE table=default.Table_d5691c9d792d49038cb518e00d54b2ac [id=53a60582a1004422a917a692217129c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.728456 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc025d20e93460e882f666d71d6caa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.736901 24173 tablet_bootstrap.cc:492] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:58.738577 24175 ts_tablet_manager.cc:1398] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20240502 08:16:58.740514 25423 raft_consensus.cc:482] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:58.740792 24175 tablet_bootstrap.cc:492] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:58.745404 24174 tablet_bootstrap.cc:492] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:58.747061 24174 tablet_bootstrap.cc:654] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.749214 25691 tablet.cc:1722] T e0b0d4871dd24607854100a4ef762e25 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:16:58.749531 24175 tablet_bootstrap.cc:654] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.753830 23606 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 4c395a29c7a1485eab39ca107dd91fca (table default.Table_1159b00d82f4483c8209c879e2d3c06a [id=272acbb9a6ca4f2dbcbfea6aacbe9cfc]). Expected version 4 got 0
W20240502 08:16:58.755509 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3777d2e9d4464552ad1c4c8c0d3e9888 consensus state has no leader
I20240502 08:16:58.758905 25664 raft_consensus.cc:482] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:58.761842 25012 ts_tablet_manager.cc:1888] T 78ddfc7b74644a0bbf8222d5f90e6658 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:58.762435 25705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f87df6a45ad94428a9aa6b2796a0fd25)
I20240502 08:16:58.762691 25012 log.cc:1196] T 78ddfc7b74644a0bbf8222d5f90e6658 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/78ddfc7b74644a0bbf8222d5f90e6658
I20240502 08:16:58.762636 25705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } }
I20240502 08:16:58.763815 25705 leader_election.cc:288] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcc8a323e6b9429eaaf4e0c1698ad077 (127.22.225.126:43265), f87df6a45ad94428a9aa6b2796a0fd25 (127.22.225.124:41745)
I20240502 08:16:58.763935 24434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1428 } } ignore_live_leader: false dest_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" is_pre_election: true
I20240502 08:16:58.764039 25012 ts_tablet_manager.cc:1909] T 78ddfc7b74644a0bbf8222d5f90e6658 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:58.764396 23620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1428 } } ignore_live_leader: false dest_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" is_pre_election: true
I20240502 08:16:58.769932 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3e66ef4a914f009cca6d418f825d4e (DEFAULT_TABLE table=default.Table_1345a4fd2d3e46deaecb136b548f5b3d [id=9da81562ad6b4ebdb75b4d4c5f2685c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.770236 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 888d258f63984f728f03c9825f175559 (DEFAULT_TABLE table=default.Table_44455df150c841b4b666c5279b4fe64d [id=3252562061db4482ae7175a2cf292d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.770646 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3e66ef4a914f009cca6d418f825d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.771024 24175 tablet_bootstrap.cc:492] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:58.771445 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888d258f63984f728f03c9825f175559. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.771624 24175 ts_tablet_manager.cc:1367] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20240502 08:16:58.772244 23752 tablet_service.cc:1467] Processing CreateTablet for tablet 24ab6b35452d4fb4b4e00adc6d6fb3e2 (DEFAULT_TABLE table=default.Table_b20da99d7e65450cb4f49c3397e1e54f [id=1fe164e6114b4ced8d3882761532a2a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.772701 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24ab6b35452d4fb4b4e00adc6d6fb3e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.773900 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 0001b9d68e0a4b21a9bfd584dcdf99ea (DEFAULT_TABLE table=default.Table_c15d8047d6fd438dbdb3f1961b549f4c [id=a5da193c209e46aeb9f2dfbba13fb9c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.774212 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0001b9d68e0a4b21a9bfd584dcdf99ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.775691 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 888d258f63984f728f03c9825f175559 (DEFAULT_TABLE table=default.Table_44455df150c841b4b666c5279b4fe64d [id=3252562061db4482ae7175a2cf292d25]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:58.776078 23993 consensus_peers.cc:485] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89f57e3a39d54b7ab372d4b145b7a696. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:58.776268 24026 tablet_service.cc:1514] Processing DeleteTablet for tablet 592eeecbf5514215b6464bd030081e4b with delete_type TABLET_DATA_DELETED (replaced by tablet 28a947b7e7484120b3f4069f6eef5520) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.776316 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 514fcfdc980f4c91a6415daea767d2ca (DEFAULT_TABLE table=default.Table_22d85074cecf41dbbe8ca2743891e567 [id=51321444e3094fef9ce7772e952cb9e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.776664 24173 ts_tablet_manager.cc:1367] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.397s user 0.010s sys 0.000s
I20240502 08:16:58.776713 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 24ab6b35452d4fb4b4e00adc6d6fb3e2 (DEFAULT_TABLE table=default.Table_b20da99d7e65450cb4f49c3397e1e54f [id=1fe164e6114b4ced8d3882761532a2a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.777354 25423 raft_consensus.cc:504] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.777292 25664 raft_consensus.cc:504] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.777552 24173 raft_consensus.cc:348] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.777801 24173 raft_consensus.cc:374] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.777932 24173 raft_consensus.cc:729] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.778136 25423 leader_election.cc:288] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:58.778280 24173 consensus_queue.cc:260] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.778524 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888d258f63984f728f03c9825f175559. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.778141 25664 leader_election.cc:288] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:58.779320 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24ab6b35452d4fb4b4e00adc6d6fb3e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.780170 24922 leader_election.cc:302] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84f3c07a31414c4f8fb61c44dc9f16e1; no voters: bcc8a323e6b9429eaaf4e0c1698ad077, f87df6a45ad94428a9aa6b2796a0fd25
W20240502 08:16:58.782439 23993 leader_election.cc:341] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: a4d151770ee44130adfd822518b9b017
I20240502 08:16:58.785086 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514fcfdc980f4c91a6415daea767d2ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.785610 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1064) took 1042 ms. Trace:
I20240502 08:16:58.785800 24181 rpcz_store.cc:276] 0502 08:16:57.675612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.675651 (+ 39us) service_pool.cc:225] Handling call
0502 08:16:58.546126 (+870475us) ts_tablet_manager.cc:1120] Deleting tablet 090bec30f5664421b9bfe3f2810afd14
0502 08:16:58.546132 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:58.572676 (+ 26544us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:58.715365 (+142689us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:58.718508 (+ 3143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":870323}
I20240502 08:16:58.786314 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 090bec30f5664421b9bfe3f2810afd14 (table default.Table_70034c04442f49f6a42cb66dccfd1cc1 [id=74be9e9508024ebeb4623c488e21faf9]) successfully deleted
I20240502 08:16:58.786396 24181 tablet_replica.cc:330] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:58.786595 24181 raft_consensus.cc:2229] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:58.786782 24181 raft_consensus.cc:2258] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:58.787452 24181 ts_tablet_manager.cc:1875] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:58.794059 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e8982fe35f47cda376d9646b52e961" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:58.794409 23786 raft_consensus.cc:2454] T b0e8982fe35f47cda376d9646b52e961 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:58.794859 23993 leader_election.cc:302] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:58.795243 25405 raft_consensus.cc:2781] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:58.795339 25405 raft_consensus.cc:482] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:58.795465 25405 raft_consensus.cc:3037] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:58.797940 25405 raft_consensus.cc:504] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.799448 25405 leader_election.cc:288] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:58.799923 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e8982fe35f47cda376d9646b52e961" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:58.800122 23786 raft_consensus.cc:3037] T b0e8982fe35f47cda376d9646b52e961 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:58.801980 24173 ts_tablet_manager.cc:1398] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.025s user 0.000s sys 0.002s
I20240502 08:16:58.804874 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c41a9c564219469d9d8b285cd4c64711" } new_table_name: "default.Table_9a2fa4232ad64879b69ae596cdfd2bed" modify_external_catalogs: true
I20240502 08:16:58.806191 24173 tablet_bootstrap.cc:492] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:58.808763 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 0001b9d68e0a4b21a9bfd584dcdf99ea (DEFAULT_TABLE table=default.Table_c15d8047d6fd438dbdb3f1961b549f4c [id=a5da193c209e46aeb9f2dfbba13fb9c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.808984 23602 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 156c7356ece342aea0d8db667ebad89c (table default.Table_d8c6955b89724769bbfd81db2b5be455 [id=15b52f0688c24df2b6f3fda3732ceeb4]). Expected version 1 got 0
I20240502 08:16:58.809191 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0001b9d68e0a4b21a9bfd584dcdf99ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.809245 23602 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 4c395a29c7a1485eab39ca107dd91fca (table default.Table_1159b00d82f4483c8209c879e2d3c06a [id=272acbb9a6ca4f2dbcbfea6aacbe9cfc]). Expected version 4 got 0
I20240502 08:16:58.811650 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 06e99a99a6c44b43af2371c81eeb4324 (DEFAULT_TABLE table=default.Table_7dd4773d221543cb990a5a6067ef2db1 [id=a11567f3525d4adcbdc68c8c5234200e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.812044 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e99a99a6c44b43af2371c81eeb4324. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.778793 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e8982fe35f47cda376d9646b52e961" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:58.813375 23996 leader_election.cc:341] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:58.778844 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d151770ee44130adfd822518b9b017" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:58.814244 23996 leader_election.cc:341] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:58.779448 23922 raft_consensus.cc:3037] T 4c395a29c7a1485eab39ca107dd91fca P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:58.814342 23996 leader_election.cc:302] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:58.815914 24174 tablet_bootstrap.cc:492] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:58.816066 24174 ts_tablet_manager.cc:1367] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.000s
I20240502 08:16:58.816859 24174 raft_consensus.cc:348] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.817071 24174 raft_consensus.cc:374] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.817119 24174 raft_consensus.cc:729] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.817283 24174 consensus_queue.cc:260] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:16:58.817520 23596 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 3777d2e9d4464552ad1c4c8c0d3e9888 (table default.Table_660621104fba4545bc2b8c30dc7d035a [id=78b34e35d763495aadf6a9eb242741c9], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3777d2e9d4464552ad1c4c8c0d3e9888 consensus state has no leader
I20240502 08:16:58.819468 25664 raft_consensus.cc:2726] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:58.819504 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 122c13f281f4488ea6c0e87084431aec (DEFAULT_TABLE table=default.Table_655a2929f122445780cd4db021cb35f4 [id=04a251d791b44ebc8c22ce2a2925f8c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.819577 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 122c13f281f4488ea6c0e87084431aec (DEFAULT_TABLE table=default.Table_655a2929f122445780cd4db021cb35f4 [id=04a251d791b44ebc8c22ce2a2925f8c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.819905 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122c13f281f4488ea6c0e87084431aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.820079 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 06e99a99a6c44b43af2371c81eeb4324 (DEFAULT_TABLE table=default.Table_7dd4773d221543cb990a5a6067ef2db1 [id=a11567f3525d4adcbdc68c8c5234200e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.821194 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122c13f281f4488ea6c0e87084431aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.779887 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 24ab6b35452d4fb4b4e00adc6d6fb3e2 (DEFAULT_TABLE table=default.Table_b20da99d7e65450cb4f49c3397e1e54f [id=1fe164e6114b4ced8d3882761532a2a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.799268 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e8982fe35f47cda376d9646b52e961" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:58.823019 23996 leader_election.cc:341] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:58.823388 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24ab6b35452d4fb4b4e00adc6d6fb3e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.827371 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 06e99a99a6c44b43af2371c81eeb4324 (DEFAULT_TABLE table=default.Table_7dd4773d221543cb990a5a6067ef2db1 [id=a11567f3525d4adcbdc68c8c5234200e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.827741 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e99a99a6c44b43af2371c81eeb4324. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.828236 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 6746c25111d04f20a6f30ce0838d71f5
I20240502 08:16:58.828537 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 001e3f4f72f64dc69dbd521724912f07
I20240502 08:16:58.829206 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e99a99a6c44b43af2371c81eeb4324. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.829362 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 001e3f4f72f64dc69dbd521724912f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.829428 24026 tablet_service.cc:1514] Processing DeleteTablet for tablet 6746c25111d04f20a6f30ce0838d71f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.830410 23897 tablet_service.cc:1467] Processing CreateTablet for tablet 0001b9d68e0a4b21a9bfd584dcdf99ea (DEFAULT_TABLE table=default.Table_c15d8047d6fd438dbdb3f1961b549f4c [id=a5da193c209e46aeb9f2dfbba13fb9c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.831353 25705 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:58.801806 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 888d258f63984f728f03c9825f175559 (DEFAULT_TABLE table=default.Table_44455df150c841b4b666c5279b4fe64d [id=3252562061db4482ae7175a2cf292d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.832346 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet 001e3f4f72f64dc69dbd521724912f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:58.832813 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 6746c25111d04f20a6f30ce0838d71f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:58.832959 23903 tablet_service.cc:1514] Processing DeleteTablet for tablet 001e3f4f72f64dc69dbd521724912f07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:58.835160 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 122c13f281f4488ea6c0e87084431aec (DEFAULT_TABLE table=default.Table_655a2929f122445780cd4db021cb35f4 [id=04a251d791b44ebc8c22ce2a2925f8c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.835525 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 7ddf9a1299944755bee1cf0178c6edf8 (DEFAULT_TABLE table=default.Table_b638d47286da4c038bf21e1fda876cff [id=fb2db1142b024839be0663235b8272be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.835969 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ddf9a1299944755bee1cf0178c6edf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.838194 24175 raft_consensus.cc:348] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.838451 24175 raft_consensus.cc:374] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.838538 24175 raft_consensus.cc:729] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.838757 24175 consensus_queue.cc:260] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.839386 24175 ts_tablet_manager.cc:1398] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:58.839679 24175 tablet_bootstrap.cc:492] T 55f3f09fc1f94c00a47b9087b98e221d P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:58.776268 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3e66ef4a914f009cca6d418f825d4e (DEFAULT_TABLE table=default.Table_1345a4fd2d3e46deaecb136b548f5b3d [id=9da81562ad6b4ebdb75b4d4c5f2685c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.840420 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3e66ef4a914f009cca6d418f825d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.842350 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 7ddf9a1299944755bee1cf0178c6edf8 (DEFAULT_TABLE table=default.Table_b638d47286da4c038bf21e1fda876cff [id=fb2db1142b024839be0663235b8272be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.831817 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 514fcfdc980f4c91a6415daea767d2ca (DEFAULT_TABLE table=default.Table_22d85074cecf41dbbe8ca2743891e567 [id=51321444e3094fef9ce7772e952cb9e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.845121 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122c13f281f4488ea6c0e87084431aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.845340 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ddf9a1299944755bee1cf0178c6edf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.847254 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0001b9d68e0a4b21a9bfd584dcdf99ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.848505 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888d258f63984f728f03c9825f175559. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.776296 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet a4d151770ee44130adfd822518b9b017 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.848851 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet 6746c25111d04f20a6f30ce0838d71f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:58.850255 23786 raft_consensus.cc:2454] T b0e8982fe35f47cda376d9646b52e961 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:58.852031 23993 leader_election.cc:302] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:58.852478 25405 raft_consensus.cc:2781] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:58.852607 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514fcfdc980f4c91a6415daea767d2ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.853138 25405 raft_consensus.cc:686] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:58.853439 25405 consensus_queue.cc:237] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.854818 24175 tablet_bootstrap.cc:654] T 55f3f09fc1f94c00a47b9087b98e221d P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.857481 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e2792c45255c42adaeefb7aef4e25715
I20240502 08:16:58.858280 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d1a3f21401914ea6b4e2309ee2f71089"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:58.857481 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 990d0cc863ee4f49a2fc2fffb217c4d7
I20240502 08:16:58.859704 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_effc9967e9074f64b1fa075d85c98470"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:58.858279 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_581ed0c699e94b5bb670d72c63f40197"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:58.861076 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1a3f21401914ea6b4e2309ee2f71089 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:58.861173 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_581ed0c699e94b5bb670d72c63f40197 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:58.862433 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_effc9967e9074f64b1fa075d85c98470 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:58.862555 24174 ts_tablet_manager.cc:1398] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20240502 08:16:58.862615 23610 catalog_manager.cc:5484] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67). New cstate: current_term: 1 leader_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:58.863168 23610 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 74b4c563d6bd4a44bb3bab58a3fcdc07 (table default.Table_2ee0f7ddd41e41ea84e25359cd3ed0a4 [id=a9c009feee0e486794a8500121fe5487]). Expected version 2 got 0
I20240502 08:16:58.864898 23610 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 55f3f09fc1f94c00a47b9087b98e221d (table default.Table_7c74c34063104d54b3ab77c878892069 [id=106dd88de5eb49e89f0042beb7920736]). Expected version 2 got 0
I20240502 08:16:58.867350 24174 tablet_bootstrap.cc:492] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:58.867705 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet 990d0cc863ee4f49a2fc2fffb217c4d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:58.867967 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet 58f538766e934282932fd4ab41b49831 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:58.866901 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet e2792c45255c42adaeefb7aef4e25715 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:58.868402 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 3ad53633ce57478194437f95135d81de
I20240502 08:16:58.868991 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ad53633ce57478194437f95135d81de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:58.873181 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ad53633ce57478194437f95135d81de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.873288 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 7ddf9a1299944755bee1cf0178c6edf8 (DEFAULT_TABLE table=default.Table_b638d47286da4c038bf21e1fda876cff [id=fb2db1142b024839be0663235b8272be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:58.873723 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 990d0cc863ee4f49a2fc2fffb217c4d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.873790 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ddf9a1299944755bee1cf0178c6edf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:58.877813 23903 tablet_service.cc:1514] Processing DeleteTablet for tablet 990d0cc863ee4f49a2fc2fffb217c4d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:58.877892 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet e2792c45255c42adaeefb7aef4e25715 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:58.879031 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ad53633ce57478194437f95135d81de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:58.882692 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet e2792c45255c42adaeefb7aef4e25715 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.764798 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 955) took 4187 ms. Trace:
I20240502 08:16:58.883985 25012 rpcz_store.cc:276] 0502 08:16:54.577319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.577378 (+ 59us) service_pool.cc:225] Handling call
0502 08:16:58.698982 (+4121604us) ts_tablet_manager.cc:1120] Deleting tablet 78ddfc7b74644a0bbf8222d5f90e6658
0502 08:16:58.698988 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:58.746547 (+ 47559us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:58.761705 (+ 15158us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:58.764778 (+ 3073us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4121331}
I20240502 08:16:58.884621 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 958) took 4150 ms. Trace:
I20240502 08:16:58.884755 25012 rpcz_store.cc:276] 0502 08:16:54.733803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.734042 (+ 239us) service_pool.cc:225] Handling call
0502 08:16:58.884487 (+4150445us) ts_tablet_manager.cc:1120] Deleting tablet bb9b75d6be924d0291f94fc4a439dea6
0502 08:16:58.884500 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:58.884604 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4150261}
I20240502 08:16:58.885084 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 78ddfc7b74644a0bbf8222d5f90e6658 (table default.Table_0b7c860d457f421294ca1d37248e40b5 [id=c06501dd41a640279c769c9ab3926ad5]) successfully deleted
I20240502 08:16:58.885242 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 959) took 4151 ms. Trace:
I20240502 08:16:58.885380 25012 rpcz_store.cc:276] 0502 08:16:54.733987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.734267 (+ 280us) service_pool.cc:225] Handling call
0502 08:16:58.885099 (+4150832us) ts_tablet_manager.cc:1120] Deleting tablet 2c28c47b4f814573971227b2208d1f83
0502 08:16:58.885109 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:58.885224 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4150691}
W20240502 08:16:58.885509 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet bb9b75d6be924d0291f94fc4a439dea6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb9b75d6be924d0291f94fc4a439dea6 already in progress: creating tablet
I20240502 08:16:58.885841 25012 tablet_replica.cc:330] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
W20240502 08:16:58.885989 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 2c28c47b4f814573971227b2208d1f83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c28c47b4f814573971227b2208d1f83 already in progress: creating tablet
I20240502 08:16:58.886123 25012 raft_consensus.cc:2229] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:58.886401 25012 raft_consensus.cc:2258] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:58.886999 25012 ts_tablet_manager.cc:1875] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:58.889458 24174 tablet_bootstrap.cc:654] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.900177 24180 ts_tablet_manager.cc:1888] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:58.900391 24180 log.cc:1196] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/92adebc0c69e4a52b975253656132495
I20240502 08:16:58.900650 24174 tablet_bootstrap.cc:492] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:58.901458 24174 ts_tablet_manager.cc:1367] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20240502 08:16:58.902200 24174 raft_consensus.cc:348] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.902402 24174 raft_consensus.cc:374] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.902462 24174 raft_consensus.cc:729] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.902654 24174 consensus_queue.cc:260] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.909452 25698 tablet.cc:1722] T 4c395a29c7a1485eab39ca107dd91fca P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240502 08:16:58.910205 24174 ts_tablet_manager.cc:1398] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 08:16:58.910504 24174 tablet_bootstrap.cc:492] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:58.915973 24174 tablet_bootstrap.cc:654] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.933095 25651 catalog_manager.cc:1259] Loaded cluster ID: d0e52a60b5c54ea9a5f0ea385bd7e9c9
I20240502 08:16:58.900776 24180 ts_tablet_manager.cc:1909] T 92adebc0c69e4a52b975253656132495 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:58.934832 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1128) took 1336 ms. Trace:
I20240502 08:16:58.937198 25651 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1: loading cluster ID for follower catalog manager: success
I20240502 08:16:58.935155 24174 tablet_bootstrap.cc:492] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:58.937754 24174 ts_tablet_manager.cc:1367] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20240502 08:16:58.938455 24174 raft_consensus.cc:348] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.938635 24174 raft_consensus.cc:374] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.938688 24174 raft_consensus.cc:729] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.938889 24174 consensus_queue.cc:260] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.939497 24174 ts_tablet_manager.cc:1398] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:58.939765 24180 rpcz_store.cc:276] 0502 08:16:57.598263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.598992 (+ 729us) service_pool.cc:225] Handling call
0502 08:16:58.652999 (+1054007us) ts_tablet_manager.cc:1120] Deleting tablet 92adebc0c69e4a52b975253656132495
0502 08:16:58.653010 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:58.700684 (+ 47674us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:58.900072 (+199388us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:58.934807 (+ 34735us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1053844}
I20240502 08:16:58.940308 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 92adebc0c69e4a52b975253656132495 (table default.Table_308171ca5d9c4bfdb1d31a3b6e92b62f [id=07b4ec205e8a47d2a5d3b46ce99e8a4d]) successfully deleted
I20240502 08:16:58.941588 24175 tablet_bootstrap.cc:492] T 55f3f09fc1f94c00a47b9087b98e221d P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:58.941787 24175 ts_tablet_manager.cc:1367] T 55f3f09fc1f94c00a47b9087b98e221d P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.102s user 0.003s sys 0.000s
I20240502 08:16:58.942479 24175 raft_consensus.cc:348] T 55f3f09fc1f94c00a47b9087b98e221d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.942739 24175 raft_consensus.cc:374] T 55f3f09fc1f94c00a47b9087b98e221d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:58.943509 24175 raft_consensus.cc:729] T 55f3f09fc1f94c00a47b9087b98e221d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:58.943773 24175 consensus_queue.cc:260] T 55f3f09fc1f94c00a47b9087b98e221d P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:58.944348 24175 ts_tablet_manager.cc:1398] T 55f3f09fc1f94c00a47b9087b98e221d P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:58.944773 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bd032329c0574a53a1d3b7d977c4783b" } new_table_name: "default.Table_e78ddf0a532740bb9a319cfb12baf0e3" modify_external_catalogs: true
I20240502 08:16:58.946309 24175 tablet_bootstrap.cc:492] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:58.947921 24175 tablet_bootstrap.cc:654] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.953830 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fa90dfc7938f455082fb034307a1ca25" } new_table_name: "default.Table_5476fef6b04143cb88280d40d634a344" modify_external_catalogs: true
I20240502 08:16:58.965554 23606 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_3edd27da6927478281e76361718425a3 [id=76b58ce209a24b9a95b405c1c50de3ca] alter complete (version 3)
W20240502 08:16:58.965695 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0e9de33c88984befb056647b4c31de6f consensus state has no leader
I20240502 08:16:58.967013 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1fa659ecf3e84aea89cf0a2ef3266bae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:58.967316 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8fb3758a4a62482bbcbcba9b7b659274"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:58.967581 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fa659ecf3e84aea89cf0a2ef3266bae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:58.967670 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fb3758a4a62482bbcbcba9b7b659274 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:58.968298 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a47e8e08a0ef4797aaf83cfa8adf0019" } new_table_name: "default.Table_de22e733a5464ad5830bf5f3a5d41cc9" modify_external_catalogs: true
I20240502 08:16:58.907940 24181 ts_tablet_manager.cc:1888] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:58.971513 24181 log.cc:1196] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/ba3aa2621fdb43fd8299f81af486ef1e
I20240502 08:16:58.971892 24181 ts_tablet_manager.cc:1909] T ba3aa2621fdb43fd8299f81af486ef1e P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:58.972609 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1065) took 1296 ms. Trace:
I20240502 08:16:58.972731 24181 rpcz_store.cc:276] 0502 08:16:57.676123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.676164 (+ 41us) service_pool.cc:225] Handling call
0502 08:16:58.786303 (+1110139us) ts_tablet_manager.cc:1120] Deleting tablet ba3aa2621fdb43fd8299f81af486ef1e
0502 08:16:58.786313 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:58.904798 (+118485us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:58.907842 (+ 3044us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:58.972584 (+ 64742us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1110010}
I20240502 08:16:58.973317 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet ba3aa2621fdb43fd8299f81af486ef1e (table default.Table_4481e3bc8b3548b0a7ecef01d39f80e8 [id=f8b12b3bb4904c18be60b567cb05b739]) successfully deleted
I20240502 08:16:58.973953 24181 tablet_replica.cc:330] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:58.974143 24181 raft_consensus.cc:2229] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:58.974316 24181 raft_consensus.cc:2258] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:58.974913 24181 ts_tablet_manager.cc:1875] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:58.975478 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba13189883454982a99f535701556c05 consensus state has no leader
I20240502 08:16:58.975844 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 58f538766e934282932fd4ab41b49831
I20240502 08:16:58.976354 23604 catalog_manager.cc:6554] ReplaceTablet: tablet 58f538766e934282932fd4ab41b49831 deleted and replaced by tablet 3ccaa109132844d4843d08b108b2960c
I20240502 08:16:58.976640 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet c4f9e61686dc417bb5a88f96b4231f2e from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:58.976966 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 58f538766e934282932fd4ab41b49831 with delete_type TABLET_DATA_DELETED (replaced by tablet 3ccaa109132844d4843d08b108b2960c) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:58.980126 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 58f538766e934282932fd4ab41b49831 with delete_type TABLET_DATA_DELETED (replaced by tablet 3ccaa109132844d4843d08b108b2960c) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:58.981617 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_4f960ddea9ab4d99ba26662e984e4884"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:58.982429 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c9c42f6f968c415d8b407324dc712f58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:58.982875 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9c42f6f968c415d8b407324dc712f58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:58.983695 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e99c6ea531ff4e9f92e130f9e21f0ff6 consensus state has no leader
W20240502 08:16:58.984038 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f960ddea9ab4d99ba26662e984e4884 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:58.985980 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_54d693cd5db04d9999ad9ac4deb13329"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:58.986478 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54d693cd5db04d9999ad9ac4deb13329 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:58.987882 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e34873a3f08043098f2740641b3dc41f consensus state has no leader
I20240502 08:16:58.988219 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e91fa6b91a704e739fbfacbe1ce7ef29" } new_table_name: "default.Table_276307bd54d443d08a9abd552d1b1445" modify_external_catalogs: true
I20240502 08:16:58.997813 25494 raft_consensus.cc:482] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 149ce744feab49b5aa5bcaa14bd6aba5)
I20240502 08:16:58.998018 25494 raft_consensus.cc:504] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:58.999568 23858 leader_election.cc:341] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: cb7b37f776df47da9aa6e7fba82a8999
I20240502 08:16:59.001498 25494 leader_election.cc:288] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:59.001722 23861 leader_election.cc:341] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: cb7b37f776df47da9aa6e7fba82a8999
I20240502 08:16:59.001839 23861 leader_election.cc:302] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:59.003607 24175 tablet_bootstrap.cc:492] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:59.003752 24175 ts_tablet_manager.cc:1367] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.000s
I20240502 08:16:59.004539 25494 raft_consensus.cc:2726] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:59.004431 24175 raft_consensus.cc:348] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.008411 24175 raft_consensus.cc:374] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:59.008534 24175 raft_consensus.cc:729] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:59.008813 24175 consensus_queue.cc:260] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.009460 24175 ts_tablet_manager.cc:1398] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:16:59.009778 24175 tablet_bootstrap.cc:492] T 9467974d67c64c7eb5e488ac51f1240d P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:59.011289 24175 tablet_bootstrap.cc:654] T 9467974d67c64c7eb5e488ac51f1240d P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:58.985613 23904 tablet_service.cc:1514] Processing DeleteTablet for tablet 58f538766e934282932fd4ab41b49831 with delete_type TABLET_DATA_DELETED (replaced by tablet 3ccaa109132844d4843d08b108b2960c) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:59.012682 24174 tablet_bootstrap.cc:492] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:59.013932 24174 tablet_bootstrap.cc:654] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:59.021286 25012 ts_tablet_manager.cc:1888] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:59.021857 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1124) took 1423 ms. Trace:
I20240502 08:16:59.021973 24180 rpcz_store.cc:276] 0502 08:16:57.597914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.599178 (+ 1264us) service_pool.cc:225] Handling call
0502 08:16:58.940612 (+1341434us) ts_tablet_manager.cc:1120] Deleting tablet 592ab8523b8d479d9caead321d5888c3
0502 08:16:59.021758 (+ 81146us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.021839 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1341226}
W20240502 08:16:59.022404 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 592ab8523b8d479d9caead321d5888c3 because the tablet was not found. No further retry: Not found: Tablet not found: 592ab8523b8d479d9caead321d5888c3
I20240502 08:16:59.022728 24180 tablet_replica.cc:330] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:59.022928 24180 raft_consensus.cc:2229] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:59.023126 24180 raft_consensus.cc:2258] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:59.023591 24180 ts_tablet_manager.cc:1875] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:59.029304 25012 log.cc:1196] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/b59c2f93d6514c9cbf358ab96f1a06b5
I20240502 08:16:59.030472 25012 ts_tablet_manager.cc:1909] T b59c2f93d6514c9cbf358ab96f1a06b5 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:59.030836 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_26889a40ce5a43bcbda607dcda6887bf" } modify_external_catalogs: true
I20240502 08:16:59.031010 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_26889a40ce5a43bcbda607dcda6887bf" } modify_external_catalogs: true
I20240502 08:16:59.032514 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5964ac143b9f461287f7b52b806195d0" } modify_external_catalogs: true
I20240502 08:16:59.032653 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5964ac143b9f461287f7b52b806195d0" } modify_external_catalogs: true
I20240502 08:16:59.035517 24180 ts_tablet_manager.cc:1888] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:59.039837 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 960) took 4305 ms. Trace:
I20240502 08:16:59.047730 25405 raft_consensus.cc:482] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:59.047750 25012 rpcz_store.cc:276] 0502 08:16:54.734667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.734715 (+ 48us) service_pool.cc:225] Handling call
0502 08:16:58.885778 (+4151063us) ts_tablet_manager.cc:1120] Deleting tablet b59c2f93d6514c9cbf358ab96f1a06b5
0502 08:16:58.885787 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:58.968554 (+ 82767us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.021190 (+ 52636us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.039782 (+ 18592us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4150918}
I20240502 08:16:59.047892 25405 raft_consensus.cc:504] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.048197 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet b59c2f93d6514c9cbf358ab96f1a06b5 (table default.Table_bbf9d7694fe942de8aa8bd4ed61336a5 [id=f1d6eb661b0143ee82e797cde61391c0]) successfully deleted
I20240502 08:16:59.048528 25405 leader_election.cc:288] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:59.049814 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a5a410b6314818b10927e35032f95d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:59.050410 25651 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1: acquiring CA information for follower catalog manager: success
W20240502 08:16:59.050410 23996 leader_election.cc:341] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:59.053952 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 961) took 4287 ms. Trace:
I20240502 08:16:59.054040 25012 rpcz_store.cc:276] 0502 08:16:54.766165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.767259 (+ 1094us) service_pool.cc:225] Handling call
0502 08:16:59.053806 (+4286547us) ts_tablet_manager.cc:1120] Deleting tablet 31775513f3a446b08c0aac25ae0cab20
0502 08:16:59.053815 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.053905 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4286381}
I20240502 08:16:59.054400 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 962) took 4288 ms. Trace:
W20240502 08:16:59.054430 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 31775513f3a446b08c0aac25ae0cab20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31775513f3a446b08c0aac25ae0cab20 already in progress: creating tablet
I20240502 08:16:59.054486 25012 rpcz_store.cc:276] 0502 08:16:54.766330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.767468 (+ 1138us) service_pool.cc:225] Handling call
0502 08:16:59.054267 (+4286799us) ts_tablet_manager.cc:1120] Deleting tablet 37b499184f31454c96f66cfdf9143d14
0502 08:16:59.054275 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.054355 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4286662}
I20240502 08:16:59.054822 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 969) took 4287 ms. Trace:
W20240502 08:16:59.054832 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 37b499184f31454c96f66cfdf9143d14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37b499184f31454c96f66cfdf9143d14 already in progress: creating tablet
I20240502 08:16:59.054914 25012 rpcz_store.cc:276] 0502 08:16:54.767197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.781658 (+ 14461us) service_pool.cc:225] Handling call
0502 08:16:59.054678 (+4273020us) ts_tablet_manager.cc:1120] Deleting tablet 870edf582ab14ad1b57c4c4469b19954
0502 08:16:59.054685 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.054780 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4272840}
I20240502 08:16:59.055179 25012 tablet_replica.cc:330] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
W20240502 08:16:59.055239 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 870edf582ab14ad1b57c4c4469b19954 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 870edf582ab14ad1b57c4c4469b19954 already in progress: creating tablet
I20240502 08:16:59.055367 25012 raft_consensus.cc:2229] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:59.055552 25012 raft_consensus.cc:2258] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:59.056102 25012 ts_tablet_manager.cc:1875] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:59.064646 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_01a5cb4b00a54d719c7f057e20bbb6e4" } modify_external_catalogs: true
I20240502 08:16:59.064798 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_01a5cb4b00a54d719c7f057e20bbb6e4" } modify_external_catalogs: true
I20240502 08:16:59.064775 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ed62fe1188a546ac8cf56b8f8adcdb62" } modify_external_catalogs: true
I20240502 08:16:59.064944 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ed62fe1188a546ac8cf56b8f8adcdb62" } modify_external_catalogs: true
I20240502 08:16:59.066802 24175 tablet_bootstrap.cc:492] T 9467974d67c64c7eb5e488ac51f1240d P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:59.066965 24175 ts_tablet_manager.cc:1367] T 9467974d67c64c7eb5e488ac51f1240d P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
I20240502 08:16:59.067721 24175 raft_consensus.cc:348] T 9467974d67c64c7eb5e488ac51f1240d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.067945 24175 raft_consensus.cc:374] T 9467974d67c64c7eb5e488ac51f1240d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:59.068038 24175 raft_consensus.cc:729] T 9467974d67c64c7eb5e488ac51f1240d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:59.068260 24175 consensus_queue.cc:260] T 9467974d67c64c7eb5e488ac51f1240d P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.069414 24175 ts_tablet_manager.cc:1398] T 9467974d67c64c7eb5e488ac51f1240d P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:59.069875 24175 tablet_bootstrap.cc:492] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:59.080950 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c738380f4b114dfeb02b6c85557daed0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:59.081463 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c738380f4b114dfeb02b6c85557daed0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.082433 24180 log.cc:1196] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/e5f5597402374527bf0e5e50178df713
I20240502 08:16:59.049247 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a5a410b6314818b10927e35032f95d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:59.063380 25531 raft_consensus.cc:482] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:59.070794 24174 tablet_bootstrap.cc:492] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:59.082660 23786 raft_consensus.cc:2454] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:16:59.082706 24174 ts_tablet_manager.cc:1367] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.070s user 0.008s sys 0.000s
I20240502 08:16:59.083195 23993 leader_election.cc:302] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:59.083395 24174 raft_consensus.cc:348] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.083564 24174 raft_consensus.cc:374] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:59.083600 24174 raft_consensus.cc:729] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:59.083611 25405 raft_consensus.cc:2781] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:59.083680 25405 raft_consensus.cc:482] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:59.083732 25405 raft_consensus.cc:3037] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:59.083760 24174 consensus_queue.cc:260] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.085942 24174 ts_tablet_manager.cc:1398] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:16:59.086212 24174 tablet_bootstrap.cc:492] T f1510777d3cb4ae899542796b5a493ad P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:59.086126 25405 raft_consensus.cc:504] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.082661 25531 raft_consensus.cc:504] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.087764 25531 leader_election.cc:288] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:59.088830 25405 leader_election.cc:288] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:59.089424 24180 ts_tablet_manager.cc:1909] T e5f5597402374527bf0e5e50178df713 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:59.089694 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a198c33a3e314b5b81a08244862b281a" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:59.089860 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a198c33a3e314b5b81a08244862b281a" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:16:59.090241 23723 leader_election.cc:341] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:59.090234 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1126) took 1492 ms. Trace:
I20240502 08:16:59.090687 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a5a410b6314818b10927e35032f95d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:59.090857 23786 raft_consensus.cc:3037] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:59.093281 23786 raft_consensus.cc:2454] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:16:59.094026 23993 leader_election.cc:302] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
W20240502 08:16:59.094489 23723 leader_election.cc:341] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:59.094847 23723 leader_election.cc:302] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:59.095247 25531 raft_consensus.cc:2726] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:59.089171 24181 ts_tablet_manager.cc:1888] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:59.095515 24181 log.cc:1196] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/d6a19bb8705c45c0bfc537d3a6bf8ae2
I20240502 08:16:59.095953 24181 ts_tablet_manager.cc:1909] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:59.095960 25664 raft_consensus.cc:2781] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:59.096276 25664 raft_consensus.cc:686] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:16:59.096558 25664 consensus_queue.cc:237] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.097664 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1066) took 1421 ms. Trace:
I20240502 08:16:59.098672 25423 consensus_queue.cc:1035] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:59.099406 24181 rpcz_store.cc:276] 0502 08:16:57.676658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.676712 (+ 54us) service_pool.cc:225] Handling call
0502 08:16:58.973908 (+1297196us) ts_tablet_manager.cc:1120] Deleting tablet d6a19bb8705c45c0bfc537d3a6bf8ae2
0502 08:16:58.973917 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.006594 (+ 32677us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.089071 (+ 82477us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.097626 (+ 8555us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1296983}
I20240502 08:16:59.099893 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 (table default.Table_c67035cbec4e42109cb3d466e0a6ad18 [id=b2cb0ed2926d4be7ade0790203b57e2e]) successfully deleted
I20240502 08:16:59.101253 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1067) took 1359 ms. Trace:
I20240502 08:16:59.102779 24181 rpcz_store.cc:276] 0502 08:16:57.742229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.742284 (+ 55us) service_pool.cc:225] Handling call
0502 08:16:59.101165 (+1358881us) ts_tablet_manager.cc:1120] Deleting tablet c00f29fb76b44d6ea4f76963d2d0751c
0502 08:16:59.101174 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.101236 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1358647}
W20240502 08:16:59.105221 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2d16819a129448d582a23d49efdaabd2 consensus state has no leader
I20240502 08:16:59.105608 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9b3b4ecdf5194a3a9c896c851c1e2eb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:59.105898 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1077) took 1149 ms. Trace:
I20240502 08:16:59.105904 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cea911d15c5b442b8c1f412b3a03539a" } modify_external_catalogs: true
I20240502 08:16:59.105962 24181 rpcz_store.cc:276] 0502 08:16:57.956471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.956523 (+ 52us) service_pool.cc:225] Handling call
0502 08:16:59.105772 (+1149249us) ts_tablet_manager.cc:1120] Deleting tablet e4a87654d97249418e5df3a99b23c557
0502 08:16:59.105797 (+ 25us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.105885 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1149064}
I20240502 08:16:59.106040 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cea911d15c5b442b8c1f412b3a03539a" } modify_external_catalogs: true
W20240502 08:16:59.106155 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b3b4ecdf5194a3a9c896c851c1e2eb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:59.107268 23597 catalog_manager.cc:4797] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet c00f29fb76b44d6ea4f76963d2d0751c because the tablet was not found. No further retry: Not found: Tablet not found: c00f29fb76b44d6ea4f76963d2d0751c
W20240502 08:16:59.107476 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e4a87654d97249418e5df3a99b23c557 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4a87654d97249418e5df3a99b23c557 already in progress: creating tablet
I20240502 08:16:59.105239 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet f03473f7db3e411186f0db827a6630f7
I20240502 08:16:59.107905 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1081) took 1102 ms. Trace:
I20240502 08:16:59.107967 24181 rpcz_store.cc:276] 0502 08:16:58.005140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.005384 (+ 244us) service_pool.cc:225] Handling call
0502 08:16:59.107802 (+1102418us) ts_tablet_manager.cc:1120] Deleting tablet 876d7dc6291f47e4b0e671b50f2fa8c7
0502 08:16:59.107810 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.107890 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1102232}
I20240502 08:16:59.108305 23603 master_service.cc:946] ReplaceTablet: received request to replace tablet 7ae9c817c7be42da8c689267c7eb5a06 from {username='slave'} at 127.0.0.1:36668
W20240502 08:16:59.108799 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 876d7dc6291f47e4b0e671b50f2fa8c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 876d7dc6291f47e4b0e671b50f2fa8c7 already in progress: creating tablet
I20240502 08:16:59.108954 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet f03473f7db3e411186f0db827a6630f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:59.109234 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1083) took 1103 ms. Trace:
I20240502 08:16:59.109462 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet f03473f7db3e411186f0db827a6630f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:59.109616 24181 rpcz_store.cc:276] 0502 08:16:58.005315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.005384 (+ 69us) service_pool.cc:225] Handling call
0502 08:16:59.108710 (+1103326us) ts_tablet_manager.cc:1120] Deleting tablet 3af208d6231a4cf18823e9fe367ca46d
0502 08:16:59.108717 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.109222 (+ 505us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1103097}
W20240502 08:16:59.111249 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 3af208d6231a4cf18823e9fe367ca46d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3af208d6231a4cf18823e9fe367ca46d already in progress: creating tablet
I20240502 08:16:59.111565 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1084) took 1104 ms. Trace:
I20240502 08:16:59.111660 24181 rpcz_store.cc:276] 0502 08:16:58.006812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.006887 (+ 75us) service_pool.cc:225] Handling call
0502 08:16:59.111483 (+1104596us) ts_tablet_manager.cc:1120] Deleting tablet d2886233097a4d19b45a4f139d82e68a
0502 08:16:59.111490 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.111552 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1096040}
W20240502 08:16:59.112077 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet d2886233097a4d19b45a4f139d82e68a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2886233097a4d19b45a4f139d82e68a already in progress: creating tablet
I20240502 08:16:59.112452 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1082) took 1107 ms. Trace:
I20240502 08:16:59.112536 24181 rpcz_store.cc:276] 0502 08:16:58.005237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.037656 (+ 32419us) service_pool.cc:225] Handling call
0502 08:16:59.112342 (+1074686us) ts_tablet_manager.cc:1120] Deleting tablet c4f9e61686dc417bb5a88f96b4231f2e
0502 08:16:59.112350 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.112436 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1074420}
I20240502 08:16:59.114284 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet b811b81ddf2b4dc9b5e5f80331bff294
I20240502 08:16:59.115648 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1080) took 1118 ms. Trace:
I20240502 08:16:59.115716 24181 rpcz_store.cc:276] 0502 08:16:57.997276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.050541 (+ 53265us) service_pool.cc:225] Handling call
0502 08:16:59.115554 (+1065013us) ts_tablet_manager.cc:1120] Deleting tablet 1b2aa20459484e4b98e0ae372418020f
0502 08:16:59.115563 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.115634 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1064812}
W20240502 08:16:59.115806 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet c4f9e61686dc417bb5a88f96b4231f2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4f9e61686dc417bb5a88f96b4231f2e already in progress: creating tablet
I20240502 08:16:59.116014 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1094) took 1011 ms. Trace:
I20240502 08:16:59.116077 24181 rpcz_store.cc:276] 0502 08:16:58.104264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.104371 (+ 107us) service_pool.cc:225] Handling call
0502 08:16:59.115924 (+1011553us) ts_tablet_manager.cc:1120] Deleting tablet 4bc133adb9a44cc7b56a3746d1eb76af
0502 08:16:59.115931 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.116000 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1011216}
I20240502 08:16:59.116514 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1078) took 1123 ms. Trace:
I20240502 08:16:59.116482 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet b811b81ddf2b4dc9b5e5f80331bff294 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:59.116578 24181 rpcz_store.cc:276] 0502 08:16:57.993498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.126961 (+133463us) service_pool.cc:225] Handling call
0502 08:16:59.116442 (+989481us) ts_tablet_manager.cc:1120] Deleting tablet 63a595654cff4565bcf21a7c8fabc2c4
0502 08:16:59.116448 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.116502 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":980266}
I20240502 08:16:59.116983 24181 tablet_replica.cc:330] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:59.117641 24173 tablet_bootstrap.cc:654] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:59.087774 24174 tablet_bootstrap.cc:654] T f1510777d3cb4ae899542796b5a493ad P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:59.119587 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 1b2aa20459484e4b98e0ae372418020f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b2aa20459484e4b98e0ae372418020f already in progress: creating tablet
W20240502 08:16:59.119813 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 4bc133adb9a44cc7b56a3746d1eb76af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4bc133adb9a44cc7b56a3746d1eb76af already in progress: creating tablet
W20240502 08:16:59.119974 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9 already in progress: creating tablet
W20240502 08:16:59.120116 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 63a595654cff4565bcf21a7c8fabc2c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63a595654cff4565bcf21a7c8fabc2c4 already in progress: creating tablet
W20240502 08:16:59.120311 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e0bd774640464d2a92b2e71126c83883 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0bd774640464d2a92b2e71126c83883 already in progress: creating tablet
I20240502 08:16:59.121829 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet b811b81ddf2b4dc9b5e5f80331bff294 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:59.126385 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a5a410b6314818b10927e35032f95d" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:16:59.127027 23996 leader_election.cc:341] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:59.129644 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet f03473f7db3e411186f0db827a6630f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:59.129881 25357 raft_consensus.cc:482] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 149ce744feab49b5aa5bcaa14bd6aba5)
I20240502 08:16:59.130017 25357 raft_consensus.cc:504] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:16:59.132283 23858 leader_election.cc:341] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 5d04312bd81f46ed8c49c42b1af1aec9
I20240502 08:16:59.090628 24180 rpcz_store.cc:276] 0502 08:16:57.598105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.599429 (+ 1324us) service_pool.cc:225] Handling call
0502 08:16:59.022682 (+1423253us) ts_tablet_manager.cc:1120] Deleting tablet e5f5597402374527bf0e5e50178df713
0502 08:16:59.022693 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.032744 (+ 10051us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.035441 (+ 2697us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.090217 (+ 54776us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1423090}
W20240502 08:16:59.133133 23861 leader_election.cc:341] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 5d04312bd81f46ed8c49c42b1af1aec9
I20240502 08:16:59.133133 25494 raft_consensus.cc:482] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 149ce744feab49b5aa5bcaa14bd6aba5)
I20240502 08:16:59.137454 25357 leader_election.cc:288] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:59.138633 23861 leader_election.cc:302] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:59.139103 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet b811b81ddf2b4dc9b5e5f80331bff294 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:59.139420 25494 raft_consensus.cc:504] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:59.140060 25357 raft_consensus.cc:2726] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:59.141897 25494 leader_election.cc:288] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:59.143288 23858 leader_election.cc:341] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 955e7bd2216646c6aef5acce8d5b0d77
I20240502 08:16:59.143442 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet e5f5597402374527bf0e5e50178df713 (table default.Table_c44e58448899483fb6f36f8ee844e61d [id=ca51b24a648a4f2ea766c671f9058a1c]) successfully deleted
W20240502 08:16:59.143828 23861 leader_election.cc:341] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 955e7bd2216646c6aef5acce8d5b0d77
I20240502 08:16:59.143931 24181 raft_consensus.cc:2229] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:59.144191 24181 pending_rounds.cc:62] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5: Trying to abort 1 pending ops.
I20240502 08:16:59.144264 24181 pending_rounds.cc:68] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156502552535040 op_type: NO_OP noop_request { }
I20240502 08:16:59.144400 24181 raft_consensus.cc:2866] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:59.144490 24181 raft_consensus.cc:2258] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:59.133091 24174 tablet_bootstrap.cc:492] T f1510777d3cb4ae899542796b5a493ad P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:59.144769 24174 ts_tablet_manager.cc:1367] T f1510777d3cb4ae899542796b5a493ad P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.004s
I20240502 08:16:59.144991 24181 ts_tablet_manager.cc:1875] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:59.145745 23861 leader_election.cc:302] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:59.144232 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1125) took 1546 ms. Trace:
I20240502 08:16:59.146198 25494 raft_consensus.cc:2726] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:16:59.146054 24180 rpcz_store.cc:276] 0502 08:16:57.598003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.599620 (+ 1617us) service_pool.cc:225] Handling call
0502 08:16:59.144150 (+1544530us) ts_tablet_manager.cc:1120] Deleting tablet ba3aa2621fdb43fd8299f81af486ef1e
0502 08:16:59.144159 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.144214 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1544378}
I20240502 08:16:59.147886 25012 ts_tablet_manager.cc:1888] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:59.148057 25012 log.cc:1196] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/cb7b37f776df47da9aa6e7fba82a8999
W20240502 08:16:59.148195 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet ba3aa2621fdb43fd8299f81af486ef1e because the tablet was not found. No further retry: Not found: Tablet not found: ba3aa2621fdb43fd8299f81af486ef1e
I20240502 08:16:59.148569 25012 ts_tablet_manager.cc:1909] T cb7b37f776df47da9aa6e7fba82a8999 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:59.148877 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 970) took 4322 ms. Trace:
I20240502 08:16:59.149104 24180 tablet_replica.cc:330] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:59.150059 25012 rpcz_store.cc:276] 0502 08:16:54.826017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.826228 (+ 211us) service_pool.cc:225] Handling call
0502 08:16:59.055110 (+4228882us) ts_tablet_manager.cc:1120] Deleting tablet cb7b37f776df47da9aa6e7fba82a8999
0502 08:16:59.055117 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.099195 (+ 44078us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.145008 (+ 45813us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.148845 (+ 3837us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4228686}
I20240502 08:16:59.150406 25012 tablet_replica.cc:330] T 751808dfc1964b539f6f836d786156aa P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:59.150522 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet cb7b37f776df47da9aa6e7fba82a8999 (table default.Table_3def35abf8ad4ff49c5ae77acb410e01 [id=5550f8374136404e84d93cc14b9cf834]) successfully deleted
I20240502 08:16:59.150701 25012 raft_consensus.cc:2229] T 751808dfc1964b539f6f836d786156aa P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:59.150856 24180 raft_consensus.cc:2229] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:59.150899 25012 raft_consensus.cc:2258] T 751808dfc1964b539f6f836d786156aa P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:59.151654 24180 raft_consensus.cc:2258] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:59.151854 25012 ts_tablet_manager.cc:1875] T 751808dfc1964b539f6f836d786156aa P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:59.154323 24180 ts_tablet_manager.cc:1875] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:59.149964 24174 raft_consensus.cc:348] T f1510777d3cb4ae899542796b5a493ad P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.155345 24174 raft_consensus.cc:374] T f1510777d3cb4ae899542796b5a493ad P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:59.155413 24174 raft_consensus.cc:729] T f1510777d3cb4ae899542796b5a493ad P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:59.155642 24174 consensus_queue.cc:260] T f1510777d3cb4ae899542796b5a493ad P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.167826 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_09bcb43a340f47248dc7fa91d2f73036" } new_table_name: "default.Table_a0e4c771591b4835a81fee0af367743e" modify_external_catalogs: true
I20240502 08:16:59.175647 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_70dac5e6bd8546ec9021c0142d578880"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:59.176215 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70dac5e6bd8546ec9021c0142d578880 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.177153 24174 ts_tablet_manager.cc:1398] T f1510777d3cb4ae899542796b5a493ad P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
I20240502 08:16:59.177403 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d731f4904f8649bdbc1369d357489e48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:59.177883 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d731f4904f8649bdbc1369d357489e48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.183148 25494 raft_consensus.cc:482] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:59.183158 24175 tablet_bootstrap.cc:654] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:59.187456 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e85cd63adc614e97b309693f22e1062a has no consensus state
I20240502 08:16:59.187881 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_138a88014a504ac68fde53841c4779a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:59.188014 24174 tablet_bootstrap.cc:492] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
W20240502 08:16:59.188380 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_138a88014a504ac68fde53841c4779a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.189582 24174 tablet_bootstrap.cc:654] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:59.194865 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c52f7bb042fb42dab6ef2adf2125a8a9" } new_table_name: "default.Table_3bbe1c8ac27b4c5985f7d6ea706d4459" modify_external_catalogs: true
I20240502 08:16:59.196650 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bebe57a661bf4fa1b75922c7847aa294" } new_table_name: "default.Table_ce1c1ef78c6440cab51b82ee16430055" modify_external_catalogs: true
I20240502 08:16:59.202373 24181 ts_tablet_manager.cc:1888] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:59.202526 24181 log.cc:1196] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/89f57e3a39d54b7ab372d4b145b7a696
I20240502 08:16:59.202831 24181 ts_tablet_manager.cc:1909] T 89f57e3a39d54b7ab372d4b145b7a696 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:59.203747 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 89f57e3a39d54b7ab372d4b145b7a696 (table default.Table_28b3620523074c23866e772b117bf0ee [id=b09ad60d787b4740a493cb73c22b94a9]) successfully deleted
I20240502 08:16:59.207374 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e6458d60fe4646ce9109c151feeefacb" } new_table_name: "default.Table_59a3f5f044b94f508d4dc3a1116e0776" modify_external_catalogs: true
W20240502 08:16:59.210005 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 11ebacd4c41941cd8a6ec634e3aeeaf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11ebacd4c41941cd8a6ec634e3aeeaf5 already in progress: creating tablet
W20240502 08:16:59.210623 23597 catalog_manager.cc:4797] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet df08f0d3af4c4c7e8abce656714b1b7f because the tablet was not found. No further retry: Not found: Tablet not found: df08f0d3af4c4c7e8abce656714b1b7f
W20240502 08:16:59.210836 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a059351d148e4376adfbe7128c88c351 consensus state has no leader
W20240502 08:16:59.211206 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 73d7633fd23541f3a553af008dd6a354 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73d7633fd23541f3a553af008dd6a354 already in progress: creating tablet
I20240502 08:16:59.211230 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_db7ab738f1d7409aa87cb31d155d0047"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:59.211946 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db7ab738f1d7409aa87cb31d155d0047 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:59.213070 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4566bc3dbdde463e9392f5ea4fa97741 consensus state has no leader
I20240502 08:16:59.213399 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_566356cef3ff47288d74e05caa23af4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:59.214962 25158 raft_consensus.cc:482] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:59.215682 25158 raft_consensus.cc:504] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.216377 25158 leader_election.cc:288] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:59.217054 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7356ece342aea0d8db667ebad89c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:16:59.217672 23723 leader_election.cc:341] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:59.218653 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 55fb41b8f8f9487a81e98bae9cddaae1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55fb41b8f8f9487a81e98bae9cddaae1 already in progress: creating tablet
I20240502 08:16:59.218938 24181 tablet_replica.cc:330] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:59.219676 24181 raft_consensus.cc:2229] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
W20240502 08:16:59.224525 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b2683333e5cf4718bcf37b3770c3d5e0 has no consensus state
I20240502 08:16:59.225171 24181 pending_rounds.cc:62] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5: Trying to abort 1 pending ops.
I20240502 08:16:59.228421 24173 tablet_bootstrap.cc:492] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:59.229882 24173 ts_tablet_manager.cc:1367] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.424s user 0.022s sys 0.000s
I20240502 08:16:59.230685 24173 raft_consensus.cc:348] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.230916 24173 raft_consensus.cc:374] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:59.230948 25649 raft_consensus.cc:482] T 156c7356ece342aea0d8db667ebad89c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:59.231073 24173 raft_consensus.cc:729] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:59.231384 24173 consensus_queue.cc:260] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.232664 24180 ts_tablet_manager.cc:1888] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:59.232883 25651 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 84f3c07a31414c4f8fb61c44dc9f16e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20240502 08:16:59.233001 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_566356cef3ff47288d74e05caa23af4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.233501 24180 log.cc:1196] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/a5a5a410b6314818b10927e35032f95d
I20240502 08:16:59.233961 24180 ts_tablet_manager.cc:1909] T a5a5a410b6314818b10927e35032f95d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:59.236371 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1123) took 1638 ms. Trace:
I20240502 08:16:59.241103 24173 ts_tablet_manager.cc:1398] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 08:16:59.241521 24180 rpcz_store.cc:276] 0502 08:16:57.597800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.599799 (+ 1999us) service_pool.cc:225] Handling call
0502 08:16:59.148611 (+1548812us) ts_tablet_manager.cc:1120] Deleting tablet a5a5a410b6314818b10927e35032f95d
0502 08:16:59.149064 (+ 453us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.226001 (+ 76937us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.232575 (+ 6574us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.236352 (+ 3777us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1548652}
I20240502 08:16:59.242336 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet a5a5a410b6314818b10927e35032f95d (table default.Table_79bb10e705c047138e1c4363673a65e6 [id=e5b9ab35a16943af94e5108dc0d29426]) successfully deleted
I20240502 08:16:59.242698 24180 tablet_replica.cc:330] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:59.243299 24180 raft_consensus.cc:2229] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:59.244375 24180 raft_consensus.cc:2258] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:59.245270 24180 ts_tablet_manager.cc:1875] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:59.245692 24173 tablet_bootstrap.cc:492] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:59.247251 24173 tablet_bootstrap.cc:654] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:59.249451 25357 raft_consensus.cc:482] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:59.249632 25357 raft_consensus.cc:504] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.250249 25357 leader_election.cc:288] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:59.250378 25649 raft_consensus.cc:504] T 156c7356ece342aea0d8db667ebad89c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.250808 25649 leader_election.cc:288] T 156c7356ece342aea0d8db667ebad89c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:59.251595 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63afd6e4fee24edeaba1209105f1c3e6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:59.251880 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7356ece342aea0d8db667ebad89c" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:59.252310 25158 raft_consensus.cc:482] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:59.252457 25158 raft_consensus.cc:504] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:16:59.252656 23858 consensus_peers.cc:485] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee022e8e86f8477a8fa1e58d52644b03. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:59.252938 25158 leader_election.cc:288] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:59.252872 25494 raft_consensus.cc:504] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.253425 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac50774b2dd241de9cae8096578d183e" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:59.229635 24181 pending_rounds.cc:68] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156504953573376 op_type: NO_OP noop_request { }
I20240502 08:16:59.253612 24181 raft_consensus.cc:2866] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:16:59.253696 24181 raft_consensus.cc:2258] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:16:59.253937 23723 leader_election.cc:341] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:59.254405 23861 leader_election.cc:341] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:59.255053 23861 leader_election.cc:341] T 156c7356ece342aea0d8db667ebad89c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:59.255435 23858 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 60322.2us
I20240502 08:16:59.256625 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63afd6e4fee24edeaba1209105f1c3e6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:59.256801 23786 raft_consensus.cc:2454] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:59.256899 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7356ece342aea0d8db667ebad89c" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:59.257133 23783 raft_consensus.cc:2454] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:16:59.258323 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e645bb6e2948e091e08433944a52f4" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:59.259245 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7356ece342aea0d8db667ebad89c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:59.259428 23916 raft_consensus.cc:2454] T 156c7356ece342aea0d8db667ebad89c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:16:59.260228 23723 leader_election.cc:302] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:59.260586 25158 raft_consensus.cc:2781] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:59.260641 25158 raft_consensus.cc:482] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:59.260685 25158 raft_consensus.cc:3037] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:59.263408 24173 tablet_bootstrap.cc:492] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:59.263674 25158 raft_consensus.cc:504] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.264189 25158 leader_election.cc:288] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:59.264714 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7356ece342aea0d8db667ebad89c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:16:59.265110 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac50774b2dd241de9cae8096578d183e" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:59.265429 24173 ts_tablet_manager.cc:1367] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20240502 08:16:59.265825 24181 ts_tablet_manager.cc:1875] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:59.269990 23723 leader_election.cc:341] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:59.270175 25494 leader_election.cc:288] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:59.266126 24173 raft_consensus.cc:348] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.270433 24173 raft_consensus.cc:374] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:59.270532 24173 raft_consensus.cc:729] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:59.271003 23858 leader_election.cc:302] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:59.270781 24173 consensus_queue.cc:260] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.273053 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6723fdf3d7de45429c461fc29b99783f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:59.274205 24173 ts_tablet_manager.cc:1398] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
W20240502 08:16:59.274420 23858 leader_election.cc:341] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 63e645bb6e2948e091e08433944a52f4
I20240502 08:16:59.274608 23858 leader_election.cc:302] T 156c7356ece342aea0d8db667ebad89c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:59.275276 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7356ece342aea0d8db667ebad89c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:59.275471 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_463d6892a6624bcaa91f7aba10334f94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:59.275727 23723 leader_election.cc:341] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:59.275837 23723 leader_election.cc:302] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
W20240502 08:16:59.275950 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_463d6892a6624bcaa91f7aba10334f94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.276353 25531 raft_consensus.cc:2726] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:59.276626 23920 raft_consensus.cc:3037] T 156c7356ece342aea0d8db667ebad89c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:59.277231 25494 raft_consensus.cc:2781] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:59.277315 25494 raft_consensus.cc:482] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:59.277499 25494 raft_consensus.cc:3037] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:59.277621 24173 tablet_bootstrap.cc:492] T 156c7356ece342aea0d8db667ebad89c P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:59.279011 24173 tablet_bootstrap.cc:654] T 156c7356ece342aea0d8db667ebad89c P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:59.283505 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6723fdf3d7de45429c461fc29b99783f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.290685 24181 ts_tablet_manager.cc:1888] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:59.298911 24181 log.cc:1196] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/592eeecbf5514215b6464bd030081e4b
I20240502 08:16:59.299830 24181 ts_tablet_manager.cc:1909] T 592eeecbf5514215b6464bd030081e4b P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:59.302582 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 592eeecbf5514215b6464bd030081e4b (table default.Table_4191e0f787d948bda654f8bbb6d8d24e [id=07050bd496e84a97a86fdc4f0b7811ac]) successfully deleted
I20240502 08:16:59.303839 24181 tablet_replica.cc:330] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:59.307883 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0e7c8c39d3c4443d8b770a580c137b2a" } new_table_name: "default.Table_088f2741fbb747c1b652fed2e838cc58" modify_external_catalogs: true
W20240502 08:16:59.310393 23861 leader_election.cc:341] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:59.310573 23861 leader_election.cc:302] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:59.312130 25357 raft_consensus.cc:2726] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:59.312330 25357 consensus_queue.cc:1035] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:59.314389 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c54c945517b5468fb9f9b25ed02691d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:59.314760 23920 raft_consensus.cc:2454] T 156c7356ece342aea0d8db667ebad89c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
W20240502 08:16:59.314850 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c54c945517b5468fb9f9b25ed02691d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.316902 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_236150d3903d437d95f67b0edec36091" } modify_external_catalogs: true
I20240502 08:16:59.317059 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_236150d3903d437d95f67b0edec36091" } modify_external_catalogs: true
I20240502 08:16:59.317457 25494 raft_consensus.cc:504] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.318039 25494 leader_election.cc:288] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:59.318224 25649 raft_consensus.cc:2741] T 156c7356ece342aea0d8db667ebad89c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240502 08:16:59.320711 23723 leader_election.cc:302] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:59.321588 24181 raft_consensus.cc:2229] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:59.321599 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63afd6e4fee24edeaba1209105f1c3e6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:16:59.321765 23783 raft_consensus.cc:3037] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:59.321900 24181 raft_consensus.cc:2258] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:59.322427 24181 ts_tablet_manager.cc:1875] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:59.324012 23783 raft_consensus.cc:2454] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:16:59.324239 25158 raft_consensus.cc:2781] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:59.324330 25158 raft_consensus.cc:686] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:59.324534 25158 consensus_queue.cc:237] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.325958 24174 tablet_bootstrap.cc:492] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:59.326082 24174 ts_tablet_manager.cc:1367] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.138s user 0.003s sys 0.003s
I20240502 08:16:59.329524 24174 raft_consensus.cc:348] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.329787 24174 raft_consensus.cc:374] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:59.329876 24174 raft_consensus.cc:729] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:59.330569 23858 leader_election.cc:302] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:16:59.330837 25649 raft_consensus.cc:2781] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:59.330962 25649 raft_consensus.cc:686] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:16:59.331215 25649 consensus_queue.cc:237] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.332234 24174 consensus_queue.cc:260] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.333142 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63afd6e4fee24edeaba1209105f1c3e6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:16:59.334075 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c61243a38bfa4964b225eeae0e8cc28e consensus state has no leader
I20240502 08:16:59.334399 24174 ts_tablet_manager.cc:1398] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:16:59.334342 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_41a2401b3f4c4748b1d3eb964d8053d5" } modify_external_catalogs: true
I20240502 08:16:59.334498 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_41a2401b3f4c4748b1d3eb964d8053d5" } modify_external_catalogs: true
I20240502 08:16:59.334631 24174 tablet_bootstrap.cc:492] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
W20240502 08:16:59.335399 23861 leader_election.cc:341] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:59.338868 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet ec10ff7251a5442a8ecc1881b34db74d
I20240502 08:16:59.338763 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_33f2ee613cae4e848bb987e9041391c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:59.339325 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33f2ee613cae4e848bb987e9041391c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.339488 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_962cbf6a1b424252a27be241d7c3173f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:59.339654 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet ec10ff7251a5442a8ecc1881b34db74d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:16:59.339883 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_962cbf6a1b424252a27be241d7c3173f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.340512 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet ec10ff7251a5442a8ecc1881b34db74d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:59.343242 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet ec10ff7251a5442a8ecc1881b34db74d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:59.345494 25405 consensus_queue.cc:1035] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:59.347342 25531 raft_consensus.cc:482] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:59.347460 25531 raft_consensus.cc:504] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.348024 25531 leader_election.cc:288] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:59.348688 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9467974d67c64c7eb5e488ac51f1240d" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:59.349406 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9467974d67c64c7eb5e488ac51f1240d" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:59.349622 23922 raft_consensus.cc:2454] T 9467974d67c64c7eb5e488ac51f1240d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
W20240502 08:16:59.349920 23723 leader_election.cc:341] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:59.350271 23723 leader_election.cc:302] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:59.350646 25531 raft_consensus.cc:2781] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:59.350715 25531 raft_consensus.cc:482] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:59.350768 25531 raft_consensus.cc:3037] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:59.351887 24174 tablet_bootstrap.cc:654] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:59.353127 25531 raft_consensus.cc:504] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.354255 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9467974d67c64c7eb5e488ac51f1240d" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:16:59.354713 23723 leader_election.cc:341] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:59.358558 25531 leader_election.cc:288] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:59.358605 25012 ts_tablet_manager.cc:1888] T 751808dfc1964b539f6f836d786156aa P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:59.363783 24181 ts_tablet_manager.cc:1888] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:59.363960 24181 log.cc:1196] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/001e3f4f72f64dc69dbd521724912f07
I20240502 08:16:59.364329 24181 ts_tablet_manager.cc:1909] T 001e3f4f72f64dc69dbd521724912f07 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:59.365207 24181 tablet_replica.cc:330] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:59.365320 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 001e3f4f72f64dc69dbd521724912f07 (table default.Table_d7cff78803d44e5aba3b50cd8ff11e92 [id=5c21117955d946418c656330984ae747]) successfully deleted
I20240502 08:16:59.365427 24181 raft_consensus.cc:2229] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:59.365727 24181 raft_consensus.cc:2258] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:59.359211 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9467974d67c64c7eb5e488ac51f1240d" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:16:59.369843 23922 raft_consensus.cc:3037] T 9467974d67c64c7eb5e488ac51f1240d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:59.372443 23922 raft_consensus.cc:2454] T 9467974d67c64c7eb5e488ac51f1240d P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:16:59.368854 24175 tablet_bootstrap.cc:492] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:59.372871 23723 leader_election.cc:302] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:16:59.372962 24175 ts_tablet_manager.cc:1367] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.303s user 0.009s sys 0.000s
I20240502 08:16:59.373353 25531 raft_consensus.cc:2781] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:59.373445 25531 raft_consensus.cc:686] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:16:59.373601 24175 raft_consensus.cc:348] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:59.373769 24175 raft_consensus.cc:374] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:59.373829 24175 raft_consensus.cc:729] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:59.373787 25531 consensus_queue.cc:237] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.374042 24175 consensus_queue.cc:260] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:59.374490 24175 ts_tablet_manager.cc:1398] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:16:59.374768 24175 tablet_bootstrap.cc:492] T b0e8982fe35f47cda376d9646b52e961 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:59.376783 24175 tablet_bootstrap.cc:654] T b0e8982fe35f47cda376d9646b52e961 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:59.379027 25012 log.cc:1196] T 751808dfc1964b539f6f836d786156aa P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/751808dfc1964b539f6f836d786156aa
I20240502 08:16:59.379455 25012 ts_tablet_manager.cc:1909] T 751808dfc1964b539f6f836d786156aa P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:16:59.380818 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 974) took 4491 ms. Trace:
I20240502 08:16:59.381315 25012 rpcz_store.cc:276] 0502 08:16:54.889495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:54.889540 (+ 45us) service_pool.cc:225] Handling call
0502 08:16:59.150341 (+4260801us) ts_tablet_manager.cc:1120] Deleting tablet 751808dfc1964b539f6f836d786156aa
0502 08:16:59.150350 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.306602 (+156252us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.358499 (+ 51897us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.380757 (+ 22258us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4260581}
I20240502 08:16:59.382278 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 751808dfc1964b539f6f836d786156aa (table default.Table_ad81523678c3450c880efd1ecfc5f95e [id=911d4d5a86e54a5696d37a2bd9f69733]) successfully deleted
I20240502 08:16:59.382586 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 975) took 4273 ms. Trace:
I20240502 08:16:59.383383 25012 rpcz_store.cc:276] 0502 08:16:55.108791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.108840 (+ 49us) service_pool.cc:225] Handling call
0502 08:16:59.382433 (+4273593us) ts_tablet_manager.cc:1120] Deleting tablet e9941681b26d4c97b06a91155043b040
0502 08:16:59.382441 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.382524 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4273391}
W20240502 08:16:59.384253 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e9941681b26d4c97b06a91155043b040 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9941681b26d4c97b06a91155043b040 already in progress: creating tablet
I20240502 08:16:59.384781 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 976) took 4221 ms. Trace:
I20240502 08:16:59.370761 24173 tablet_bootstrap.cc:492] T 156c7356ece342aea0d8db667ebad89c P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:59.385739 24173 ts_tablet_manager.cc:1367] T 156c7356ece342aea0d8db667ebad89c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.108s user 0.005s sys 0.000s
I20240502 08:16:59.386104 24181 ts_tablet_manager.cc:1875] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:59.386634 24173 raft_consensus.cc:348] T 156c7356ece342aea0d8db667ebad89c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.386904 24173 raft_consensus.cc:374] T 156c7356ece342aea0d8db667ebad89c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:59.386965 24180 ts_tablet_manager.cc:1888] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:59.387089 24180 log.cc:1196] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/91f1ab141585441db46d3027833932b0
I20240502 08:16:59.387017 24173 raft_consensus.cc:729] T 156c7356ece342aea0d8db667ebad89c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:59.387439 24180 ts_tablet_manager.cc:1909] T 91f1ab141585441db46d3027833932b0 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:59.387722 24173 consensus_queue.cc:260] T 156c7356ece342aea0d8db667ebad89c P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.388334 25012 rpcz_store.cc:276] 0502 08:16:55.163314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.163395 (+ 81us) service_pool.cc:225] Handling call
0502 08:16:59.384637 (+4221242us) ts_tablet_manager.cc:1120] Deleting tablet 89fefde55e514e5fabae3f4e21f44ca3
0502 08:16:59.384647 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.384733 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4220990}
I20240502 08:16:59.388480 24173 ts_tablet_manager.cc:1398] T 156c7356ece342aea0d8db667ebad89c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:16:59.388818 24173 tablet_bootstrap.cc:492] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:16:59.388979 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 89fefde55e514e5fabae3f4e21f44ca3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89fefde55e514e5fabae3f4e21f44ca3 already in progress: creating tablet
I20240502 08:16:59.389353 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 983) took 3567 ms. Trace:
I20240502 08:16:59.390084 24174 tablet_bootstrap.cc:492] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:16:59.391158 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c1c64a390b0440cb9ff7c4d5ae9f9c5f" } new_table_name: "default.Table_8cfb7508d3ca45f5a9c3a8317f071b7c" modify_external_catalogs: true
I20240502 08:16:59.391330 24173 tablet_bootstrap.cc:654] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:59.392582 25012 rpcz_store.cc:276] 0502 08:16:55.821801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.821856 (+ 55us) service_pool.cc:225] Handling call
0502 08:16:59.389230 (+3567374us) ts_tablet_manager.cc:1120] Deleting tablet 39cc4ef1174a4358bad777b0471c5f76
0502 08:16:59.389237 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.389307 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3567189}
I20240502 08:16:59.392912 24174 ts_tablet_manager.cc:1367] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.058s user 0.005s sys 0.000s
W20240502 08:16:59.393409 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 39cc4ef1174a4358bad777b0471c5f76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39cc4ef1174a4358bad777b0471c5f76 already in progress: creating tablet
I20240502 08:16:59.394188 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 979) took 3572 ms. Trace:
I20240502 08:16:59.394966 25012 rpcz_store.cc:276] 0502 08:16:55.821492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.823482 (+ 1990us) service_pool.cc:225] Handling call
0502 08:16:59.394008 (+3570526us) ts_tablet_manager.cc:1120] Deleting tablet 0281c8123eab4c48bddc377f5695c6e0
0502 08:16:59.394019 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.394130 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3570373}
I20240502 08:16:59.395749 24174 raft_consensus.cc:348] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:16:59.397181 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 0281c8123eab4c48bddc377f5695c6e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0281c8123eab4c48bddc377f5695c6e0 already in progress: creating tablet
I20240502 08:16:59.397310 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7aa8726847d34e7facc024d5a2287fcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:59.397655 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 980) took 3576 ms. Trace:
W20240502 08:16:59.397859 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7aa8726847d34e7facc024d5a2287fcf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.397729 25012 rpcz_store.cc:276] 0502 08:16:55.821566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.823660 (+ 2094us) service_pool.cc:225] Handling call
0502 08:16:59.397479 (+3573819us) ts_tablet_manager.cc:1120] Deleting tablet 558a3a4a8383439bb64af1d3eaa64572
0502 08:16:59.397488 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.397602 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3573682}
W20240502 08:16:59.399300 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 558a3a4a8383439bb64af1d3eaa64572 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 558a3a4a8383439bb64af1d3eaa64572 already in progress: creating tablet
I20240502 08:16:59.400651 24174 raft_consensus.cc:374] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:59.400960 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 987) took 3578 ms. Trace:
I20240502 08:16:59.401120 24174 raft_consensus.cc:729] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:16:59.401481 24174 consensus_queue.cc:260] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.401780 25012 rpcz_store.cc:276] 0502 08:16:55.822180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.825093 (+ 2913us) service_pool.cc:225] Handling call
0502 08:16:59.400827 (+3575734us) ts_tablet_manager.cc:1120] Deleting tablet bb57cb00e884420bacc48f16739eb29c
0502 08:16:59.400835 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.400903 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3575573}
W20240502 08:16:59.403112 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet bb57cb00e884420bacc48f16739eb29c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb57cb00e884420bacc48f16739eb29c already in progress: creating tablet
I20240502 08:16:59.403709 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 989) took 3581 ms. Trace:
I20240502 08:16:59.404232 25012 rpcz_store.cc:276] 0502 08:16:55.822333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.825295 (+ 2962us) service_pool.cc:225] Handling call
0502 08:16:59.403589 (+3578294us) ts_tablet_manager.cc:1120] Deleting tablet dfd897ba97204b6788fd2d99cf2a0303
0502 08:16:59.403599 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.403665 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3578172}
W20240502 08:16:59.405131 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet dfd897ba97204b6788fd2d99cf2a0303 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfd897ba97204b6788fd2d99cf2a0303 already in progress: creating tablet
I20240502 08:16:59.405581 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 988) took 3583 ms. Trace:
I20240502 08:16:59.402757 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1130) took 1804 ms. Trace:
I20240502 08:16:59.410964 24180 rpcz_store.cc:276] 0502 08:16:57.598405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.600053 (+ 1648us) service_pool.cc:225] Handling call
0502 08:16:59.242655 (+1642602us) ts_tablet_manager.cc:1120] Deleting tablet 91f1ab141585441db46d3027833932b0
0502 08:16:59.242664 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.354715 (+112051us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.386862 (+ 32147us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.402737 (+ 15875us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1642153}
I20240502 08:16:59.411450 25012 rpcz_store.cc:276] 0502 08:16:55.822259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.833574 (+ 11315us) service_pool.cc:225] Handling call
0502 08:16:59.405436 (+3571862us) ts_tablet_manager.cc:1120] Deleting tablet 36dece738afd42b99f3f38d7fd11a0af
0502 08:16:59.405445 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.405514 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3571714}
I20240502 08:16:59.411984 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 91f1ab141585441db46d3027833932b0 (table default.Table_b141c2b67ffe43ff97d32c9fff1d1fe0 [id=b2163faeeed146d9936a6e1a5cfbc46e]) successfully deleted
I20240502 08:16:59.412462 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1139) took 1707 ms. Trace:
W20240502 08:16:59.412910 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 36dece738afd42b99f3f38d7fd11a0af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36dece738afd42b99f3f38d7fd11a0af already in progress: creating tablet
I20240502 08:16:59.414018 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 982) took 3592 ms. Trace:
W20240502 08:16:59.419814 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f7874af53b4a4120a01e2c92e681ad79 has no consensus state
I20240502 08:16:59.420089 24180 rpcz_store.cc:276] 0502 08:16:57.705415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.705466 (+ 51us) service_pool.cc:225] Handling call
0502 08:16:59.412345 (+1706879us) ts_tablet_manager.cc:1120] Deleting tablet e4a87654d97249418e5df3a99b23c557
0502 08:16:59.412355 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.412446 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1706685}
I20240502 08:16:59.420140 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9cff194855ab490b9918ae0224f606ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:59.420568 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cff194855ab490b9918ae0224f606ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.420723 25012 rpcz_store.cc:276] 0502 08:16:55.821727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:55.837613 (+ 15886us) service_pool.cc:225] Handling call
0502 08:16:59.413865 (+3576252us) ts_tablet_manager.cc:1120] Deleting tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6
0502 08:16:59.413875 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.413956 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3576094}
W20240502 08:16:59.421126 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6 already in progress: creating tablet
W20240502 08:16:59.421578 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e4a87654d97249418e5df3a99b23c557 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4a87654d97249418e5df3a99b23c557 already in progress: creating tablet
I20240502 08:16:59.422394 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 995) took 3414 ms. Trace:
I20240502 08:16:59.422725 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 0dae6a96776c477fb0ada8226d68ea25
I20240502 08:16:59.423430 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dae6a96776c477fb0ada8226d68ea25 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:59.423466 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_4a7258628dc14959818ff6f14d1059f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:59.423605 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_44455df150c841b4b666c5279b4fe64d" } new_table_name: "default.Table_f715c6457abc4380b7a0a0c0a089c744" modify_external_catalogs: true
W20240502 08:16:59.423858 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a7258628dc14959818ff6f14d1059f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.424155 24181 ts_tablet_manager.cc:1888] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:59.424249 24181 log.cc:1196] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/6746c25111d04f20a6f30ce0838d71f5
I20240502 08:16:59.424283 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dae6a96776c477fb0ada8226d68ea25 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:59.424547 24181 ts_tablet_manager.cc:1909] T 6746c25111d04f20a6f30ce0838d71f5 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:59.424548 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dae6a96776c477fb0ada8226d68ea25 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:59.424922 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1140) took 1719 ms. Trace:
I20240502 08:16:59.426291 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 22bc17792cf040748ab672cfc6451e3c
I20240502 08:16:59.426844 23608 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9d372763eb03435f8b7074a802d86ffe" } new_table_name: "default.Table_1b91eb0181b74d5ea0d24f2b9731c155" modify_external_catalogs: true
I20240502 08:16:59.427459 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 22bc17792cf040748ab672cfc6451e3c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:59.427758 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet 22bc17792cf040748ab672cfc6451e3c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:59.428397 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 22bc17792cf040748ab672cfc6451e3c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:59.429515 24180 rpcz_store.cc:276] 0502 08:16:57.705919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.705962 (+ 43us) service_pool.cc:225] Handling call
0502 08:16:59.421703 (+1715741us) ts_tablet_manager.cc:1120] Deleting tablet 63a595654cff4565bcf21a7c8fabc2c4
0502 08:16:59.421714 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.424906 (+ 3192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1715586}
I20240502 08:16:59.438650 25012 rpcz_store.cc:276] 0502 08:16:56.007641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.007706 (+ 65us) service_pool.cc:225] Handling call
0502 08:16:59.422264 (+3414558us) ts_tablet_manager.cc:1120] Deleting tablet fc87f91093b94b209caa6f89691bcde3
0502 08:16:59.422272 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.422347 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3414375}
W20240502 08:16:59.439477 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 63a595654cff4565bcf21a7c8fabc2c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63a595654cff4565bcf21a7c8fabc2c4 already in progress: creating tablet
I20240502 08:16:59.440425 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 63afd6e4fee24edeaba1209105f1c3e6 (table default.Table_9f3ea2fdcdc2477b89d509e868ffe4de [id=84e36915119b4125a1b0d64b2d1647fa]). Expected version 1 got 0
I20240502 08:16:59.440654 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 55f3f09fc1f94c00a47b9087b98e221d (table default.Table_7c74c34063104d54b3ab77c878892069 [id=106dd88de5eb49e89f0042beb7920736]). Expected version 2 got 0
W20240502 08:16:59.440934 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 888d258f63984f728f03c9825f175559 consensus state has no leader
W20240502 08:16:59.442152 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet fc87f91093b94b209caa6f89691bcde3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc87f91093b94b209caa6f89691bcde3 already in progress: creating tablet
I20240502 08:16:59.443179 24180 tablet_replica.cc:330] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:59.443758 24173 tablet_bootstrap.cc:492] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:16:59.444195 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3d8235c36ecc44c6bae973b3cbafa696" } new_table_name: "default.Table_36799657d69e43759def365198950d2e" modify_external_catalogs: true
I20240502 08:16:59.445086 24180 raft_consensus.cc:2229] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:59.445008 24173 ts_tablet_manager.cc:1367] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.000s
I20240502 08:16:59.445349 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 994) took 3437 ms. Trace:
I20240502 08:16:59.445454 25012 rpcz_store.cc:276] 0502 08:16:56.007526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.008546 (+ 1020us) service_pool.cc:225] Handling call
0502 08:16:59.444586 (+3436040us) ts_tablet_manager.cc:1120] Deleting tablet 506644a1ead54a1faeaa8583ef96b449
0502 08:16:59.444596 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.445291 (+ 695us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3435891}
I20240502 08:16:59.446215 24180 raft_consensus.cc:2258] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:59.446147 24173 raft_consensus.cc:348] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.446377 24173 raft_consensus.cc:374] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:59.446468 24173 raft_consensus.cc:729] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:16:59.446655 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1cddae55441d4e559facde8e7046d68a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:59.446738 24173 consensus_queue.cc:260] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:16:59.447175 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1cddae55441d4e559facde8e7046d68a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.447588 24173 ts_tablet_manager.cc:1398] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:59.448009 24173 tablet_bootstrap.cc:492] T 8910bc67c6234a538c8160e7c020653c P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:16:59.449633 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_865f3e827f494b4cb09516c62d3ed201"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:59.450074 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_865f3e827f494b4cb09516c62d3ed201 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.450295 24180 ts_tablet_manager.cc:1875] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:59.450748 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 506644a1ead54a1faeaa8583ef96b449 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 506644a1ead54a1faeaa8583ef96b449 already in progress: creating tablet
I20240502 08:16:59.451330 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 996) took 3399 ms. Trace:
I20240502 08:16:59.455411 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 4e32319fa9244ed2a323b2b3f5940748
I20240502 08:16:59.459921 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 7ae9c817c7be42da8c689267c7eb5a06
W20240502 08:16:59.460013 23608 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 96b8f21c64e343d88d84e9acd0bc69b4 consensus state has no leader
I20240502 08:16:59.460281 23603 catalog_manager.cc:6554] ReplaceTablet: tablet 7ae9c817c7be42da8c689267c7eb5a06 deleted and replaced by tablet b3b0c0dcb2184ad39b984f718f8a2082
I20240502 08:16:59.462769 23601 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 156c7356ece342aea0d8db667ebad89c (table default.Table_d8c6955b89724769bbfd81db2b5be455 [id=15b52f0688c24df2b6f3fda3732ceeb4]). Expected version 1 got 0
I20240502 08:16:59.462965 23601 catalog_manager.cc:5484] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 2, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 2 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:59.463430 23601 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 55f3f09fc1f94c00a47b9087b98e221d (table default.Table_7c74c34063104d54b3ab77c878892069 [id=106dd88de5eb49e89f0042beb7920736]). Expected version 2 got 0
I20240502 08:16:59.463567 23601 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 41f2e98ef2884b05994d02114e992e80 (table default.Table_0a76b688246e4aaaabcbc5ca97aaa72e [id=b6dee89ecfb443d99267917d4df78e4d]). Expected version 2 got 0
I20240502 08:16:59.463856 23610 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_1159b00d82f4483c8209c879e2d3c06a [id=272acbb9a6ca4f2dbcbfea6aacbe9cfc] alter complete (version 4)
I20240502 08:16:59.467592 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet b1a2bcc6937c4a95b71103dbf8738702
I20240502 08:16:59.456869 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e32319fa9244ed2a323b2b3f5940748 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:59.460724 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae9c817c7be42da8c689267c7eb5a06 with delete_type TABLET_DATA_DELETED (replaced by tablet b3b0c0dcb2184ad39b984f718f8a2082) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:59.468935 25012 rpcz_store.cc:276] 0502 08:16:56.051921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.052146 (+ 225us) service_pool.cc:225] Handling call
0502 08:16:59.451160 (+3399014us) ts_tablet_manager.cc:1120] Deleting tablet 16a1c5276f014410b740c87b9971fb48
0502 08:16:59.451172 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.451271 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3398820}
I20240502 08:16:59.463704 25494 raft_consensus.cc:482] T 55f3f09fc1f94c00a47b9087b98e221d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:59.469314 25012 tablet_replica.cc:330] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:16:59.469244 25494 raft_consensus.cc:504] T 55f3f09fc1f94c00a47b9087b98e221d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.469498 25012 raft_consensus.cc:2229] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:16:59.469879 25012 raft_consensus.cc:2258] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:59.468652 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet b1a2bcc6937c4a95b71103dbf8738702 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:59.470451 25012 ts_tablet_manager.cc:1875] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:16:59.470890 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 16a1c5276f014410b740c87b9971fb48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16a1c5276f014410b740c87b9971fb48 already in progress: creating tablet
I20240502 08:16:59.449393 25745 raft_consensus.cc:482] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:59.457795 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e32319fa9244ed2a323b2b3f5940748 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:59.461071 23752 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae9c817c7be42da8c689267c7eb5a06 with delete_type TABLET_DATA_DELETED (replaced by tablet b3b0c0dcb2184ad39b984f718f8a2082) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:59.471283 25745 raft_consensus.cc:504] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.470022 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 2613f7e1576d4c399412720492969fce (DEFAULT_TABLE table=default.Table_195e6d7dde6049f188b2b301fac87a03 [id=b25627144731422f80bdbf7af9cb2837]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.471886 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2613f7e1576d4c399412720492969fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.471935 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 6746c25111d04f20a6f30ce0838d71f5 (table default.Table_02f448a8a9844def8ecda8716fdc352e [id=79a03d22b7e34f76bb3242c640129a13]) successfully deleted
W20240502 08:16:59.471980 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1218fd56b8a4e88827f275f952ad782 consensus state has no leader
I20240502 08:16:59.471346 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f3f09fc1f94c00a47b9087b98e221d" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:59.469285 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet b1a2bcc6937c4a95b71103dbf8738702 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:59.484700 25745 leader_election.cc:288] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:59.493676 23606 master_service.cc:946] ReplaceTablet: received request to replace tablet fd32de329009465c89a4917e2716d3e9 from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:59.497532 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 7624c3fac7c94786b17ed79f90296a37 (DEFAULT_TABLE table=default.Table_225db6e818fc4f8d85229eef53824c79 [id=43dc72fe78704b1b8da42fb3d3e43e31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.497905 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7624c3fac7c94786b17ed79f90296a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.501678 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 0561d829e9c54239958fd2ebb2f332a2 (DEFAULT_TABLE table=default.Table_d9e8b22b519c433ab85326949d23aeb4 [id=f26be750a19c459c99d46b9f7df70010]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.502076 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0561d829e9c54239958fd2ebb2f332a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.503235 23752 tablet_service.cc:1467] Processing CreateTablet for tablet 0561d829e9c54239958fd2ebb2f332a2 (DEFAULT_TABLE table=default.Table_d9e8b22b519c433ab85326949d23aeb4 [id=f26be750a19c459c99d46b9f7df70010]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.503537 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0561d829e9c54239958fd2ebb2f332a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.503491 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 7855653d11d24e10ac8911f0fe5c98af (DEFAULT_TABLE table=default.Table_d18d476de2064b649fe32153819d1176 [id=4e7e2a4b4e69443bbafe89557524e611]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.503768 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7855653d11d24e10ac8911f0fe5c98af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.505843 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 2613f7e1576d4c399412720492969fce (DEFAULT_TABLE table=default.Table_195e6d7dde6049f188b2b301fac87a03 [id=b25627144731422f80bdbf7af9cb2837]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.506219 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2613f7e1576d4c399412720492969fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.509227 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 7624c3fac7c94786b17ed79f90296a37 (DEFAULT_TABLE table=default.Table_225db6e818fc4f8d85229eef53824c79 [id=43dc72fe78704b1b8da42fb3d3e43e31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.509537 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7624c3fac7c94786b17ed79f90296a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.514302 23609 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_5301ffccbac743618a78289c2c1415e1 [id=910c7cae36264e39bbfca0be919538e0] alter complete (version 1)
I20240502 08:16:59.469903 25494 leader_election.cc:288] T 55f3f09fc1f94c00a47b9087b98e221d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:59.521541 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 451e499e498c478b8cce9b187362aa4c (DEFAULT_TABLE table=default.Table_8ffe5dd9e3154677a31e2eac5921f22d [id=13e076118dc74fa098123fe50883baf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.484479 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe583cbf4af493e82d3ec0508ea0f0b" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:59.521692 23764 tablet_service.cc:1467] Processing CreateTablet for tablet fd35353b4ea5406f952a5c4941032c41 (DEFAULT_TABLE table=default.Table_67d67d37c02a4642ad3d42d348ffbbac [id=f95c07de19264445aaa4209c7a565e03]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.494941 23920 raft_consensus.cc:3037] T 55f3f09fc1f94c00a47b9087b98e221d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:59.521907 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451e499e498c478b8cce9b187362aa4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.522164 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 30de4d9cb40e4037b99b71bff5667e8c (DEFAULT_TABLE table=default.Table_ff3ffdf15eba464b8f3d89be9f35a38f [id=cd3d264c45e14e29804a8d3a46daf6af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.522457 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30de4d9cb40e4037b99b71bff5667e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.523502 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_df77dff2af6544ff9fa5702b6768b0d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:59.524014 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df77dff2af6544ff9fa5702b6768b0d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.525353 23759 tablet_service.cc:1467] Processing CreateTablet for tablet af68443901cf44ff8e6981e7bfbdcc59 (DEFAULT_TABLE table=default.Table_4717a4024e6a4e4aadf5e4dd07a27d7a [id=88aaccf645054cd1a0be20d7338facdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.525439 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_52d62af96d2c404982cd9ba08010884b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:59.525693 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af68443901cf44ff8e6981e7bfbdcc59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.525756 23755 tablet_service.cc:1467] Processing CreateTablet for tablet f2016fc145314ad788b8c8c2c53bf810 (DEFAULT_TABLE table=default.Table_d20a3b8ecb6449a5a4cc9837354619b7 [id=1c25911cce0845ffb760a96926714b0f]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:59.525935 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52d62af96d2c404982cd9ba08010884b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.526083 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2016fc145314ad788b8c8c2c53bf810. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.526721 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 7855653d11d24e10ac8911f0fe5c98af (DEFAULT_TABLE table=default.Table_d18d476de2064b649fe32153819d1176 [id=4e7e2a4b4e69443bbafe89557524e611]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.527052 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7855653d11d24e10ac8911f0fe5c98af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.527478 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet fd32de329009465c89a4917e2716d3e9
I20240502 08:16:59.527873 23606 catalog_manager.cc:6554] ReplaceTablet: tablet fd32de329009465c89a4917e2716d3e9 deleted and replaced by tablet 5fd449e9a7f84f7880254e859d128755
I20240502 08:16:59.528479 23756 tablet_service.cc:1467] Processing CreateTablet for tablet b2683333e5cf4718bcf37b3770c3d5e0 (DEFAULT_TABLE table=default.Table_59a3f5f044b94f508d4dc3a1116e0776 [id=ffcdb4f0361849039a59f021e82e2ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.528755 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2683333e5cf4718bcf37b3770c3d5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.529666 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 9467974d67c64c7eb5e488ac51f1240d with delete_type TABLET_DATA_DELETED (replaced by tablet 29f6c3cea4b14364b6d073a92171119d) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:59.529755 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 12ab5772462d49878f54ba515d8f8f4c (DEFAULT_TABLE table=default.Table_42e74a4247614e8d9cafd1adf4935783 [id=863543703e7b45b5bc51d35f5be9d1e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.529790 23895 tablet_service.cc:1467] Processing CreateTablet for tablet c5fa429485c34d4395ab139c3872610c (DEFAULT_TABLE table=default.Table_b79fbb283cbd4146aa1b162c998ab4e3 [id=aaab57313bf242839e27610e765fe67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.529953 23757 tablet_service.cc:1467] Processing CreateTablet for tablet c5fa429485c34d4395ab139c3872610c (DEFAULT_TABLE table=default.Table_b79fbb283cbd4146aa1b162c998ab4e3 [id=aaab57313bf242839e27610e765fe67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.530064 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ab5772462d49878f54ba515d8f8f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.530086 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5fa429485c34d4395ab139c3872610c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.530251 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5fa429485c34d4395ab139c3872610c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.530001 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet fd32de329009465c89a4917e2716d3e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 5fd449e9a7f84f7880254e859d128755) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:59.530387 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fe583cbf4af493e82d3ec0508ea0f0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:59.530861 23903 tablet_service.cc:1467] Processing CreateTablet for tablet f7874af53b4a4120a01e2c92e681ad79 (DEFAULT_TABLE table=default.Table_8cfb7508d3ca45f5a9c3a8317f071b7c [id=a9870d37ceb5462aa83d9ad41e11f60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.531183 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7874af53b4a4120a01e2c92e681ad79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.521692 23762 tablet_service.cc:1467] Processing CreateTablet for tablet f7874af53b4a4120a01e2c92e681ad79 (DEFAULT_TABLE table=default.Table_8cfb7508d3ca45f5a9c3a8317f071b7c [id=a9870d37ceb5462aa83d9ad41e11f60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.533923 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7874af53b4a4120a01e2c92e681ad79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.534078 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet e5f5597402374527bf0e5e50178df713 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:59.534368 23887 tablet_service.cc:1467] Processing CreateTablet for tablet fd35353b4ea5406f952a5c4941032c41 (DEFAULT_TABLE table=default.Table_67d67d37c02a4642ad3d42d348ffbbac [id=f95c07de19264445aaa4209c7a565e03]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:59.535363 23723 leader_election.cc:341] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:59.536490 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_206aa937a259485a90fa3dd13e7bd838" } new_table_name: "default.Table_3d9f65a3a821487f9988657695881965" modify_external_catalogs: true
I20240502 08:16:59.537715 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd35353b4ea5406f952a5c4941032c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.541414 25531 raft_consensus.cc:482] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:59.541525 25531 raft_consensus.cc:504] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.545308 25531 leader_election.cc:288] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:59.545682 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1345a4fd2d3e46deaecb136b548f5b3d" } modify_external_catalogs: true
I20240502 08:16:59.545859 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1345a4fd2d3e46deaecb136b548f5b3d" } modify_external_catalogs: true
I20240502 08:16:59.547276 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_686a41d458d04191b29159f0630ae285"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:59.547751 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_686a41d458d04191b29159f0630ae285 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.548959 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 30de4d9cb40e4037b99b71bff5667e8c (DEFAULT_TABLE table=default.Table_ff3ffdf15eba464b8f3d89be9f35a38f [id=cd3d264c45e14e29804a8d3a46daf6af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.552743 23602 catalog_manager.cc:5484] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:59.553138 23602 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 156c7356ece342aea0d8db667ebad89c (table default.Table_d8c6955b89724769bbfd81db2b5be455 [id=15b52f0688c24df2b6f3fda3732ceeb4]). Expected version 1 got 0
I20240502 08:16:59.553318 23602 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 74b4c563d6bd4a44bb3bab58a3fcdc07 (table default.Table_2ee0f7ddd41e41ea84e25359cd3ed0a4 [id=a9c009feee0e486794a8500121fe5487]). Expected version 2 got 0
I20240502 08:16:59.555346 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fcfd135764fe45359b74185bcca4bd5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:59.555402 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1e603637c1ff4cb5855d8b4c7bbb2ea2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:59.555756 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fcfd135764fe45359b74185bcca4bd5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:59.555835 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e603637c1ff4cb5855d8b4c7bbb2ea2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.529762 23752 tablet_service.cc:1467] Processing CreateTablet for tablet 859694d7f6114a7f8657836763b3a48a (DEFAULT_TABLE table=default.Table_4e4cde0945344d63bd522607b883ed2f [id=535cd1961eb74df9996b9ab3e29378fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.557142 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 859694d7f6114a7f8657836763b3a48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.553524 23602 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 9761df0bbdee44af90135bffb8b5828f (table default.Table_8ba240c391da43978f93a27dce48e557 [id=1b5536950db244d1978b154da0be33e0]). Expected version 2 got 0
I20240502 08:16:59.558288 23602 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet ac50774b2dd241de9cae8096578d183e (table default.Table_2e0fc2a0993e4208b05240b1ddc4d2ad [id=30defee3de2445f7bc5017fb23ab2ee7]). Expected version 1 got 0
I20240502 08:16:59.558410 23602 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 41f2e98ef2884b05994d02114e992e80 (table default.Table_0a76b688246e4aaaabcbc5ca97aaa72e [id=b6dee89ecfb443d99267917d4df78e4d]). Expected version 2 got 0
W20240502 08:16:59.559440 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8c93c89e1341474eafb6030b95f1c68c consensus state has no leader
I20240502 08:16:59.559688 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 451e499e498c478b8cce9b187362aa4c (DEFAULT_TABLE table=default.Table_8ffe5dd9e3154677a31e2eac5921f22d [id=13e076118dc74fa098123fe50883baf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.560946 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd35353b4ea5406f952a5c4941032c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.561791 23897 tablet_service.cc:1467] Processing CreateTablet for tablet 859694d7f6114a7f8657836763b3a48a (DEFAULT_TABLE table=default.Table_4e4cde0945344d63bd522607b883ed2f [id=535cd1961eb74df9996b9ab3e29378fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.563395 25494 raft_consensus.cc:482] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a297a6c7c2504ca68e87b24f6691ffb2)
I20240502 08:16:59.563544 25494 raft_consensus.cc:504] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:59.564975 23891 tablet_service.cc:1467] Processing CreateTablet for tablet af68443901cf44ff8e6981e7bfbdcc59 (DEFAULT_TABLE table=default.Table_4717a4024e6a4e4aadf5e4dd07a27d7a [id=88aaccf645054cd1a0be20d7338facdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.567713 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 859694d7f6114a7f8657836763b3a48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.568804 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 2a3e66ef4a914f009cca6d418f825d4e
I20240502 08:16:59.569589 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a3e66ef4a914f009cca6d418f825d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:16:59.570255 23858 leader_election.cc:341] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 45f02efa659e457ab57bc32ed3f66bb9
I20240502 08:16:59.584599 24174 ts_tablet_manager.cc:1398] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.191s user 0.003s sys 0.003s
I20240502 08:16:59.584388 25743 raft_consensus.cc:482] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:59.586512 25743 raft_consensus.cc:504] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.595510 23429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2vVFYb/build/asan/bin/kudu pid 24241
I20240502 08:16:59.604297 23602 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 74b4c563d6bd4a44bb3bab58a3fcdc07 (table default.Table_2ee0f7ddd41e41ea84e25359cd3ed0a4 [id=a9c009feee0e486794a8500121fe5487]). Expected version 2 got 0
I20240502 08:16:59.617432 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet fd32de329009465c89a4917e2716d3e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 5fd449e9a7f84f7880254e859d128755) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:59.617393 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 12ab5772462d49878f54ba515d8f8f4c (DEFAULT_TABLE table=default.Table_42e74a4247614e8d9cafd1adf4935783 [id=863543703e7b45b5bc51d35f5be9d1e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.617475 23898 tablet_service.cc:1467] Processing CreateTablet for tablet f2016fc145314ad788b8c8c2c53bf810 (DEFAULT_TABLE table=default.Table_d20a3b8ecb6449a5a4cc9837354619b7 [id=1c25911cce0845ffb760a96926714b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.619117 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451e499e498c478b8cce9b187362aa4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.623777 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_38d10d7d3ecf4f89accb1c4dbfebdbea" } new_table_name: "default.Table_3a2e4270c3d04bb39c94940fb2adb74a" modify_external_catalogs: true
I20240502 08:16:59.625406 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18625a1d971c43549b9f80e6046efc9c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:59.626552 25494 leader_election.cc:288] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:59.628424 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e29ac7dfdff84ee7b3859d45e8ff4af6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:59.628580 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3edd27da6927478281e76361718425a3" } modify_external_catalogs: true
I20240502 08:16:59.628746 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3edd27da6927478281e76361718425a3" } modify_external_catalogs: true
W20240502 08:16:59.629019 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e29ac7dfdff84ee7b3859d45e8ff4af6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.629632 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b465ef0aa8664f5d86736ed308c81e32" } modify_external_catalogs: true
I20240502 08:16:59.629756 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b465ef0aa8664f5d86736ed308c81e32" } modify_external_catalogs: true
W20240502 08:16:59.635546 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3519b9c0ff2546c882ac9a92add72b40 consensus state has no leader
I20240502 08:16:59.641264 23890 tablet_service.cc:1467] Processing CreateTablet for tablet b2683333e5cf4718bcf37b3770c3d5e0 (DEFAULT_TABLE table=default.Table_59a3f5f044b94f508d4dc3a1116e0776 [id=ffcdb4f0361849039a59f021e82e2ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.641721 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2683333e5cf4718bcf37b3770c3d5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.641642 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_04fa70e3d5d941d5b22e076bc4c3daca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:59.642076 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04fa70e3d5d941d5b22e076bc4c3daca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.645678 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet d9b67c8cd7f34cbba6401da59dec526c
I20240502 08:16:59.646978 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6b486df3232a4864b82575bd742573c2" } new_table_name: "default.Table_e883d2bea1d049fea7b1486d4de57d81" modify_external_catalogs: true
I20240502 08:16:59.647344 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet a2da8b9d3f374d8a812336057c0ce7d2
I20240502 08:16:59.652345 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_00868cc9cd204b939d987c1878b0b1e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:59.652833 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00868cc9cd204b939d987c1878b0b1e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.657136 23610 catalog_manager.cc:5484] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66). New cstate: current_term: 1 leader_uuid: "5b89a1f0485a44b7ac1710e2369a809b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: HEALTHY } } }
W20240502 08:16:59.657560 23861 leader_election.cc:341] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 45f02efa659e457ab57bc32ed3f66bb9
I20240502 08:16:59.658548 23858 leader_election.cc:302] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:16:59.658944 23861 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 90739us
I20240502 08:16:59.660403 25494 raft_consensus.cc:482] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:59.663194 25494 raft_consensus.cc:504] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.663874 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2016fc145314ad788b8c8c2c53bf810. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.665133 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ab5772462d49878f54ba515d8f8f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.665771 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_42e57ecead1e4463ba08536696995fec" } new_table_name: "default.Table_3f20959ef29c41fd9d855ea08db2764b" modify_external_catalogs: true
I20240502 08:16:59.666119 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3d9f65a3a821487f9988657695881965" } modify_external_catalogs: true
I20240502 08:16:59.676237 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3d9f65a3a821487f9988657695881965" } modify_external_catalogs: true
W20240502 08:16:59.669858 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8d8a96668e14c088de6c82d549d35ec consensus state has no leader
I20240502 08:16:59.669951 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_06da23829ca541b7ae4be7b15db26c34" } modify_external_catalogs: true
I20240502 08:16:59.677637 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_06da23829ca541b7ae4be7b15db26c34" } modify_external_catalogs: true
I20240502 08:16:59.678555 25494 leader_election.cc:288] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:59.678717 25494 raft_consensus.cc:2726] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:16:59.681022 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fc85cce715742cf999ad41ebc1deebd consensus state has no leader
I20240502 08:16:59.682633 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_55e1a889107f4aa8894c1ce227ddba9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:59.683113 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55e1a889107f4aa8894c1ce227ddba9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.684006 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30de4d9cb40e4037b99b71bff5667e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.684844 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 8c93c89e1341474eafb6030b95f1c68c
I20240502 08:16:59.684841 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet c44ce35c463f416794edccc7149aa0cf
I20240502 08:16:59.686836 25494 raft_consensus.cc:482] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a297a6c7c2504ca68e87b24f6691ffb2)
I20240502 08:16:59.686964 25494 raft_consensus.cc:504] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:59.673765 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5aa2066b846c4cc2b13f2cfa5c2dec4f" } new_table_name: "default.Table_07ecb6448334470f87b32c50e3430f4c" modify_external_catalogs: true
I20240502 08:16:59.687623 25494 leader_election.cc:288] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:59.674130 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a676dcd2d5234d0f896dd378de2b9f34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:59.688323 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a676dcd2d5234d0f896dd378de2b9f34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:59.688788 23861 leader_election.cc:341] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: c00f29fb76b44d6ea4f76963d2d0751c
I20240502 08:16:59.692410 23610 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 63afd6e4fee24edeaba1209105f1c3e6 (table default.Table_9f3ea2fdcdc2477b89d509e868ffe4de [id=84e36915119b4125a1b0d64b2d1647fa]). Expected version 1 got 0
W20240502 08:16:59.697203 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1626382a260044eb9100b8048332abb0 consensus state has no leader
W20240502 08:16:59.701967 23858 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.65:38709, user_credentials={real_user=slave}} blocked reactor thread for 107195us
I20240502 08:16:59.705663 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e11d405a2e4e518a10bf383e9ae6c4" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:59.707351 23916 raft_consensus.cc:1261] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [term 2 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 08:16:59.709007 25494 raft_consensus.cc:482] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a297a6c7c2504ca68e87b24f6691ffb2)
I20240502 08:16:59.709079 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 88db59bea69747ca92e66ed0f611db71 (DEFAULT_TABLE table=default.Table_04ee6fb74c7c4370a57d57b3b9aa3226 [id=1574ae68b6a24298b5cbfab7e975a9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.709270 25494 raft_consensus.cc:504] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:59.709468 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88db59bea69747ca92e66ed0f611db71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.709822 25494 leader_election.cc:288] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:16:59.710688 23861 leader_election.cc:341] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 4b8d81f071d74b218fcfd4f4e8c20bab
I20240502 08:16:59.712174 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet d9b67c8cd7f34cbba6401da59dec526c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:59.712554 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 5047bd45cb4745c09e984f4ff1485d00 (DEFAULT_TABLE table=default.Table_037c4757772643e5bf3a793f1ed95e38 [id=66c92c55b42d480f8e3b5128b016df8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.712894 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5047bd45cb4745c09e984f4ff1485d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.717399 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 077dad62b3474965b229fc3fd6cc7460 (DEFAULT_TABLE table=default.Table_bef474d64fe543c28023e61872f4b725 [id=69450580b22d4ffe962c6ade508a2bae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.717875 23895 tablet_service.cc:1467] Processing CreateTablet for tablet e85cd63adc614e97b309693f22e1062a (DEFAULT_TABLE table=default.Table_a0e4c771591b4835a81fee0af367743e [id=3a6df813fd7043afa480aa3825af2030]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.718349 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85cd63adc614e97b309693f22e1062a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.721696 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af68443901cf44ff8e6981e7bfbdcc59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.725706 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a3e66ef4a914f009cca6d418f825d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:59.726537 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 077dad62b3474965b229fc3fd6cc7460. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.731245 23897 tablet_service.cc:1467] Processing CreateTablet for tablet 9578fd7ce1d545fca6524accc783e38b (DEFAULT_TABLE table=default.Table_0b5edfb1bf6e470f97bbbb62d09e353f [id=c79cdf3f081f4e42a01d425cc46ba015]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.731778 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9578fd7ce1d545fca6524accc783e38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.747995 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 890face76104472587bfefb04b809bbe (DEFAULT_TABLE table=default.Table_6d600d6bd3814d73ad7a816b2ef4ee3e [id=272ab3a7afae4728940be20469bf1c42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.748447 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890face76104472587bfefb04b809bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.752813 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet a2da8b9d3f374d8a812336057c0ce7d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:59.753434 23896 tablet_service.cc:1467] Processing CreateTablet for tablet afeb2d145d2349df9e4a67ce54697ece (DEFAULT_TABLE table=default.Table_0911b0f21cf34f8a80774273c164a5d1 [id=2c385279bc084bf3b33d0212cdf6d272]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.753747 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afeb2d145d2349df9e4a67ce54697ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.758772 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_77418fa37d9c4f4fbe5b9dc2120df056"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:59.759092 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 8d09ed60e1b24d468311764a6e5da03c (DEFAULT_TABLE table=default.Table_91e1ada585af482cbe82348b4f4c2087 [id=775e82aac71c4a378d29a10e65b5bf87]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:59.759274 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77418fa37d9c4f4fbe5b9dc2120df056 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.759357 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d09ed60e1b24d468311764a6e5da03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.759317 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_498fd09f968c4a7a944e5ad1e5326439"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:59.759740 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_498fd09f968c4a7a944e5ad1e5326439 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.760452 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 6b084cd1ba154e98afe828b219a3cbd5 (DEFAULT_TABLE table=default.Table_fa1b2e7c94124643a4e7e8a0d0a4939d [id=990ae264e6234226bfc8307e4bf75e6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.760715 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b084cd1ba154e98afe828b219a3cbd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.761344 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2ec1dfc748994bfe8841412d61167608" } new_table_name: "default.Table_0162a223a99040c5822d77ce464300c3" modify_external_catalogs: true
I20240502 08:16:59.764195 23889 tablet_service.cc:1467] Processing CreateTablet for tablet adce127b0ca7458cbb836b8a5a159a8a (DEFAULT_TABLE table=default.Table_15060d38700048cf9cd741f97742efe9 [id=4486f7afe1244d1b846fecce2db9e23b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.764469 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adce127b0ca7458cbb836b8a5a159a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.770576 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 50df099e743d49fab20e46a6c66e0b0a (DEFAULT_TABLE table=default.Table_eccfd1a3ea3d4ab286b931f657c62607 [id=0f5008dd89ff4db69275e80653d5d40c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.771025 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50df099e743d49fab20e46a6c66e0b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.771680 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a394560ca3454cb88c3cbc2e87ab4fa7" } modify_external_catalogs: true
I20240502 08:16:59.771848 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a394560ca3454cb88c3cbc2e87ab4fa7" } modify_external_catalogs: true
I20240502 08:16:59.778199 24175 tablet_bootstrap.cc:492] T b0e8982fe35f47cda376d9646b52e961 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:59.778349 24175 ts_tablet_manager.cc:1367] T b0e8982fe35f47cda376d9646b52e961 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.404s user 0.008s sys 0.002s
I20240502 08:16:59.779137 24175 raft_consensus.cc:348] T b0e8982fe35f47cda376d9646b52e961 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.779320 24175 raft_consensus.cc:374] T b0e8982fe35f47cda376d9646b52e961 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:59.779368 24175 raft_consensus.cc:729] T b0e8982fe35f47cda376d9646b52e961 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:59.779559 24175 consensus_queue.cc:260] T b0e8982fe35f47cda376d9646b52e961 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.780098 24175 ts_tablet_manager.cc:1398] T b0e8982fe35f47cda376d9646b52e961 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:16:59.780366 24175 tablet_bootstrap.cc:492] T a198c33a3e314b5b81a08244862b281a P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:16:59.782608 24175 tablet_bootstrap.cc:654] T a198c33a3e314b5b81a08244862b281a P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:59.783938 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 2035b4a2c2524c3b86b20ff1bfbd3a0a (DEFAULT_TABLE table=default.Table_249529c4082f44ccba05014ebef7543b [id=0248850c5eb94d9b961561fc9d528922]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.784373 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2035b4a2c2524c3b86b20ff1bfbd3a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.790129 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet c44ce35c463f416794edccc7149aa0cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:59.790297 23888 tablet_service.cc:1467] Processing CreateTablet for tablet fc36a14cbf0d42b59aad7f8078c71e2f (DEFAULT_TABLE table=default.Table_6dd2e8e9898c4e51945829fcf6afe67c [id=85895c4187254e70b001856e0ea93385]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.790860 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 1140f52809d04a0fb077b819f8f78bed (DEFAULT_TABLE table=default.Table_98f610e5d20b467e8e69ed5915b862ab [id=16086c07bca8466fab8540c1f29ab4d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.791077 23901 tablet_service.cc:1467] Processing CreateTablet for tablet ab22a189c3a34136878bef9165fbac09 (DEFAULT_TABLE table=default.Table_6c97a1a9206b45f39095fef784e77788 [id=1aca948dcfce46b78ee9998da5548c90]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.791384 23899 tablet_service.cc:1467] Processing CreateTablet for tablet cb9f8c92b3d34caab0b9f28770ae9c88 (DEFAULT_TABLE table=default.Table_d871e4eda37844d49e6042631c03b1c3 [id=b6761e183a014ff388faed33967b1c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.790863 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c93c89e1341474eafb6030b95f1c68c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:59.791077 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 2d10ff285c724baeaf766ab2fe3585f3 (DEFAULT_TABLE table=default.Table_465187f6217440139197568c3a016f33 [id=2814526dc6bc4fae936ba1413b5c07aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.793222 24175 tablet_bootstrap.cc:492] T a198c33a3e314b5b81a08244862b281a P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:16:59.793395 24175 ts_tablet_manager.cc:1367] T a198c33a3e314b5b81a08244862b281a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 08:16:59.794338 24175 raft_consensus.cc:348] T a198c33a3e314b5b81a08244862b281a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.794567 24175 raft_consensus.cc:374] T a198c33a3e314b5b81a08244862b281a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:59.794664 24175 raft_consensus.cc:729] T a198c33a3e314b5b81a08244862b281a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:16:59.794948 24175 consensus_queue.cc:260] T a198c33a3e314b5b81a08244862b281a P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.795590 24175 ts_tablet_manager.cc:1398] T a198c33a3e314b5b81a08244862b281a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:16:59.797864 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e645bb6e2948e091e08433944a52f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:59.798306 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 9467974d67c64c7eb5e488ac51f1240d with delete_type TABLET_DATA_DELETED (replaced by tablet 29f6c3cea4b14364b6d073a92171119d) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:59.798696 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet eb96ad2289754e428a1c109b2d59c53b with delete_type TABLET_DATA_DELETED (replaced by tablet 6ee764398b364d029ae7f0f575a30d64) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:59.807782 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf9fc9e13d840fdbbf2fe498410c288 (DEFAULT_TABLE table=default.Table_1692280f84c44944b308c28c5918f6f8 [id=a9fd4ae9bc7a408585a109fbdce1b4b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.588691 25743 leader_election.cc:288] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:59.594103 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 18625a1d971c43549b9f80e6046efc9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:59.814862 23767 tablet_service.cc:1467] Processing CreateTablet for tablet fc36a14cbf0d42b59aad7f8078c71e2f (DEFAULT_TABLE table=default.Table_6dd2e8e9898c4e51945829fcf6afe67c [id=85895c4187254e70b001856e0ea93385]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.815021 25649 raft_consensus.cc:482] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a297a6c7c2504ca68e87b24f6691ffb2)
I20240502 08:16:59.815222 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc36a14cbf0d42b59aad7f8078c71e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.815137 25649 raft_consensus.cc:504] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:16:59.816499 23861 leader_election.cc:341] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: d6a19bb8705c45c0bfc537d3a6bf8ae2
I20240502 08:16:59.816629 25649 leader_election.cc:288] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:59.621116 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 890face76104472587bfefb04b809bbe (DEFAULT_TABLE table=default.Table_6d600d6bd3814d73ad7a816b2ef4ee3e [id=272ab3a7afae4728940be20469bf1c42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.817245 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890face76104472587bfefb04b809bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.818840 24173 tablet_bootstrap.cc:654] T 8910bc67c6234a538c8160e7c020653c P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:59.627679 23762 tablet_service.cc:1467] Processing CreateTablet for tablet e85cd63adc614e97b309693f22e1062a (DEFAULT_TABLE table=default.Table_a0e4c771591b4835a81fee0af367743e [id=3a6df813fd7043afa480aa3825af2030]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.819801 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85cd63adc614e97b309693f22e1062a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.618046 24180 ts_tablet_manager.cc:1888] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:59.821713 24180 log.cc:1196] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/eb96ad2289754e428a1c109b2d59c53b
I20240502 08:16:59.821974 24180 ts_tablet_manager.cc:1909] T eb96ad2289754e428a1c109b2d59c53b P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:59.822645 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1141) took 2019 ms. Trace:
I20240502 08:16:59.822712 24180 rpcz_store.cc:276] 0502 08:16:57.803073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.803126 (+ 53us) service_pool.cc:225] Handling call
0502 08:16:59.443128 (+1640002us) ts_tablet_manager.cc:1120] Deleting tablet eb96ad2289754e428a1c109b2d59c53b
0502 08:16:59.443138 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.491938 (+ 48800us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.617961 (+126023us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.822625 (+204664us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1639828}
I20240502 08:16:59.457056 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e32319fa9244ed2a323b2b3f5940748 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:59.823109 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1142) took 1880 ms. Trace:
I20240502 08:16:59.823218 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet fd32de329009465c89a4917e2716d3e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 5fd449e9a7f84f7880254e859d128755) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:59.823171 24180 rpcz_store.cc:276] 0502 08:16:57.942403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.942456 (+ 53us) service_pool.cc:225] Handling call
0502 08:16:59.823000 (+1880544us) ts_tablet_manager.cc:1120] Deleting tablet 3af208d6231a4cf18823e9fe367ca46d
0502 08:16:59.823010 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.823095 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1880338}
I20240502 08:16:59.823355 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a3e66ef4a914f009cca6d418f825d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:59.823446 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1143) took 1880 ms. Trace:
I20240502 08:16:59.823501 24180 rpcz_store.cc:276] 0502 08:16:57.942933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.942975 (+ 42us) service_pool.cc:225] Handling call
0502 08:16:59.823356 (+1880381us) ts_tablet_manager.cc:1120] Deleting tablet d2886233097a4d19b45a4f139d82e68a
0502 08:16:59.823364 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.823434 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1880227}
I20240502 08:16:59.823742 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1145) took 1880 ms. Trace:
I20240502 08:16:59.823798 24180 rpcz_store.cc:276] 0502 08:16:57.943578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.943622 (+ 44us) service_pool.cc:225] Handling call
0502 08:16:59.823663 (+1880041us) ts_tablet_manager.cc:1120] Deleting tablet 876d7dc6291f47e4b0e671b50f2fa8c7
0502 08:16:59.823671 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.823730 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1879919}
I20240502 08:16:59.823938 24032 tablet_service.cc:1467] Processing CreateTablet for tablet e85cd63adc614e97b309693f22e1062a (DEFAULT_TABLE table=default.Table_a0e4c771591b4835a81fee0af367743e [id=3a6df813fd7043afa480aa3825af2030]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.824067 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1144) took 1880 ms. Trace:
I20240502 08:16:59.632738 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 9578fd7ce1d545fca6524accc783e38b (DEFAULT_TABLE table=default.Table_0b5edfb1bf6e470f97bbbb62d09e353f [id=c79cdf3f081f4e42a01d425cc46ba015]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.824127 24180 rpcz_store.cc:276] 0502 08:16:57.943355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.943771 (+ 416us) service_pool.cc:225] Handling call
0502 08:16:59.823983 (+1880212us) ts_tablet_manager.cc:1120] Deleting tablet 1b2aa20459484e4b98e0ae372418020f
0502 08:16:59.823990 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.824054 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1880062}
I20240502 08:16:59.824368 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9578fd7ce1d545fca6524accc783e38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.589993 24174 tablet_bootstrap.cc:492] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:16:59.825706 24174 tablet_bootstrap.cc:654] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:16:59.826594 23596 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 3777d2e9d4464552ad1c4c8c0d3e9888 (table default.Table_660621104fba4545bc2b8c30dc7d035a [id=78b34e35d763495aadf6a9eb242741c9], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3777d2e9d4464552ad1c4c8c0d3e9888 consensus state has no leader
I20240502 08:16:59.829330 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c93c89e1341474eafb6030b95f1c68c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:59.829905 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet eb96ad2289754e428a1c109b2d59c53b (table default.Table_bc1e9d346aa846f8abca7504185f7cdb [id=8d27878c6c8f4234908a2c78cb8bbd29]) successfully deleted
I20240502 08:16:59.460683 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae9c817c7be42da8c689267c7eb5a06 with delete_type TABLET_DATA_DELETED (replaced by tablet b3b0c0dcb2184ad39b984f718f8a2082) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:59.830384 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 077dad62b3474965b229fc3fd6cc7460 (DEFAULT_TABLE table=default.Table_bef474d64fe543c28023e61872f4b725 [id=69450580b22d4ffe962c6ade508a2bae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.465938 24026 tablet_service.cc:1514] Processing DeleteTablet for tablet ee022e8e86f8477a8fa1e58d52644b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:59.831439 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 9578fd7ce1d545fca6524accc783e38b (DEFAULT_TABLE table=default.Table_0b5edfb1bf6e470f97bbbb62d09e353f [id=c79cdf3f081f4e42a01d425cc46ba015]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.467314 24034 tablet_service.cc:1514] Processing DeleteTablet for tablet 592eeecbf5514215b6464bd030081e4b with delete_type TABLET_DATA_DELETED (replaced by tablet 28a947b7e7484120b3f4069f6eef5520) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:59.831835 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 5047bd45cb4745c09e984f4ff1485d00 (DEFAULT_TABLE table=default.Table_037c4757772643e5bf3a793f1ed95e38 [id=66c92c55b42d480f8e3b5128b016df8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.468183 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet b1a2bcc6937c4a95b71103dbf8738702 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:16:59.832217 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 3af208d6231a4cf18823e9fe367ca46d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3af208d6231a4cf18823e9fe367ca46d already in progress: creating tablet
I20240502 08:16:59.832275 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 88db59bea69747ca92e66ed0f611db71 (DEFAULT_TABLE table=default.Table_04ee6fb74c7c4370a57d57b3b9aa3226 [id=1574ae68b6a24298b5cbfab7e975a9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.469657 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 2613f7e1576d4c399412720492969fce (DEFAULT_TABLE table=default.Table_195e6d7dde6049f188b2b301fac87a03 [id=b25627144731422f80bdbf7af9cb2837]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:59.832453 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet d2886233097a4d19b45a4f139d82e68a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2886233097a4d19b45a4f139d82e68a already in progress: creating tablet
I20240502 08:16:59.470464 24053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f3f09fc1f94c00a47b9087b98e221d" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:59.471827 24181 tablet_replica.cc:330] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
W20240502 08:16:59.832624 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 876d7dc6291f47e4b0e671b50f2fa8c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 876d7dc6291f47e4b0e671b50f2fa8c7 already in progress: creating tablet
W20240502 08:16:59.832777 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 1b2aa20459484e4b98e0ae372418020f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b2aa20459484e4b98e0ae372418020f already in progress: creating tablet
I20240502 08:16:59.483657 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe583cbf4af493e82d3ec0508ea0f0b" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:59.638566 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 5047bd45cb4745c09e984f4ff1485d00 (DEFAULT_TABLE table=default.Table_037c4757772643e5bf3a793f1ed95e38 [id=66c92c55b42d480f8e3b5128b016df8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.833760 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5047bd45cb4745c09e984f4ff1485d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.500485 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 7624c3fac7c94786b17ed79f90296a37 (DEFAULT_TABLE table=default.Table_225db6e818fc4f8d85229eef53824c79 [id=43dc72fe78704b1b8da42fb3d3e43e31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.500864 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 0561d829e9c54239958fd2ebb2f332a2 (DEFAULT_TABLE table=default.Table_d9e8b22b519c433ab85326949d23aeb4 [id=f26be750a19c459c99d46b9f7df70010]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.519197 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 7855653d11d24e10ac8911f0fe5c98af (DEFAULT_TABLE table=default.Table_d18d476de2064b649fe32153819d1176 [id=4e7e2a4b4e69443bbafe89557524e611]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.519474 24021 tablet_service.cc:1467] Processing CreateTablet for tablet 859694d7f6114a7f8657836763b3a48a (DEFAULT_TABLE table=default.Table_4e4cde0945344d63bd522607b883ed2f [id=535cd1961eb74df9996b9ab3e29378fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.519691 24023 tablet_service.cc:1467] Processing CreateTablet for tablet b2683333e5cf4718bcf37b3770c3d5e0 (DEFAULT_TABLE table=default.Table_59a3f5f044b94f508d4dc3a1116e0776 [id=ffcdb4f0361849039a59f021e82e2ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.519912 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 451e499e498c478b8cce9b187362aa4c (DEFAULT_TABLE table=default.Table_8ffe5dd9e3154677a31e2eac5921f22d [id=13e076118dc74fa098123fe50883baf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.520143 24035 tablet_service.cc:1467] Processing CreateTablet for tablet f7874af53b4a4120a01e2c92e681ad79 (DEFAULT_TABLE table=default.Table_8cfb7508d3ca45f5a9c3a8317f071b7c [id=a9870d37ceb5462aa83d9ad41e11f60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.520218 24022 tablet_service.cc:1467] Processing CreateTablet for tablet c5fa429485c34d4395ab139c3872610c (DEFAULT_TABLE table=default.Table_b79fbb283cbd4146aa1b162c998ab4e3 [id=aaab57313bf242839e27610e765fe67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.520378 24038 tablet_service.cc:1467] Processing CreateTablet for tablet fd35353b4ea5406f952a5c4941032c41 (DEFAULT_TABLE table=default.Table_67d67d37c02a4642ad3d42d348ffbbac [id=f95c07de19264445aaa4209c7a565e03]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.520570 24040 tablet_service.cc:1467] Processing CreateTablet for tablet af68443901cf44ff8e6981e7bfbdcc59 (DEFAULT_TABLE table=default.Table_4717a4024e6a4e4aadf5e4dd07a27d7a [id=88aaccf645054cd1a0be20d7338facdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.520620 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 30de4d9cb40e4037b99b71bff5667e8c (DEFAULT_TABLE table=default.Table_ff3ffdf15eba464b8f3d89be9f35a38f [id=cd3d264c45e14e29804a8d3a46daf6af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.520855 24027 tablet_service.cc:1467] Processing CreateTablet for tablet f2016fc145314ad788b8c8c2c53bf810 (DEFAULT_TABLE table=default.Table_d20a3b8ecb6449a5a4cc9837354619b7 [id=1c25911cce0845ffb760a96926714b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.520884 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 12ab5772462d49878f54ba515d8f8f4c (DEFAULT_TABLE table=default.Table_42e74a4247614e8d9cafd1adf4935783 [id=863543703e7b45b5bc51d35f5be9d1e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.521104 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 890face76104472587bfefb04b809bbe (DEFAULT_TABLE table=default.Table_6d600d6bd3814d73ad7a816b2ef4ee3e [id=272ab3a7afae4728940be20469bf1c42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.543661 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18625a1d971c43549b9f80e6046efc9c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:59.836892 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 353db059d276404ca9f338ebbca021ac (DEFAULT_TABLE table=default.Table_cbac61918e774f599192caeeaef51577 [id=8a198543bbf1462eaf9c0d1577190ab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.545892 24052 raft_consensus.cc:1261] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Refusing update from remote peer a297a6c7c2504ca68e87b24f6691ffb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 08:16:59.573666 25423 raft_consensus.cc:482] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:59.837415 25423 raft_consensus.cc:504] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.838017 25423 leader_election.cc:288] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:59.645817 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 88db59bea69747ca92e66ed0f611db71 (DEFAULT_TABLE table=default.Table_04ee6fb74c7c4370a57d57b3b9aa3226 [id=1574ae68b6a24298b5cbfab7e975a9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.824373 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1146) took 1880 ms. Trace:
I20240502 08:16:59.840291 24180 rpcz_store.cc:276] 0502 08:16:57.944221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.944263 (+ 42us) service_pool.cc:225] Handling call
0502 08:16:59.824293 (+1880030us) ts_tablet_manager.cc:1120] Deleting tablet c4f9e61686dc417bb5a88f96b4231f2e
0502 08:16:59.824301 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.824361 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1879890}
I20240502 08:16:59.840478 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88db59bea69747ca92e66ed0f611db71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.840595 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1156) took 1869 ms. Trace:
I20240502 08:16:59.840646 24180 rpcz_store.cc:276] 0502 08:16:57.971426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.971471 (+ 45us) service_pool.cc:225] Handling call
0502 08:16:59.840505 (+1869034us) ts_tablet_manager.cc:1120] Deleting tablet 4bc133adb9a44cc7b56a3746d1eb76af
0502 08:16:59.840511 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.840581 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1868878}
I20240502 08:16:59.840901 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1162) took 1749 ms. Trace:
I20240502 08:16:59.840979 24180 rpcz_store.cc:276] 0502 08:16:58.091314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.092113 (+ 799us) service_pool.cc:225] Handling call
0502 08:16:59.840820 (+1748707us) ts_tablet_manager.cc:1120] Deleting tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9
0502 08:16:59.840827 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.840889 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1562390}
I20240502 08:16:59.841205 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1175) took 1574 ms. Trace:
I20240502 08:16:59.841252 24180 rpcz_store.cc:276] 0502 08:16:58.266858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.278460 (+ 11602us) service_pool.cc:225] Handling call
0502 08:16:59.841133 (+1562673us) ts_tablet_manager.cc:1120] Deleting tablet 11ebacd4c41941cd8a6ec634e3aeeaf5
0502 08:16:59.841139 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.841195 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1562540}
I20240502 08:16:59.841454 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1176) took 1574 ms. Trace:
I20240502 08:16:59.841502 24180 rpcz_store.cc:276] 0502 08:16:58.267196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.278603 (+ 11407us) service_pool.cc:225] Handling call
0502 08:16:59.841388 (+1562785us) ts_tablet_manager.cc:1120] Deleting tablet 73d7633fd23541f3a553af008dd6a354
0502 08:16:59.841395 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.841445 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1562699}
I20240502 08:16:59.841768 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1177) took 1568 ms. Trace:
I20240502 08:16:59.841815 24180 rpcz_store.cc:276] 0502 08:16:58.272821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.278697 (+ 5876us) service_pool.cc:225] Handling call
0502 08:16:59.841696 (+1562999us) ts_tablet_manager.cc:1120] Deleting tablet 55fb41b8f8f9487a81e98bae9cddaae1
0502 08:16:59.841703 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.841757 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1562918}
I20240502 08:16:59.842051 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1174) took 1577 ms. Trace:
I20240502 08:16:59.842103 24180 rpcz_store.cc:276] 0502 08:16:58.264503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.266313 (+ 1810us) service_pool.cc:225] Handling call
0502 08:16:59.841969 (+1575656us) ts_tablet_manager.cc:1120] Deleting tablet e0bd774640464d2a92b2e71126c83883
0502 08:16:59.841978 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.842038 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1547922}
I20240502 08:16:59.842278 24180 tablet_replica.cc:330] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:59.842432 24180 raft_consensus.cc:2229] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:59.842540 24180 raft_consensus.cc:2258] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:59.842995 24180 ts_tablet_manager.cc:1875] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:59.844417 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet c44ce35c463f416794edccc7149aa0cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:59.844704 23764 tablet_service.cc:1467] Processing CreateTablet for tablet ab22a189c3a34136878bef9165fbac09 (DEFAULT_TABLE table=default.Table_6c97a1a9206b45f39095fef784e77788 [id=1aca948dcfce46b78ee9998da5548c90]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.845103 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 1140f52809d04a0fb077b819f8f78bed (DEFAULT_TABLE table=default.Table_98f610e5d20b467e8e69ed5915b862ab [id=16086c07bca8466fab8540c1f29ab4d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.646791 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet d9b67c8cd7f34cbba6401da59dec526c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:59.648514 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet a2da8b9d3f374d8a812336057c0ce7d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:59.648679 23761 tablet_service.cc:1467] Processing CreateTablet for tablet afeb2d145d2349df9e4a67ce54697ece (DEFAULT_TABLE table=default.Table_0911b0f21cf34f8a80774273c164a5d1 [id=2c385279bc084bf3b33d0212cdf6d272]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.653483 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 8d09ed60e1b24d468311764a6e5da03c (DEFAULT_TABLE table=default.Table_91e1ada585af482cbe82348b4f4c2087 [id=775e82aac71c4a378d29a10e65b5bf87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.845693 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab22a189c3a34136878bef9165fbac09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.845922 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d09ed60e1b24d468311764a6e5da03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.658485 23765 tablet_service.cc:1467] Processing CreateTablet for tablet cb9f8c92b3d34caab0b9f28770ae9c88 (DEFAULT_TABLE table=default.Table_d871e4eda37844d49e6042631c03b1c3 [id=b6761e183a014ff388faed33967b1c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.849841 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9f8c92b3d34caab0b9f28770ae9c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.660578 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf9fc9e13d840fdbbf2fe498410c288 (DEFAULT_TABLE table=default.Table_1692280f84c44944b308c28c5918f6f8 [id=a9fd4ae9bc7a408585a109fbdce1b4b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.853811 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf9fc9e13d840fdbbf2fe498410c288. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.838558 23861 leader_election.cc:302] T 55f3f09fc1f94c00a47b9087b98e221d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:16:59.854765 25649 raft_consensus.cc:2726] T 55f3f09fc1f94c00a47b9087b98e221d P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:59.663440 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 50df099e743d49fab20e46a6c66e0b0a (DEFAULT_TABLE table=default.Table_eccfd1a3ea3d4ab286b931f657c62607 [id=0f5008dd89ff4db69275e80653d5d40c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.858393 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50df099e743d49fab20e46a6c66e0b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.859658 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc36a14cbf0d42b59aad7f8078c71e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.672324 23750 tablet_service.cc:1467] Processing CreateTablet for tablet adce127b0ca7458cbb836b8a5a159a8a (DEFAULT_TABLE table=default.Table_15060d38700048cf9cd741f97742efe9 [id=4486f7afe1244d1b846fecce2db9e23b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.861800 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adce127b0ca7458cbb836b8a5a159a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.674309 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 6b084cd1ba154e98afe828b219a3cbd5 (DEFAULT_TABLE table=default.Table_fa1b2e7c94124643a4e7e8a0d0a4939d [id=990ae264e6234226bfc8307e4bf75e6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.866192 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b084cd1ba154e98afe828b219a3cbd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.866470 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf9fc9e13d840fdbbf2fe498410c288. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:59.702538 23723 leader_election.cc:341] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:59.869586 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 167050us
W20240502 08:16:59.870069 23723 leader_election.cc:341] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:59.870170 23723 leader_election.cc:302] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
W20240502 08:16:59.870565 23723 leader_election.cc:341] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:59.870641 23723 leader_election.cc:302] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:16:59.871032 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_04f57727640b4aceabd37ba02a21237a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:59.871471 23723 leader_election.cc:341] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:59.871564 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04f57727640b4aceabd37ba02a21237a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.871824 25531 raft_consensus.cc:2726] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:59.871940 25531 raft_consensus.cc:2726] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:59.872082 25531 consensus_queue.cc:1035] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:59.872435 25531 consensus_queue.cc:1035] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:59.873260 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_791a010a6e484e10a6749eefcfdc293d" } new_table_name: "default.Table_540a424c4bfa4484a65f0b24140a4bbb" modify_external_catalogs: true
I20240502 08:16:59.873268 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9ff802fb6dbc44b7bd29d961c12e0d3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:59.677522 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 2035b4a2c2524c3b86b20ff1bfbd3a0a (DEFAULT_TABLE table=default.Table_249529c4082f44ccba05014ebef7543b [id=0248850c5eb94d9b961561fc9d528922]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.873720 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2035b4a2c2524c3b86b20ff1bfbd3a0a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:59.873728 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ff802fb6dbc44b7bd29d961c12e0d3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.875555 23602 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 55f3f09fc1f94c00a47b9087b98e221d (table default.Table_7c74c34063104d54b3ab77c878892069 [id=106dd88de5eb49e89f0042beb7920736]). Expected version 2 got 0
I20240502 08:16:59.876310 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ca502551d3ba4c76868d9a79c06a8569" } modify_external_catalogs: true
I20240502 08:16:59.876485 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ca502551d3ba4c76868d9a79c06a8569" } modify_external_catalogs: true
I20240502 08:16:59.877044 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_97f00fbbeae84ee782f7028ca32b8556" } modify_external_catalogs: true
I20240502 08:16:59.877203 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_97f00fbbeae84ee782f7028ca32b8556" } modify_external_catalogs: true
I20240502 08:16:59.877792 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9edd6347f7cb47229111f08382effee5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:59.682152 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 353db059d276404ca9f338ebbca021ac (DEFAULT_TABLE table=default.Table_cbac61918e774f599192caeeaef51577 [id=8a198543bbf1462eaf9c0d1577190ab0]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:59.878173 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9edd6347f7cb47229111f08382effee5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.878206 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 353db059d276404ca9f338ebbca021ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.881362 24180 ts_tablet_manager.cc:1888] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:59.881443 24180 log.cc:1196] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/a198c33a3e314b5b81a08244862b281a
I20240502 08:16:59.881727 24180 ts_tablet_manager.cc:1909] T a198c33a3e314b5b81a08244862b281a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:59.881841 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1140f52809d04a0fb077b819f8f78bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.599704 23752 tablet_service.cc:1514] Processing DeleteTablet for tablet 42e11d405a2e4e518a10bf383e9ae6c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:16:59.632584 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 077dad62b3474965b229fc3fd6cc7460 (DEFAULT_TABLE table=default.Table_bef474d64fe543c28023e61872f4b725 [id=69450580b22d4ffe962c6ade508a2bae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.882203 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 077dad62b3474965b229fc3fd6cc7460. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.885816 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab22a189c3a34136878bef9165fbac09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.679826 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 2d10ff285c724baeaf766ab2fe3585f3 (DEFAULT_TABLE table=default.Table_465187f6217440139197568c3a016f33 [id=2814526dc6bc4fae936ba1413b5c07aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.886946 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d10ff285c724baeaf766ab2fe3585f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.890486 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1184) took 1376 ms. Trace:
I20240502 08:16:59.890556 24180 rpcz_store.cc:276] 0502 08:16:58.513926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.513983 (+ 57us) service_pool.cc:225] Handling call
0502 08:16:59.842246 (+1328263us) ts_tablet_manager.cc:1120] Deleting tablet a198c33a3e314b5b81a08244862b281a
0502 08:16:59.842253 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.850970 (+ 8717us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.881295 (+ 30325us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.890467 (+ 9172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1328060}
I20240502 08:16:59.890861 24180 tablet_replica.cc:330] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:59.890908 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afeb2d145d2349df9e4a67ce54697ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.891034 24180 raft_consensus.cc:2229] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:59.891170 24180 raft_consensus.cc:2258] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:59.891649 24180 ts_tablet_manager.cc:1875] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:59.902335 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_3da446d477714e8cbb0ac5817ec962a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:59.902511 23993 leader_election.cc:341] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 91f1ab141585441db46d3027833932b0
W20240502 08:16:59.902901 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3da446d477714e8cbb0ac5817ec962a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:16:59.661214 23996 consensus_peers.cc:485] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:16:59.903141 23996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 241961us
I20240502 08:16:59.647500 25664 consensus_queue.cc:1035] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:59.674589 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5f5597402374527bf0e5e50178df713" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:59.904080 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85cd63adc614e97b309693f22e1062a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.905736 24181 raft_consensus.cc:2229] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:59.905905 24181 raft_consensus.cc:2258] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:59.906414 24181 ts_tablet_manager.cc:1875] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:59.909111 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 077dad62b3474965b229fc3fd6cc7460. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.913774 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9578fd7ce1d545fca6524accc783e38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.914018 25649 raft_consensus.cc:482] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:59.914141 25649 raft_consensus.cc:504] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:16:59.914273 24180 ts_tablet_manager.cc:1888] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:16:59.914382 24180 log.cc:1196] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/001e3f4f72f64dc69dbd521724912f07
I20240502 08:16:59.914706 24180 ts_tablet_manager.cc:1909] T 001e3f4f72f64dc69dbd521724912f07 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:16:59.914696 25649 leader_election.cc:288] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:16:59.915380 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1194) took 1084 ms. Trace:
I20240502 08:16:59.915452 24180 rpcz_store.cc:276] 0502 08:16:58.830742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.832286 (+ 1544us) service_pool.cc:225] Handling call
0502 08:16:59.890800 (+1058514us) ts_tablet_manager.cc:1120] Deleting tablet 001e3f4f72f64dc69dbd521724912f07
0502 08:16:59.890808 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.911459 (+ 20651us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.914196 (+ 2737us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.915361 (+ 1165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1058325}
I20240502 08:16:59.915735 24180 tablet_replica.cc:330] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:16:59.915925 24180 raft_consensus.cc:2229] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:16:59.916064 24180 raft_consensus.cc:2258] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:59.916561 24180 ts_tablet_manager.cc:1875] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:59.917848 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5047bd45cb4745c09e984f4ff1485d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.918128 24181 ts_tablet_manager.cc:1888] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:16:59.918213 24181 log.cc:1196] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/a4d151770ee44130adfd822518b9b017
I20240502 08:16:59.918540 24181 ts_tablet_manager.cc:1909] T a4d151770ee44130adfd822518b9b017 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:16:59.919479 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1128) took 1158 ms. Trace:
I20240502 08:16:59.919565 24181 rpcz_store.cc:276] 0502 08:16:58.761021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.766489 (+ 5468us) service_pool.cc:225] Handling call
0502 08:16:59.471779 (+705290us) ts_tablet_manager.cc:1120] Deleting tablet a4d151770ee44130adfd822518b9b017
0502 08:16:59.471788 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.915431 (+443643us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.918057 (+ 2626us) tablet_metadata.cc:641] Metadata flushed
0502 08:16:59.919445 (+ 1388us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72966,"tablet-delete.queue_time_us":623010}
I20240502 08:16:59.920018 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1140) took 1047 ms. Trace:
I20240502 08:16:59.920056 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet a4d151770ee44130adfd822518b9b017 (table default.Table_fb39fa7949f3412c83974132184b5d9f [id=8b62b02f324d4b6cbc7095de109955b5]) successfully deleted
I20240502 08:16:59.920076 24181 rpcz_store.cc:276] 0502 08:16:58.873019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.873076 (+ 57us) service_pool.cc:225] Handling call
0502 08:16:59.919916 (+1046840us) ts_tablet_manager.cc:1120] Deleting tablet 3ad53633ce57478194437f95135d81de
0502 08:16:59.919923 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.920004 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1046578}
I20240502 08:16:59.920356 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1138) took 1047 ms. Trace:
W20240502 08:16:59.920437 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 3ad53633ce57478194437f95135d81de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ad53633ce57478194437f95135d81de already in progress: creating tablet
I20240502 08:16:59.920413 24181 rpcz_store.cc:276] 0502 08:16:58.872759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.873620 (+ 861us) service_pool.cc:225] Handling call
0502 08:16:59.920267 (+1046647us) ts_tablet_manager.cc:1120] Deleting tablet 990d0cc863ee4f49a2fc2fffb217c4d7
0502 08:16:59.920276 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.920343 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1046425}
I20240502 08:16:59.920675 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1137) took 1048 ms. Trace:
W20240502 08:16:59.920725 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 990d0cc863ee4f49a2fc2fffb217c4d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 990d0cc863ee4f49a2fc2fffb217c4d7 already in progress: creating tablet
I20240502 08:16:59.920730 24181 rpcz_store.cc:276] 0502 08:16:58.872627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.882603 (+ 9976us) service_pool.cc:225] Handling call
0502 08:16:59.920589 (+1037986us) ts_tablet_manager.cc:1120] Deleting tablet e2792c45255c42adaeefb7aef4e25715
0502 08:16:59.920598 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.920661 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1037338}
I20240502 08:16:59.921247 24181 tablet_replica.cc:330] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:16:59.921406 24181 raft_consensus.cc:2229] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 3 LEADER]: Raft consensus shutting down.
W20240502 08:16:59.921416 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e2792c45255c42adaeefb7aef4e25715 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2792c45255c42adaeefb7aef4e25715 already in progress: creating tablet
W20240502 08:16:59.921598 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 58f538766e934282932fd4ab41b49831 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58f538766e934282932fd4ab41b49831 already in progress: creating tablet
I20240502 08:16:59.921705 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88db59bea69747ca92e66ed0f611db71. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:59.921782 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet f03473f7db3e411186f0db827a6630f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f03473f7db3e411186f0db827a6630f7 already in progress: creating tablet
I20240502 08:16:59.921901 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0561d829e9c54239958fd2ebb2f332a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.904100 25664 raft_consensus.cc:482] T 156c7356ece342aea0d8db667ebad89c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:59.925768 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7855653d11d24e10ac8911f0fe5c98af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.925829 25664 raft_consensus.cc:504] T 156c7356ece342aea0d8db667ebad89c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.926402 25664 leader_election.cc:288] T 156c7356ece342aea0d8db667ebad89c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:59.927557 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7356ece342aea0d8db667ebad89c" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:16:59.929459 24181 raft_consensus.cc:2258] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:16:59.929774 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 859694d7f6114a7f8657836763b3a48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.930107 24181 ts_tablet_manager.cc:1875] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:16:59.930874 24034 tablet_service.cc:1514] Processing DeleteTablet for tablet d9b67c8cd7f34cbba6401da59dec526c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:59.931003 24034 tablet_service.cc:1514] Processing DeleteTablet for tablet a2da8b9d3f374d8a812336057c0ce7d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:16:59.931264 24034 tablet_service.cc:1467] Processing CreateTablet for tablet afeb2d145d2349df9e4a67ce54697ece (DEFAULT_TABLE table=default.Table_0911b0f21cf34f8a80774273c164a5d1 [id=2c385279bc084bf3b33d0212cdf6d272]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.931561 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afeb2d145d2349df9e4a67ce54697ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.934186 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2683333e5cf4718bcf37b3770c3d5e0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:59.937546 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet c4f9e61686dc417bb5a88f96b4231f2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4f9e61686dc417bb5a88f96b4231f2e already in progress: creating tablet
I20240502 08:16:59.937767 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451e499e498c478b8cce9b187362aa4c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:59.937785 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 4bc133adb9a44cc7b56a3746d1eb76af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4bc133adb9a44cc7b56a3746d1eb76af already in progress: creating tablet
W20240502 08:16:59.937937 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9 already in progress: creating tablet
W20240502 08:16:59.938124 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 11ebacd4c41941cd8a6ec634e3aeeaf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11ebacd4c41941cd8a6ec634e3aeeaf5 already in progress: creating tablet
W20240502 08:16:59.938273 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 73d7633fd23541f3a553af008dd6a354 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73d7633fd23541f3a553af008dd6a354 already in progress: creating tablet
W20240502 08:16:59.938393 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 55fb41b8f8f9487a81e98bae9cddaae1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55fb41b8f8f9487a81e98bae9cddaae1 already in progress: creating tablet
W20240502 08:16:59.938517 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e0bd774640464d2a92b2e71126c83883 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0bd774640464d2a92b2e71126c83883 already in progress: creating tablet
I20240502 08:16:59.938869 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet a198c33a3e314b5b81a08244862b281a (table default.Table_22b226147b894b50a865f33b9f41491c [id=337cf81e0af74e8ca4333e4b1bfe4065]) successfully deleted
I20240502 08:16:59.938989 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 001e3f4f72f64dc69dbd521724912f07 (table default.Table_d7cff78803d44e5aba3b50cd8ff11e92 [id=5c21117955d946418c656330984ae747]) successfully deleted
I20240502 08:16:59.942164 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7874af53b4a4120a01e2c92e681ad79. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:59.942117 23861 leader_election.cc:341] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:59.942370 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af68443901cf44ff8e6981e7bfbdcc59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.942552 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ab5772462d49878f54ba515d8f8f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.942683 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2016fc145314ad788b8c8c2c53bf810. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.946750 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb96ad2289754e428a1c109b2d59c53b" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:16:59.947185 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30de4d9cb40e4037b99b71bff5667e8c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:59.947667 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a015eaae830549769bd1561684b04bca consensus state has no leader
I20240502 08:16:59.948107 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1f74b893a9be40b6919d054cc22dbf98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:16:59.948637 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f74b893a9be40b6919d054cc22dbf98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.949693 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd35353b4ea5406f952a5c4941032c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.950457 25664 raft_consensus.cc:482] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:59.950553 25664 raft_consensus.cc:504] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.951020 25664 leader_election.cc:288] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:59.951519 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 8d09ed60e1b24d468311764a6e5da03c (DEFAULT_TABLE table=default.Table_91e1ada585af482cbe82348b4f4c2087 [id=775e82aac71c4a378d29a10e65b5bf87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.951792 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d09ed60e1b24d468311764a6e5da03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.953155 24040 tablet_service.cc:1467] Processing CreateTablet for tablet cb9f8c92b3d34caab0b9f28770ae9c88 (DEFAULT_TABLE table=default.Table_d871e4eda37844d49e6042631c03b1c3 [id=b6761e183a014ff388faed33967b1c0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.953462 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9f8c92b3d34caab0b9f28770ae9c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.953660 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 890face76104472587bfefb04b809bbe. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:16:59.953851 23993 leader_election.cc:341] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 07344b78894c42809391d0fe2a9cd8f7
I20240502 08:16:59.956384 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf9fc9e13d840fdbbf2fe498410c288 (DEFAULT_TABLE table=default.Table_1692280f84c44944b308c28c5918f6f8 [id=a9fd4ae9bc7a408585a109fbdce1b4b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.956555 25664 raft_consensus.cc:482] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:59.956698 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf9fc9e13d840fdbbf2fe498410c288. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.956656 25664 raft_consensus.cc:504] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:16:59.957228 25664 leader_election.cc:288] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:16:59.957656 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7624c3fac7c94786b17ed79f90296a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.957841 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9e9d68e4681a4d8989286fc07bee3791"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:16:59.958228 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet a5768cb928e244d2914f10f4eda3c1cd
W20240502 08:16:59.958283 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e9d68e4681a4d8989286fc07bee3791 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:16:59.958771 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a27bac2d73304d178179313a70c29d86" } new_table_name: "default.Table_64f173e9aa5947069cae45a6cb8dd18f" modify_external_catalogs: true
I20240502 08:16:59.961694 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5fa429485c34d4395ab139c3872610c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.962615 23604 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 55f3f09fc1f94c00a47b9087b98e221d (table default.Table_7c74c34063104d54b3ab77c878892069 [id=106dd88de5eb49e89f0042beb7920736]). Expected version 2 got 0
I20240502 08:16:59.962677 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 50df099e743d49fab20e46a6c66e0b0a (DEFAULT_TABLE table=default.Table_eccfd1a3ea3d4ab286b931f657c62607 [id=0f5008dd89ff4db69275e80653d5d40c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.962957 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50df099e743d49fab20e46a6c66e0b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.962981 23604 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 156c7356ece342aea0d8db667ebad89c (table default.Table_d8c6955b89724769bbfd81db2b5be455 [id=15b52f0688c24df2b6f3fda3732ceeb4]). Expected version 1 got 0
I20240502 08:16:59.963248 23604 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 74b4c563d6bd4a44bb3bab58a3fcdc07 (table default.Table_2ee0f7ddd41e41ea84e25359cd3ed0a4 [id=a9c009feee0e486794a8500121fe5487]). Expected version 2 got 0
I20240502 08:16:59.963476 23604 catalog_manager.cc:5484] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67). New cstate: current_term: 1 leader_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:59.965683 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2613f7e1576d4c399412720492969fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.966032 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ac7f78ed7ac94104a09a418eed600e2d" } new_table_name: "default.Table_68a38c13229040daac749e3816047500" modify_external_catalogs: true
W20240502 08:16:59.967159 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0501575a84ec454780f48574bf3155fe consensus state has no leader
I20240502 08:16:59.967350 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet 9850e84117044d2f9dc82b6314dac2cc from {username='slave'} at 127.0.0.1:36668
I20240502 08:16:59.969988 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8c96a3fc08e94e36af627e4c8dddc7ae" } modify_external_catalogs: true
I20240502 08:16:59.970134 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8c96a3fc08e94e36af627e4c8dddc7ae" } modify_external_catalogs: true
I20240502 08:16:59.971396 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0751e871f6e14c0697044070629e6d26" } modify_external_catalogs: true
I20240502 08:16:59.971535 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0751e871f6e14c0697044070629e6d26" } modify_external_catalogs: true
I20240502 08:16:59.973533 24487 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:59.978607 24022 tablet_service.cc:1467] Processing CreateTablet for tablet adce127b0ca7458cbb836b8a5a159a8a (DEFAULT_TABLE table=default.Table_15060d38700048cf9cd741f97742efe9 [id=4486f7afe1244d1b846fecce2db9e23b]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:16:59.978950 23993 leader_election.cc:341] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: a198c33a3e314b5b81a08244862b281a
W20240502 08:16:59.980098 23861 leader_election.cc:341] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:59.980584 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156c7356ece342aea0d8db667ebad89c" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:16:59.980798 23916 raft_consensus.cc:2379] T 156c7356ece342aea0d8db667ebad89c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 149ce744feab49b5aa5bcaa14bd6aba5 in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:16:59.986101 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet a5768cb928e244d2914f10f4eda3c1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:16:59.992971 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 6b084cd1ba154e98afe828b219a3cbd5 (DEFAULT_TABLE table=default.Table_fa1b2e7c94124643a4e7e8a0d0a4939d [id=990ae264e6234226bfc8307e4bf75e6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.996670 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 2035b4a2c2524c3b86b20ff1bfbd3a0a (DEFAULT_TABLE table=default.Table_249529c4082f44ccba05014ebef7543b [id=0248850c5eb94d9b961561fc9d528922]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:59.978951 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adce127b0ca7458cbb836b8a5a159a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.001051 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b084cd1ba154e98afe828b219a3cbd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.998847 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 2d10ff285c724baeaf766ab2fe3585f3 (DEFAULT_TABLE table=default.Table_465187f6217440139197568c3a016f33 [id=2814526dc6bc4fae936ba1413b5c07aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.001539 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2035b4a2c2524c3b86b20ff1bfbd3a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.001821 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f1ab141585441db46d3027833932b0" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:00.003317 25744 raft_consensus.cc:482] T f1510777d3cb4ae899542796b5a493ad P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:00.003441 25744 raft_consensus.cc:504] T f1510777d3cb4ae899542796b5a493ad P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.003957 25744 leader_election.cc:288] T f1510777d3cb4ae899542796b5a493ad P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:00.004911 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a198c33a3e314b5b81a08244862b281a" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:00.005067 23920 raft_consensus.cc:2454] T a198c33a3e314b5b81a08244862b281a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:17:00.005412 24175 tablet_bootstrap.cc:492] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:00.008719 24021 tablet_service.cc:1467] Processing CreateTablet for tablet 353db059d276404ca9f338ebbca021ac (DEFAULT_TABLE table=default.Table_cbac61918e774f599192caeeaef51577 [id=8a198543bbf1462eaf9c0d1577190ab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.012389 23996 leader_election.cc:302] T 156c7356ece342aea0d8db667ebad89c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:00.012530 24028 tablet_service.cc:1467] Processing CreateTablet for tablet fc36a14cbf0d42b59aad7f8078c71e2f (DEFAULT_TABLE table=default.Table_6dd2e8e9898c4e51945829fcf6afe67c [id=85895c4187254e70b001856e0ea93385]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:17:00.012840 23996 leader_election.cc:341] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Not found: Tablet not found: 07344b78894c42809391d0fe2a9cd8f7
I20240502 08:17:00.012985 23996 leader_election.cc:302] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:00.013238 25423 raft_consensus.cc:3037] T 156c7356ece342aea0d8db667ebad89c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:17:00.013339 23996 leader_election.cc:341] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:00.013406 23996 leader_election.cc:302] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:00.014274 23996 leader_election.cc:302] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:00.014916 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c93c89e1341474eafb6030b95f1c68c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:00.015270 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet c44ce35c463f416794edccc7149aa0cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:00.015611 25423 raft_consensus.cc:2726] T 156c7356ece342aea0d8db667ebad89c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:00.015790 25423 raft_consensus.cc:2726] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:00.017297 25664 raft_consensus.cc:2781] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:00.017385 25664 raft_consensus.cc:482] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:00.017448 25664 raft_consensus.cc:3037] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:00.015933 25423 raft_consensus.cc:2726] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:00.020002 25664 raft_consensus.cc:504] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.020380 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 1140f52809d04a0fb077b819f8f78bed (DEFAULT_TABLE table=default.Table_98f610e5d20b467e8e69ed5915b862ab [id=16086c07bca8466fab8540c1f29ab4d1]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:17:00.021438 23993 leader_election.cc:341] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: a198c33a3e314b5b81a08244862b281a
W20240502 08:17:00.023967 23858 leader_election.cc:341] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: e5f5597402374527bf0e5e50178df713
I20240502 08:17:00.024072 23858 leader_election.cc:302] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:17:00.024433 23858 leader_election.cc:341] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: d6a19bb8705c45c0bfc537d3a6bf8ae2
I20240502 08:17:00.024518 23858 leader_election.cc:302] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:00.024700 25494 raft_consensus.cc:2726] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:00.024693 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet a5768cb928e244d2914f10f4eda3c1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:17:00.024806 23858 leader_election.cc:341] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: c00f29fb76b44d6ea4f76963d2d0751c
I20240502 08:17:00.024842 25494 raft_consensus.cc:2726] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:17:00.024886 23858 leader_election.cc:302] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:00.025049 25494 raft_consensus.cc:2726] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:17:00.025007 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 548839acf20f4bf7a10eee836fc3b0fd (DEFAULT_TABLE table=default.Table_55e1a889107f4aa8894c1ce227ddba9b [id=72da77f9508743889e99be2de1bc5d4b]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:17:00.025175 23858 leader_election.cc:341] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 4b8d81f071d74b218fcfd4f4e8c20bab
I20240502 08:17:00.025240 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d10ff285c724baeaf766ab2fe3585f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.025244 23858 leader_election.cc:302] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:00.025417 25494 raft_consensus.cc:2726] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:17:00.025674 23858 leader_election.cc:341] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: eb96ad2289754e428a1c109b2d59c53b
I20240502 08:17:00.025750 23858 leader_election.cc:302] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:00.026120 25494 raft_consensus.cc:2726] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:00.026736 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1140f52809d04a0fb077b819f8f78bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.027243 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 353db059d276404ca9f338ebbca021ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.027575 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1510777d3cb4ae899542796b5a493ad" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:00.031610 25664 leader_election.cc:288] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:00.032140 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc36a14cbf0d42b59aad7f8078c71e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.032778 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc45553d80d4c7699457e4b62cb3814 (DEFAULT_TABLE table=default.Table_04fa70e3d5d941d5b22e076bc4c3daca [id=e901b74c7964405f84278c0f36edb714]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.033829 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 12dd8b6e55044638a330503c49d4d936 (DEFAULT_TABLE table=default.Table_00868cc9cd204b939d987c1878b0b1e0 [id=d7f235e8e77249f69cb507ba0c275ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.039323 23922 raft_consensus.cc:3037] T b0e8982fe35f47cda376d9646b52e961 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:00.039564 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a198c33a3e314b5b81a08244862b281a" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:00.039738 23916 raft_consensus.cc:3037] T a198c33a3e314b5b81a08244862b281a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:00.032836 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecf6b824ef541de894cb6ec0dabebe5 (DEFAULT_TABLE table=default.Table_1cddae55441d4e559facde8e7046d68a [id=f4914c2b740d47c19a4cc232462a44f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.015874 24038 tablet_service.cc:1467] Processing CreateTablet for tablet ab22a189c3a34136878bef9165fbac09 (DEFAULT_TABLE table=default.Table_6c97a1a9206b45f39095fef784e77788 [id=1aca948dcfce46b78ee9998da5548c90]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:17:00.038239 23723 leader_election.cc:341] T f1510777d3cb4ae899542796b5a493ad P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:00.040367 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 3fcb173255f84a7b94e11f58dbf43c64 (DEFAULT_TABLE table=default.Table_4a7258628dc14959818ff6f14d1059f4 [id=aefcc3e5a49140ca970a10593d8191f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.042423 23916 raft_consensus.cc:2454] T a198c33a3e314b5b81a08244862b281a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:17:00.040689 24027 tablet_service.cc:1467] Processing CreateTablet for tablet af67ce3da5ba4a448eab31d62bcb1e7f (DEFAULT_TABLE table=default.Table_9cff194855ab490b9918ae0224f606ef [id=00b8748213104031933fb14e69a7999a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.044209 24036 tablet_service.cc:1467] Processing CreateTablet for tablet e3f3de0cffda4ee2b43a88470d2e9795 (DEFAULT_TABLE table=default.Table_6723fdf3d7de45429c461fc29b99783f [id=e6f8418bbb104cfbb344a1dbda80d1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.041510 24033 tablet_service.cc:1467] Processing CreateTablet for tablet cc4c8aea662149148032a1f13adddc5f (DEFAULT_TABLE table=default.Table_962cbf6a1b424252a27be241d7c3173f [id=87fa3f49d0654898a2de8ccbebcc3d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.041615 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc45553d80d4c7699457e4b62cb3814. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.042786 23996 leader_election.cc:302] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:00.047379 25012 ts_tablet_manager.cc:1888] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:00.048444 25405 raft_consensus.cc:2781] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:00.048548 25405 raft_consensus.cc:686] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:17:00.048568 25012 log.cc:1196] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/128c88af54d24ffd895e9de19cdb4c8a
I20240502 08:17:00.048774 25405 consensus_queue.cc:237] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.048898 25012 ts_tablet_manager.cc:1909] T 128c88af54d24ffd895e9de19cdb4c8a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:00.049139 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1020) took 3736 ms. Trace:
I20240502 08:17:00.049197 25012 rpcz_store.cc:276] 0502 08:16:56.312551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.312599 (+ 48us) service_pool.cc:225] Handling call
0502 08:16:59.469225 (+3156626us) ts_tablet_manager.cc:1120] Deleting tablet 128c88af54d24ffd895e9de19cdb4c8a
0502 08:16:59.469243 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.957644 (+488401us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.047283 (+ 89639us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.049123 (+ 1840us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3156413}
I20240502 08:17:00.049448 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12dd8b6e55044638a330503c49d4d936. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.050269 25012 tablet_replica.cc:330] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:00.050521 25012 raft_consensus.cc:2229] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.050720 25012 raft_consensus.cc:2258] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.055552 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecf6b824ef541de894cb6ec0dabebe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.057642 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 17027bfd8c7a415591fdc3083bec83bf (DEFAULT_TABLE table=default.Table_463d6892a6624bcaa91f7aba10334f94 [id=26e3cb7e495645b79b8d9b29118633b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.057941 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17027bfd8c7a415591fdc3083bec83bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.058775 24040 tablet_service.cc:1467] Processing CreateTablet for tablet f14ff4b7b7404b819b7feb118efa03af (DEFAULT_TABLE table=default.Table_566356cef3ff47288d74e05caa23af4f [id=435ccc3c80e14a65857a46020c4943b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.061362 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecf6b824ef541de894cb6ec0dabebe5 (DEFAULT_TABLE table=default.Table_1cddae55441d4e559facde8e7046d68a [id=f4914c2b740d47c19a4cc232462a44f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.061746 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecf6b824ef541de894cb6ec0dabebe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.063102 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc45553d80d4c7699457e4b62cb3814 (DEFAULT_TABLE table=default.Table_04fa70e3d5d941d5b22e076bc4c3daca [id=e901b74c7964405f84278c0f36edb714]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.063401 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc45553d80d4c7699457e4b62cb3814. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.064138 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 548839acf20f4bf7a10eee836fc3b0fd (DEFAULT_TABLE table=default.Table_55e1a889107f4aa8894c1ce227ddba9b [id=72da77f9508743889e99be2de1bc5d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.064229 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 0968682f1369466f9773d31563ebce82 (DEFAULT_TABLE table=default.Table_db7ab738f1d7409aa87cb31d155d0047 [id=9f2aad4016344e60ac7b210ad2da04e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.064419 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548839acf20f4bf7a10eee836fc3b0fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.065090 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 12dd8b6e55044638a330503c49d4d936 (DEFAULT_TABLE table=default.Table_00868cc9cd204b939d987c1878b0b1e0 [id=d7f235e8e77249f69cb507ba0c275ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.065394 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12dd8b6e55044638a330503c49d4d936. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.066144 23757 tablet_service.cc:1514] Processing DeleteTablet for tablet a5768cb928e244d2914f10f4eda3c1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:00.080606 23764 tablet_service.cc:1467] Processing CreateTablet for tablet f14ff4b7b7404b819b7feb118efa03af (DEFAULT_TABLE table=default.Table_566356cef3ff47288d74e05caa23af4f [id=435ccc3c80e14a65857a46020c4943b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.081111 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14ff4b7b7404b819b7feb118efa03af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.053853 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 548839acf20f4bf7a10eee836fc3b0fd (DEFAULT_TABLE table=default.Table_55e1a889107f4aa8894c1ce227ddba9b [id=72da77f9508743889e99be2de1bc5d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.064819 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 353db059d276404ca9f338ebbca021ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.066627 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecf6b824ef541de894cb6ec0dabebe5 (DEFAULT_TABLE table=default.Table_1cddae55441d4e559facde8e7046d68a [id=f4914c2b740d47c19a4cc232462a44f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.067201 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 3fcb173255f84a7b94e11f58dbf43c64 (DEFAULT_TABLE table=default.Table_4a7258628dc14959818ff6f14d1059f4 [id=aefcc3e5a49140ca970a10593d8191f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.067502 23889 tablet_service.cc:1467] Processing CreateTablet for tablet af67ce3da5ba4a448eab31d62bcb1e7f (DEFAULT_TABLE table=default.Table_9cff194855ab490b9918ae0224f606ef [id=00b8748213104031933fb14e69a7999a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.068436 23897 tablet_service.cc:1467] Processing CreateTablet for tablet cc4c8aea662149148032a1f13adddc5f (DEFAULT_TABLE table=default.Table_962cbf6a1b424252a27be241d7c3173f [id=87fa3f49d0654898a2de8ccbebcc3d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.081549 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548839acf20f4bf7a10eee836fc3b0fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.081890 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc4c8aea662149148032a1f13adddc5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.082726 24175 tablet_bootstrap.cc:654] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:00.083405 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9f8c92b3d34caab0b9f28770ae9c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.084609 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af67ce3da5ba4a448eab31d62bcb1e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.085656 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecf6b824ef541de894cb6ec0dabebe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.088773 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d10ff285c724baeaf766ab2fe3585f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.068650 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 17027bfd8c7a415591fdc3083bec83bf (DEFAULT_TABLE table=default.Table_463d6892a6624bcaa91f7aba10334f94 [id=26e3cb7e495645b79b8d9b29118633b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.068706 23903 tablet_service.cc:1467] Processing CreateTablet for tablet e3f3de0cffda4ee2b43a88470d2e9795 (DEFAULT_TABLE table=default.Table_6723fdf3d7de45429c461fc29b99783f [id=e6f8418bbb104cfbb344a1dbda80d1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.070850 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1510777d3cb4ae899542796b5a493ad" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:00.096611 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f3de0cffda4ee2b43a88470d2e9795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.096819 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17027bfd8c7a415591fdc3083bec83bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.096230 23750 tablet_service.cc:1467] Processing CreateTablet for tablet e3f3de0cffda4ee2b43a88470d2e9795 (DEFAULT_TABLE table=default.Table_6723fdf3d7de45429c461fc29b99783f [id=e6f8418bbb104cfbb344a1dbda80d1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.101840 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fcb173255f84a7b94e11f58dbf43c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.101881 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f3de0cffda4ee2b43a88470d2e9795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.103006 23754 tablet_service.cc:1467] Processing CreateTablet for tablet cc4c8aea662149148032a1f13adddc5f (DEFAULT_TABLE table=default.Table_962cbf6a1b424252a27be241d7c3173f [id=87fa3f49d0654898a2de8ccbebcc3d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.103368 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc4c8aea662149148032a1f13adddc5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.104391 23759 tablet_service.cc:1467] Processing CreateTablet for tablet af67ce3da5ba4a448eab31d62bcb1e7f (DEFAULT_TABLE table=default.Table_9cff194855ab490b9918ae0224f606ef [id=00b8748213104031933fb14e69a7999a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.104701 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af67ce3da5ba4a448eab31d62bcb1e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.107821 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 17027bfd8c7a415591fdc3083bec83bf (DEFAULT_TABLE table=default.Table_463d6892a6624bcaa91f7aba10334f94 [id=26e3cb7e495645b79b8d9b29118633b4]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:17:00.108050 23993 consensus_peers.cc:485] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5a5a410b6314818b10927e35032f95d. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:00.108180 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17027bfd8c7a415591fdc3083bec83bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.109154 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 3fcb173255f84a7b94e11f58dbf43c64 (DEFAULT_TABLE table=default.Table_4a7258628dc14959818ff6f14d1059f4 [id=aefcc3e5a49140ca970a10593d8191f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.109444 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fcb173255f84a7b94e11f58dbf43c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.071206 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc45553d80d4c7699457e4b62cb3814 (DEFAULT_TABLE table=default.Table_04fa70e3d5d941d5b22e076bc4c3daca [id=e901b74c7964405f84278c0f36edb714]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.109720 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc45553d80d4c7699457e4b62cb3814. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.071707 23895 tablet_service.cc:1467] Processing CreateTablet for tablet f14ff4b7b7404b819b7feb118efa03af (DEFAULT_TABLE table=default.Table_566356cef3ff47288d74e05caa23af4f [id=435ccc3c80e14a65857a46020c4943b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.113900 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14ff4b7b7404b819b7feb118efa03af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.113968 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab22a189c3a34136878bef9165fbac09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.114198 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14ff4b7b7404b819b7feb118efa03af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.115240 25757 consensus_queue.cc:1035] T b0e8982fe35f47cda376d9646b52e961 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.049s
I20240502 08:17:00.117130 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548839acf20f4bf7a10eee836fc3b0fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.117686 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f3de0cffda4ee2b43a88470d2e9795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.118623 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af67ce3da5ba4a448eab31d62bcb1e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.073755 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 12dd8b6e55044638a330503c49d4d936 (DEFAULT_TABLE table=default.Table_00868cc9cd204b939d987c1878b0b1e0 [id=d7f235e8e77249f69cb507ba0c275ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.119966 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12dd8b6e55044638a330503c49d4d936. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.120193 24181 ts_tablet_manager.cc:1888] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.3
I20240502 08:17:00.120447 24181 log.cc:1196] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/b811b81ddf2b4dc9b5e5f80331bff294
I20240502 08:17:00.120759 24181 ts_tablet_manager.cc:1909] T b811b81ddf2b4dc9b5e5f80331bff294 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:00.121589 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1143) took 1005 ms. Trace:
I20240502 08:17:00.121735 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc4c8aea662149148032a1f13adddc5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.122017 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0968682f1369466f9773d31563ebce82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.122207 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fcb173255f84a7b94e11f58dbf43c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.122431 25357 raft_consensus.cc:482] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 149ce744feab49b5aa5bcaa14bd6aba5)
I20240502 08:17:00.122570 25357 raft_consensus.cc:504] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:17:00.123730 23861 leader_election.cc:341] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 33c8131846dc41678827b86e87be18d3
I20240502 08:17:00.123896 25357 leader_election.cc:288] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:00.126331 23897 tablet_service.cc:1467] Processing CreateTablet for tablet 0968682f1369466f9773d31563ebce82 (DEFAULT_TABLE table=default.Table_db7ab738f1d7409aa87cb31d155d0047 [id=9f2aad4016344e60ac7b210ad2da04e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.126649 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0968682f1369466f9773d31563ebce82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.126791 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 11136814c942402d824a096313308960 (DEFAULT_TABLE table=default.Table_c738380f4b114dfeb02b6c85557daed0 [id=46fd6e3d767845c4a18bbdaaa5f88fbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.127166 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11136814c942402d824a096313308960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.129884 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 393bd3b1c4e040a0b05f3e4bfa4fd151 (DEFAULT_TABLE table=default.Table_54d693cd5db04d9999ad9ac4deb13329 [id=5a8474c95751402e9314b366a4b1d455]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.130225 24022 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd01c27f71b41408309c0a5937fff82 (DEFAULT_TABLE table=default.Table_effc9967e9074f64b1fa075d85c98470 [id=e8882955fc11495a9ea5aa258ef580ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.130385 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 5fd449e9a7f84f7880254e859d128755 (DEFAULT_TABLE table=default.Table_465db433d6a5443bac1914682bc40769 [id=54ecfc07bae64409aae7ba8effde4685]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.130565 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 1720b7d3b59e4d52992410ac7a7c0284 (DEFAULT_TABLE table=default.Table_fcfd135764fe45359b74185bcca4bd5a [id=f27692b9a9134a5bb523f16b6c683207]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.130946 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 11136814c942402d824a096313308960 (DEFAULT_TABLE table=default.Table_c738380f4b114dfeb02b6c85557daed0 [id=46fd6e3d767845c4a18bbdaaa5f88fbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.131224 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11136814c942402d824a096313308960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.131191 24021 tablet_service.cc:1467] Processing CreateTablet for tablet 2c325463e66b42eea6e9b954ce69dccb (DEFAULT_TABLE table=default.Table_4f960ddea9ab4d99ba26662e984e4884 [id=78d3f6424870485aa7fc0e1d0eaf2a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.131352 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 393bd3b1c4e040a0b05f3e4bfa4fd151 (DEFAULT_TABLE table=default.Table_54d693cd5db04d9999ad9ac4deb13329 [id=5a8474c95751402e9314b366a4b1d455]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.131525 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd01c27f71b41408309c0a5937fff82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.131633 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393bd3b1c4e040a0b05f3e4bfa4fd151. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.134980 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 44323fea7dcc445abc18993741c3e758 (DEFAULT_TABLE table=default.Table_865f3e827f494b4cb09516c62d3ed201 [id=b0231a17f9054ed7af580cbb342a7f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.135239 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44323fea7dcc445abc18993741c3e758. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.134423 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fd449e9a7f84f7880254e859d128755. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.135973 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1720b7d3b59e4d52992410ac7a7c0284. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.588940 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e11d405a2e4e518a10bf383e9ae6c4" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:00.137012 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 44323fea7dcc445abc18993741c3e758 (DEFAULT_TABLE table=default.Table_865f3e827f494b4cb09516c62d3ed201 [id=b0231a17f9054ed7af580cbb342a7f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.137315 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44323fea7dcc445abc18993741c3e758. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.137665 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393bd3b1c4e040a0b05f3e4bfa4fd151. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.138784 23884 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:33614 (request call id 175) took 1079 ms. Trace:
I20240502 08:17:00.138854 23884 rpcz_store.cc:276] 0502 08:16:59.059524 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.060295 (+ 771us) service_pool.cc:225] Handling call
0502 08:16:59.560214 (+499919us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x616000152854. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:59.811695 (+251481us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x6160000a8754. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:16:59.858622 (+ 46927us) spinlock_profiling.cc:243] Waited 43.7 ms on lock 0x616000143e54. stack: 00007fb1f5611756 00007fb1f5611510 00007fb1f464448d 0000000000955804 00000000009557cd 000000000095519a 00007fb200e4fb00 00007fb204b60b7f 00007fb1ff2e38e6 00007fb1ff2e3615 00007fb1fc31904e 00007fb1fc317f2e 00007fb1fc31e405 00007fb1fc31ed2f 00007fb1fc31eb58 0000000000a60bcc
0502 08:17:00.138767 (+280145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":874680192}
I20240502 08:17:00.140224 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c325463e66b42eea6e9b954ce69dccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.141350 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd01c27f71b41408309c0a5937fff82 (DEFAULT_TABLE table=default.Table_effc9967e9074f64b1fa075d85c98470 [id=e8882955fc11495a9ea5aa258ef580ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.141650 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd01c27f71b41408309c0a5937fff82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.145673 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 2f34a6f104dd467b98254cd77a059bfc (DEFAULT_TABLE table=default.Table_686a41d458d04191b29159f0630ae285 [id=8cf73339e012452c981c5a1e38f33db5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.145879 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 5f9ba58808ec45c4bf053076c8c1e1a1 (DEFAULT_TABLE table=default.Table_581ed0c699e94b5bb670d72c63f40197 [id=505abcea8ca245e5b906f0ce588d8963]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.146027 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f34a6f104dd467b98254cd77a059bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.146198 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f9ba58808ec45c4bf053076c8c1e1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.146170 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 2c325463e66b42eea6e9b954ce69dccb (DEFAULT_TABLE table=default.Table_4f960ddea9ab4d99ba26662e984e4884 [id=78d3f6424870485aa7fc0e1d0eaf2a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.147943 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 5fd449e9a7f84f7880254e859d128755 (DEFAULT_TABLE table=default.Table_465db433d6a5443bac1914682bc40769 [id=54ecfc07bae64409aae7ba8effde4685]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.152531 23752 tablet_service.cc:1467] Processing CreateTablet for tablet 393bd3b1c4e040a0b05f3e4bfa4fd151 (DEFAULT_TABLE table=default.Table_54d693cd5db04d9999ad9ac4deb13329 [id=5a8474c95751402e9314b366a4b1d455]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.152603 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 1720b7d3b59e4d52992410ac7a7c0284 (DEFAULT_TABLE table=default.Table_fcfd135764fe45359b74185bcca4bd5a [id=f27692b9a9134a5bb523f16b6c683207]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.152839 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393bd3b1c4e040a0b05f3e4bfa4fd151. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.152940 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 11136814c942402d824a096313308960 (DEFAULT_TABLE table=default.Table_c738380f4b114dfeb02b6c85557daed0 [id=46fd6e3d767845c4a18bbdaaa5f88fbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.153227 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11136814c942402d824a096313308960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.153435 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 2f34a6f104dd467b98254cd77a059bfc (DEFAULT_TABLE table=default.Table_686a41d458d04191b29159f0630ae285 [id=8cf73339e012452c981c5a1e38f33db5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.153723 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 44323fea7dcc445abc18993741c3e758 (DEFAULT_TABLE table=default.Table_865f3e827f494b4cb09516c62d3ed201 [id=b0231a17f9054ed7af580cbb342a7f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.153915 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f34a6f104dd467b98254cd77a059bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.154006 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44323fea7dcc445abc18993741c3e758. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.154273 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1140f52809d04a0fb077b819f8f78bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.154314 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 5955411f06e04737bbe0f0d8efc77102 (DEFAULT_TABLE table=default.Table_7aa8726847d34e7facc024d5a2287fcf [id=bb6f41053e584d728714119bfc144587]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.155710 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c325463e66b42eea6e9b954ce69dccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.156004 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 0968682f1369466f9773d31563ebce82 (DEFAULT_TABLE table=default.Table_db7ab738f1d7409aa87cb31d155d0047 [id=9f2aad4016344e60ac7b210ad2da04e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.156296 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0968682f1369466f9773d31563ebce82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.156522 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 903df34c76554ae49e1b39e04fb4f824 (DEFAULT_TABLE table=default.Table_e29ac7dfdff84ee7b3859d45e8ff4af6 [id=af9c5adc9c294e848b7929e53d1d7ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.156996 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 5955411f06e04737bbe0f0d8efc77102 (DEFAULT_TABLE table=default.Table_7aa8726847d34e7facc024d5a2287fcf [id=bb6f41053e584d728714119bfc144587]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.157191 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5955411f06e04737bbe0f0d8efc77102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.157373 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fd449e9a7f84f7880254e859d128755. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.157373 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 2c325463e66b42eea6e9b954ce69dccb (DEFAULT_TABLE table=default.Table_4f960ddea9ab4d99ba26662e984e4884 [id=78d3f6424870485aa7fc0e1d0eaf2a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.157706 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c325463e66b42eea6e9b954ce69dccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.159646 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd01c27f71b41408309c0a5937fff82 (DEFAULT_TABLE table=default.Table_effc9967e9074f64b1fa075d85c98470 [id=e8882955fc11495a9ea5aa258ef580ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.159920 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd01c27f71b41408309c0a5937fff82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.160462 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 1720b7d3b59e4d52992410ac7a7c0284 (DEFAULT_TABLE table=default.Table_fcfd135764fe45359b74185bcca4bd5a [id=f27692b9a9134a5bb523f16b6c683207]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.160738 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1720b7d3b59e4d52992410ac7a7c0284. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.162525 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1720b7d3b59e4d52992410ac7a7c0284. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.165302 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 3ccaa109132844d4843d08b108b2960c (DEFAULT_TABLE table=default.Table_4610f864dbdd41378b26279a24da77f9 [id=0357f7b063e64b88a93bf4bbaee37b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.167768 24022 tablet_service.cc:1467] Processing CreateTablet for tablet 903df34c76554ae49e1b39e04fb4f824 (DEFAULT_TABLE table=default.Table_e29ac7dfdff84ee7b3859d45e8ff4af6 [id=af9c5adc9c294e848b7929e53d1d7ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.168987 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903df34c76554ae49e1b39e04fb4f824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.169395 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 9734918c88074e6c928a9510107a2e64 (DEFAULT_TABLE table=default.Table_c9c42f6f968c415d8b407324dc712f58 [id=8f4d51fb48434f3795e44aa4e00273fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.169390 23888 tablet_service.cc:1467] Processing CreateTablet for tablet 9734918c88074e6c928a9510107a2e64 (DEFAULT_TABLE table=default.Table_c9c42f6f968c415d8b407324dc712f58 [id=8f4d51fb48434f3795e44aa4e00273fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.170120 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9734918c88074e6c928a9510107a2e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.170553 24025 tablet_service.cc:1467] Processing CreateTablet for tablet 36cd7b4a84d64808ba123951350400a9 (DEFAULT_TABLE table=default.Table_8fb3758a4a62482bbcbcba9b7b659274 [id=870ff97b4edb469bb3a454b8c9e82e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.172020 24021 tablet_service.cc:1467] Processing CreateTablet for tablet 1498089eb91d470a8360c61968814fe1 (DEFAULT_TABLE table=default.Table_9b3b4ecdf5194a3a9c896c851c1e2eb9 [id=c27025abdf1943c2bd3dd150b5078c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.173506 24028 tablet_service.cc:1467] Processing CreateTablet for tablet a0cd92cdb18442b48f61de8c597dc4c8 (DEFAULT_TABLE table=default.Table_1fa659ecf3e84aea89cf0a2ef3266bae [id=f9732af223f5435390613b0bac265f6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.173875 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 5f9ba58808ec45c4bf053076c8c1e1a1 (DEFAULT_TABLE table=default.Table_581ed0c699e94b5bb670d72c63f40197 [id=505abcea8ca245e5b906f0ce588d8963]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.175923 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5955411f06e04737bbe0f0d8efc77102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.177845 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 1498089eb91d470a8360c61968814fe1 (DEFAULT_TABLE table=default.Table_9b3b4ecdf5194a3a9c896c851c1e2eb9 [id=c27025abdf1943c2bd3dd150b5078c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.178201 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1498089eb91d470a8360c61968814fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.183041 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f9ba58808ec45c4bf053076c8c1e1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.183490 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 3ccaa109132844d4843d08b108b2960c (DEFAULT_TABLE table=default.Table_4610f864dbdd41378b26279a24da77f9 [id=0357f7b063e64b88a93bf4bbaee37b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.187248 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ccaa109132844d4843d08b108b2960c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.189667 24025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36cd7b4a84d64808ba123951350400a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.808216 25738 raft_consensus.cc:482] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:00.189805 25738 raft_consensus.cc:504] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.194206 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ccaa109132844d4843d08b108b2960c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.195291 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9734918c88074e6c928a9510107a2e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.188735 23889 tablet_service.cc:1467] Processing CreateTablet for tablet a0cd92cdb18442b48f61de8c597dc4c8 (DEFAULT_TABLE table=default.Table_1fa659ecf3e84aea89cf0a2ef3266bae [id=f9732af223f5435390613b0bac265f6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.196852 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0cd92cdb18442b48f61de8c597dc4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.200033 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1498089eb91d470a8360c61968814fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.186931 25012 ts_tablet_manager.cc:1875] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.203965 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903df34c76554ae49e1b39e04fb4f824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.208477 25776 tablet.cc:1722] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:17:00.213016 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 36cd7b4a84d64808ba123951350400a9 (DEFAULT_TABLE table=default.Table_8fb3758a4a62482bbcbcba9b7b659274 [id=870ff97b4edb469bb3a454b8c9e82e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.214416 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36cd7b4a84d64808ba123951350400a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.219549 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0cd92cdb18442b48f61de8c597dc4c8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:00.224979 23723 leader_election.cc:341] T f1510777d3cb4ae899542796b5a493ad P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:00.225340 23723 leader_election.cc:302] T f1510777d3cb4ae899542796b5a493ad P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:00.226765 25744 raft_consensus.cc:2726] T f1510777d3cb4ae899542796b5a493ad P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:00.230497 25744 consensus_queue.cc:1035] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:00.231325 23723 leader_election.cc:341] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:00.231580 23723 leader_election.cc:302] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:00.231123 25012 ts_tablet_manager.cc:1888] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:00.233206 25012 log.cc:1196] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/c00f29fb76b44d6ea4f76963d2d0751c
I20240502 08:17:00.233548 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 1a0249ded3344e49acc07f0b549f079f (DEFAULT_TABLE table=default.Table_c54c945517b5468fb9f9b25ed02691d0 [id=e6b2bd3371b8437c9808b71c1b7d8a79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.239913 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a0249ded3344e49acc07f0b549f079f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.243031 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 71adaca77b3a42b4966130de6741f5ed (DEFAULT_TABLE table=default.Table_a676dcd2d5234d0f896dd378de2b9f34 [id=ef1f2adeaca94cd58632608d748f3042]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.245363 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 0aababc690904da096423fc910fce2ab (DEFAULT_TABLE table=default.Table_33f2ee613cae4e848bb987e9041391c1 [id=911f3bae460f408f810631fd79fc956a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.291162 25012 ts_tablet_manager.cc:1909] T c00f29fb76b44d6ea4f76963d2d0751c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:00.291522 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1016) took 3979 ms. Trace:
I20240502 08:17:00.291594 25012 rpcz_store.cc:276] 0502 08:16:56.312191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.312725 (+ 534us) service_pool.cc:225] Handling call
0502 08:17:00.050228 (+3737503us) ts_tablet_manager.cc:1120] Deleting tablet c00f29fb76b44d6ea4f76963d2d0751c
0502 08:17:00.050236 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.204278 (+154042us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.231034 (+ 26756us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.291501 (+ 60467us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3737354}
I20240502 08:17:00.291883 25012 tablet_replica.cc:330] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:00.292088 25012 raft_consensus.cc:2229] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.292240 25012 raft_consensus.cc:2258] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.292436 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd9e392bd0b4e678f441bba813b4549 (DEFAULT_TABLE table=default.Table_d1a3f21401914ea6b4e2309ee2f71089 [id=83728cf1bef04ce0a10a01dec7369721]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.292760 25012 ts_tablet_manager.cc:1875] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.292851 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd9e392bd0b4e678f441bba813b4549. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.293108 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 71adaca77b3a42b4966130de6741f5ed (DEFAULT_TABLE table=default.Table_a676dcd2d5234d0f896dd378de2b9f34 [id=ef1f2adeaca94cd58632608d748f3042]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.293678 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 0aababc690904da096423fc910fce2ab (DEFAULT_TABLE table=default.Table_33f2ee613cae4e848bb987e9041391c1 [id=911f3bae460f408f810631fd79fc956a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.294273 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 0c54690dc3244942a80bde80449f8daf (DEFAULT_TABLE table=default.Table_52d62af96d2c404982cd9ba08010884b [id=c329a7bb9be2410a94c89854f5ea2c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.295804 23898 tablet_service.cc:1467] Processing CreateTablet for tablet b3b0c0dcb2184ad39b984f718f8a2082 (DEFAULT_TABLE table=default.Table_d718a36474964fc09d2813f1210c5ce2 [id=d9f38d2a946c498f86eddd3d3277464d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.297518 25357 raft_consensus.cc:482] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:00.297645 25357 raft_consensus.cc:504] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.298286 23888 tablet_service.cc:1467] Processing CreateTablet for tablet b6bb4a4ccd574f4cadc0b9d81ef4dd5b (DEFAULT_TABLE table=default.Table_d731f4904f8649bdbc1369d357489e48 [id=7cefc2e9af9d4923ac5bf936a3faee02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.298609 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 128c88af54d24ffd895e9de19cdb4c8a (table default.Table_d67be3894f494223a45d8c5a2c3e6113 [id=f7982b1d892942648cf3fca657c2668c]) successfully deleted
I20240502 08:17:00.298754 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 208c582f82e4433196ac1b7a5c1dfac8 (DEFAULT_TABLE table=default.Table_1e603637c1ff4cb5855d8b4c7bbb2ea2 [id=a12056ce8ccf4b7580d15fe5ca8fb8d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.293982 23890 tablet_service.cc:1467] Processing CreateTablet for tablet c977d023d3a241a2b82dbb5ab416ce5c (DEFAULT_TABLE table=default.Table_df77dff2af6544ff9fa5702b6768b0d8 [id=cd108e9e1daf45bda9587efde242c72e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.299659 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 1a0249ded3344e49acc07f0b549f079f (DEFAULT_TABLE table=default.Table_c54c945517b5468fb9f9b25ed02691d0 [id=e6b2bd3371b8437c9808b71c1b7d8a79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.306149 23604 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 55f3f09fc1f94c00a47b9087b98e221d (table default.Table_7c74c34063104d54b3ab77c878892069 [id=106dd88de5eb49e89f0042beb7920736]). Expected version 2 got 0
W20240502 08:17:00.306491 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 66ea6743a3a44f9b95e1ca3d50a3dd40 consensus state has no leader
I20240502 08:17:00.309046 23604 master_service.cc:946] ReplaceTablet: received request to replace tablet 87a3df4fbb224dd0b4ec6141584d374b from {username='slave'} at 127.0.0.1:36668
W20240502 08:17:00.309689 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 665bbee2d15b4112b651844431535d0e consensus state has no leader
I20240502 08:17:00.232024 25744 raft_consensus.cc:2726] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:17:00.328914 23723 consensus_peers.cc:485] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:00.351689 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 5fd449e9a7f84f7880254e859d128755 (DEFAULT_TABLE table=default.Table_465db433d6a5443bac1914682bc40769 [id=54ecfc07bae64409aae7ba8effde4685]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.352259 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fd449e9a7f84f7880254e859d128755. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.363690 25494 raft_consensus.cc:482] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 149ce744feab49b5aa5bcaa14bd6aba5)
I20240502 08:17:00.363870 25494 raft_consensus.cc:504] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.364557 25494 leader_election.cc:288] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:17:00.365743 23861 leader_election.cc:341] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: df08f0d3af4c4c7e8abce656714b1b7f
I20240502 08:17:00.372787 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 5f9ba58808ec45c4bf053076c8c1e1a1 (DEFAULT_TABLE table=default.Table_581ed0c699e94b5bb670d72c63f40197 [id=505abcea8ca245e5b906f0ce588d8963]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.373308 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f9ba58808ec45c4bf053076c8c1e1a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.375559 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 2f34a6f104dd467b98254cd77a059bfc (DEFAULT_TABLE table=default.Table_686a41d458d04191b29159f0630ae285 [id=8cf73339e012452c981c5a1e38f33db5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.376147 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f34a6f104dd467b98254cd77a059bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.383927 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 3ccaa109132844d4843d08b108b2960c (DEFAULT_TABLE table=default.Table_4610f864dbdd41378b26279a24da77f9 [id=0357f7b063e64b88a93bf4bbaee37b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.384379 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ccaa109132844d4843d08b108b2960c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.399924 23752 tablet_service.cc:1467] Processing CreateTablet for tablet 903df34c76554ae49e1b39e04fb4f824 (DEFAULT_TABLE table=default.Table_e29ac7dfdff84ee7b3859d45e8ff4af6 [id=af9c5adc9c294e848b7929e53d1d7ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.400319 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903df34c76554ae49e1b39e04fb4f824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.422098 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 9734918c88074e6c928a9510107a2e64 (DEFAULT_TABLE table=default.Table_c9c42f6f968c415d8b407324dc712f58 [id=8f4d51fb48434f3795e44aa4e00273fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.444602 25738 leader_election.cc:288] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:00.451040 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9734918c88074e6c928a9510107a2e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.460952 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71adaca77b3a42b4966130de6741f5ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.461141 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aababc690904da096423fc910fce2ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.461061 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 02ed4b6aa1884d08ae08d8dcb9be38bb (DEFAULT_TABLE table=default.Table_70dac5e6bd8546ec9021c0142d578880 [id=2d18e395f9ed4f3591aa8569710d206b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.461828 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c54690dc3244942a80bde80449f8daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.461949 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 208c582f82e4433196ac1b7a5c1dfac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.462173 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ed4b6aa1884d08ae08d8dcb9be38bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.462330 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6bb4a4ccd574f4cadc0b9d81ef4dd5b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:00.462847 23861 consensus_peers.cc:485] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:00.463577 25357 leader_election.cc:288] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:00.463896 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a0249ded3344e49acc07f0b549f079f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.464172 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c977d023d3a241a2b82dbb5ab416ce5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.464864 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b0c0dcb2184ad39b984f718f8a2082. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:00.465107 23861 leader_election.cc:341] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 592ab8523b8d479d9caead321d5888c3
I20240502 08:17:00.443385 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 36cd7b4a84d64808ba123951350400a9 (DEFAULT_TABLE table=default.Table_8fb3758a4a62482bbcbcba9b7b659274 [id=870ff97b4edb469bb3a454b8c9e82e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.465747 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36cd7b4a84d64808ba123951350400a9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:00.468042 23858 leader_election.cc:341] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 33c8131846dc41678827b86e87be18d3
I20240502 08:17:00.468190 23858 leader_election.cc:302] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:00.468442 25357 raft_consensus.cc:2726] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:17:00.451376 23751 tablet_service.cc:1467] Processing CreateTablet for tablet a0cd92cdb18442b48f61de8c597dc4c8 (DEFAULT_TABLE table=default.Table_1fa659ecf3e84aea89cf0a2ef3266bae [id=f9732af223f5435390613b0bac265f6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.449785 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 1498089eb91d470a8360c61968814fe1 (DEFAULT_TABLE table=default.Table_9b3b4ecdf5194a3a9c896c851c1e2eb9 [id=c27025abdf1943c2bd3dd150b5078c48]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.451676 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 1a0249ded3344e49acc07f0b549f079f (DEFAULT_TABLE table=default.Table_c54c945517b5468fb9f9b25ed02691d0 [id=e6b2bd3371b8437c9808b71c1b7d8a79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.451984 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 71adaca77b3a42b4966130de6741f5ed (DEFAULT_TABLE table=default.Table_a676dcd2d5234d0f896dd378de2b9f34 [id=ef1f2adeaca94cd58632608d748f3042]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.483414 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 0aababc690904da096423fc910fce2ab (DEFAULT_TABLE table=default.Table_33f2ee613cae4e848bb987e9041391c1 [id=911f3bae460f408f810631fd79fc956a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.487816 25782 tablet.cc:1722] T 7ed919909289410e867881baab841f53 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:17:00.496919 23760 tablet_service.cc:1467] Processing CreateTablet for tablet b3b0c0dcb2184ad39b984f718f8a2082 (DEFAULT_TABLE table=default.Table_d718a36474964fc09d2813f1210c5ce2 [id=d9f38d2a946c498f86eddd3d3277464d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.501889 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 208c582f82e4433196ac1b7a5c1dfac8 (DEFAULT_TABLE table=default.Table_1e603637c1ff4cb5855d8b4c7bbb2ea2 [id=a12056ce8ccf4b7580d15fe5ca8fb8d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.503237 24175 tablet_bootstrap.cc:492] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:00.503405 24175 ts_tablet_manager.cc:1367] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.498s user 0.004s sys 0.005s
I20240502 08:17:00.504155 24175 raft_consensus.cc:348] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:00.504426 24175 raft_consensus.cc:374] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:00.504515 24175 raft_consensus.cc:729] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:00.504731 24175 consensus_queue.cc:260] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:00.505487 24175 ts_tablet_manager.cc:1398] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:00.507354 24175 tablet_bootstrap.cc:492] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:00.514312 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a0249ded3344e49acc07f0b549f079f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.514513 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0cd92cdb18442b48f61de8c597dc4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.515086 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1498089eb91d470a8360c61968814fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.522559 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 5955411f06e04737bbe0f0d8efc77102 (DEFAULT_TABLE table=default.Table_7aa8726847d34e7facc024d5a2287fcf [id=bb6f41053e584d728714119bfc144587]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.523535 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71adaca77b3a42b4966130de6741f5ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.524330 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b0c0dcb2184ad39b984f718f8a2082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.492723 23750 tablet_service.cc:1467] Processing CreateTablet for tablet c977d023d3a241a2b82dbb5ab416ce5c (DEFAULT_TABLE table=default.Table_df77dff2af6544ff9fa5702b6768b0d8 [id=cd108e9e1daf45bda9587efde242c72e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.529388 25012 ts_tablet_manager.cc:1888] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:00.529493 25012 log.cc:1196] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/5d04312bd81f46ed8c49c42b1af1aec9
I20240502 08:17:00.531164 25012 ts_tablet_manager.cc:1909] T 5d04312bd81f46ed8c49c42b1af1aec9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:00.531457 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1017) took 4219 ms. Trace:
I20240502 08:17:00.531543 25012 rpcz_store.cc:276] 0502 08:16:56.312270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.312912 (+ 642us) service_pool.cc:225] Handling call
0502 08:17:00.291839 (+3978927us) ts_tablet_manager.cc:1120] Deleting tablet 5d04312bd81f46ed8c49c42b1af1aec9
0502 08:17:00.291847 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.526772 (+234925us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.529317 (+ 2545us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.531441 (+ 2124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3978798}
I20240502 08:17:00.531833 25012 tablet_replica.cc:330] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:00.533133 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd9e392bd0b4e678f441bba813b4549 (DEFAULT_TABLE table=default.Table_d1a3f21401914ea6b4e2309ee2f71089 [id=83728cf1bef04ce0a10a01dec7369721]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.526096 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 0c54690dc3244942a80bde80449f8daf (DEFAULT_TABLE table=default.Table_52d62af96d2c404982cd9ba08010884b [id=c329a7bb9be2410a94c89854f5ea2c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.535815 25012 raft_consensus.cc:2229] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.535996 25012 raft_consensus.cc:2258] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.536156 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c54690dc3244942a80bde80449f8daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.536522 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aababc690904da096423fc910fce2ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.536505 25012 ts_tablet_manager.cc:1875] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.538177 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_f92f900ce0114ea6ac0e01a049d206e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:00.538250 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5c50ed61204d4468990c37540b069f08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:00.538664 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f92f900ce0114ea6ac0e01a049d206e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:17:00.538686 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c50ed61204d4468990c37540b069f08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:00.539247 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5d3aa312157c4ed7baafaa4864bcc205"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:00.539599 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d3aa312157c4ed7baafaa4864bcc205 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:00.544296 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 9850e84117044d2f9dc82b6314dac2cc
I20240502 08:17:00.544658 23601 catalog_manager.cc:6554] ReplaceTablet: tablet 9850e84117044d2f9dc82b6314dac2cc deleted and replaced by tablet 0cff5d6a37ac45a5a9d3cea7a493e601
I20240502 08:17:00.551916 23764 tablet_service.cc:1467] Processing CreateTablet for tablet b6bb4a4ccd574f4cadc0b9d81ef4dd5b (DEFAULT_TABLE table=default.Table_d731f4904f8649bdbc1369d357489e48 [id=7cefc2e9af9d4923ac5bf936a3faee02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.248283 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71adaca77b3a42b4966130de6741f5ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.561161 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aababc690904da096423fc910fce2ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.251461 24031 tablet_service.cc:1467] Processing CreateTablet for tablet c977d023d3a241a2b82dbb5ab416ce5c (DEFAULT_TABLE table=default.Table_df77dff2af6544ff9fa5702b6768b0d8 [id=cd108e9e1daf45bda9587efde242c72e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.251653 24022 tablet_service.cc:1467] Processing CreateTablet for tablet 208c582f82e4433196ac1b7a5c1dfac8 (DEFAULT_TABLE table=default.Table_1e603637c1ff4cb5855d8b4c7bbb2ea2 [id=a12056ce8ccf4b7580d15fe5ca8fb8d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.565977 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 208c582f82e4433196ac1b7a5c1dfac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.571779 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c977d023d3a241a2b82dbb5ab416ce5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.259423 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd9e392bd0b4e678f441bba813b4549 (DEFAULT_TABLE table=default.Table_d1a3f21401914ea6b4e2309ee2f71089 [id=83728cf1bef04ce0a10a01dec7369721]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.260254 24036 tablet_service.cc:1467] Processing CreateTablet for tablet b6bb4a4ccd574f4cadc0b9d81ef4dd5b (DEFAULT_TABLE table=default.Table_d731f4904f8649bdbc1369d357489e48 [id=7cefc2e9af9d4923ac5bf936a3faee02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.261978 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 02ed4b6aa1884d08ae08d8dcb9be38bb (DEFAULT_TABLE table=default.Table_70dac5e6bd8546ec9021c0142d578880 [id=2d18e395f9ed4f3591aa8569710d206b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.289198 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 79294fbe9daa4789b270c0a0ea9ed37f (DEFAULT_TABLE table=default.Table_138a88014a504ac68fde53841c4779a0 [id=de2ae245c3d245d5a2f2f39c8fc64149]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:17:00.297287 23996 consensus_peers.cc:485] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Couldn't send request to peer 5b89a1f0485a44b7ac1710e2369a809b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:00.573918 23996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 276653us
I20240502 08:17:00.445087 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f2e98ef2884b05994d02114e992e80" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:00.457621 24173 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.638s user 0.024s sys 0.067s
W20240502 08:17:00.574101 24173 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.755s user 0.024s sys 0.067s
I20240502 08:17:00.475266 25776 tablet.cc:1722] T 7ed919909289410e867881baab841f53 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:17:00.240527 24023 tablet_service.cc:1467] Processing CreateTablet for tablet b3b0c0dcb2184ad39b984f718f8a2082 (DEFAULT_TABLE table=default.Table_d718a36474964fc09d2813f1210c5ce2 [id=d9f38d2a946c498f86eddd3d3277464d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.574280 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79294fbe9daa4789b270c0a0ea9ed37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.574482 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b0c0dcb2184ad39b984f718f8a2082. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.575371 24173 tablet_bootstrap.cc:492] T 8910bc67c6234a538c8160e7c020653c P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:00.575510 24173 ts_tablet_manager.cc:1367] T 8910bc67c6234a538c8160e7c020653c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 1.128s user 0.027s sys 0.068s
I20240502 08:17:00.576503 24173 raft_consensus.cc:348] T 8910bc67c6234a538c8160e7c020653c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:00.576722 24173 raft_consensus.cc:374] T 8910bc67c6234a538c8160e7c020653c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:00.576781 24173 raft_consensus.cc:729] T 8910bc67c6234a538c8160e7c020653c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:00.576988 24173 consensus_queue.cc:260] T 8910bc67c6234a538c8160e7c020653c P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:00.577457 24173 ts_tablet_manager.cc:1398] T 8910bc67c6234a538c8160e7c020653c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:00.577970 24173 tablet_bootstrap.cc:492] T e0b0d4871dd24607854100a4ef762e25 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:00.121714 24181 rpcz_store.cc:276] 0502 08:16:59.116264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.116423 (+ 159us) service_pool.cc:225] Handling call
0502 08:16:59.921209 (+804786us) ts_tablet_manager.cc:1120] Deleting tablet b811b81ddf2b4dc9b5e5f80331bff294
0502 08:16:59.921217 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:16:59.976566 (+ 55349us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.120118 (+143552us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.121507 (+ 1389us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":804581}
I20240502 08:17:00.578294 24181 tablet_replica.cc:330] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:00.578456 24181 raft_consensus.cc:2229] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:00.578758 24181 raft_consensus.cc:2258] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.579260 24181 ts_tablet_manager.cc:1875] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.579447 24173 tablet_bootstrap.cc:654] T e0b0d4871dd24607854100a4ef762e25 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:00.580010 24175 tablet_bootstrap.cc:654] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:00.581913 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 79294fbe9daa4789b270c0a0ea9ed37f (DEFAULT_TABLE table=default.Table_138a88014a504ac68fde53841c4779a0 [id=de2ae245c3d245d5a2f2f39c8fc64149]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.582262 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79294fbe9daa4789b270c0a0ea9ed37f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:00.583098 23723 leader_election.cc:341] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:00.587284 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ed4b6aa1884d08ae08d8dcb9be38bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.588223 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 02ed4b6aa1884d08ae08d8dcb9be38bb (DEFAULT_TABLE table=default.Table_70dac5e6bd8546ec9021c0142d578880 [id=2d18e395f9ed4f3591aa8569710d206b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.589749 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f2e98ef2884b05994d02114e992e80" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:00.590144 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6bb4a4ccd574f4cadc0b9d81ef4dd5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.591563 25738 consensus_queue.cc:1035] T 156c7356ece342aea0d8db667ebad89c P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:00.593156 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd9e392bd0b4e678f441bba813b4549. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.594177 25012 ts_tablet_manager.cc:1888] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:00.594341 25012 log.cc:1196] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/955e7bd2216646c6aef5acce8d5b0d77
I20240502 08:17:00.594738 25012 ts_tablet_manager.cc:1909] T 955e7bd2216646c6aef5acce8d5b0d77 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:00.595113 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1018) took 4282 ms. Trace:
I20240502 08:17:00.597187 25012 rpcz_store.cc:276] 0502 08:16:56.312353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.313070 (+ 717us) service_pool.cc:225] Handling call
0502 08:17:00.531791 (+4218721us) ts_tablet_manager.cc:1120] Deleting tablet 955e7bd2216646c6aef5acce8d5b0d77
0502 08:17:00.531800 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.567328 (+ 35528us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.594103 (+ 26775us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.595095 (+ 992us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4218589}
I20240502 08:17:00.597420 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 79294fbe9daa4789b270c0a0ea9ed37f (DEFAULT_TABLE table=default.Table_138a88014a504ac68fde53841c4779a0 [id=de2ae245c3d245d5a2f2f39c8fc64149]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.598198 25012 tablet_replica.cc:330] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:00.598373 25012 raft_consensus.cc:2229] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.598505 25012 raft_consensus.cc:2258] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.598969 25012 ts_tablet_manager.cc:1875] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.602319 24175 tablet_bootstrap.cc:492] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:00.602497 24175 ts_tablet_manager.cc:1367] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.097s user 0.000s sys 0.003s
I20240502 08:17:00.603174 24175 raft_consensus.cc:348] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:00.603413 24175 raft_consensus.cc:374] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:00.603516 24175 raft_consensus.cc:729] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:00.603757 24175 consensus_queue.cc:260] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:00.604296 24175 ts_tablet_manager.cc:1398] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:00.604563 24175 tablet_bootstrap.cc:492] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:00.605857 24175 tablet_bootstrap.cc:654] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:00.606804 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 9850e84117044d2f9dc82b6314dac2cc with delete_type TABLET_DATA_DELETED (replaced by tablet 0cff5d6a37ac45a5a9d3cea7a493e601) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:00.607898 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c977d023d3a241a2b82dbb5ab416ce5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.608584 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6bb4a4ccd574f4cadc0b9d81ef4dd5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.614243 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd9e392bd0b4e678f441bba813b4549. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.614976 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5955411f06e04737bbe0f0d8efc77102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.615969 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 0c54690dc3244942a80bde80449f8daf (DEFAULT_TABLE table=default.Table_52d62af96d2c404982cd9ba08010884b [id=c329a7bb9be2410a94c89854f5ea2c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.616447 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c54690dc3244942a80bde80449f8daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.621459 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 208c582f82e4433196ac1b7a5c1dfac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.622452 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_076bc414d7814432bfd813a959dc2730" } new_table_name: "default.Table_84794e5b0f1b41aeb9a61102c97135ce" modify_external_catalogs: true
I20240502 08:17:00.625229 25012 ts_tablet_manager.cc:1888] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:00.626394 24173 tablet_bootstrap.cc:492] T e0b0d4871dd24607854100a4ef762e25 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:00.626382 24175 tablet_bootstrap.cc:492] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:00.626579 24175 ts_tablet_manager.cc:1367] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.001s
I20240502 08:17:00.626690 24173 ts_tablet_manager.cc:1367] T e0b0d4871dd24607854100a4ef762e25 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.004s
I20240502 08:17:00.627049 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 156c7356ece342aea0d8db667ebad89c (table default.Table_d8c6955b89724769bbfd81db2b5be455 [id=15b52f0688c24df2b6f3fda3732ceeb4]). Expected version 1 got 0
I20240502 08:17:00.627281 24175 raft_consensus.cc:348] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.627411 24173 raft_consensus.cc:348] T e0b0d4871dd24607854100a4ef762e25 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.627569 24175 raft_consensus.cc:374] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:00.627662 24173 raft_consensus.cc:374] T e0b0d4871dd24607854100a4ef762e25 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:00.627672 24175 raft_consensus.cc:729] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:00.627838 24173 raft_consensus.cc:729] T e0b0d4871dd24607854100a4ef762e25 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:00.627905 24175 consensus_queue.cc:260] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.628165 24173 consensus_queue.cc:260] T e0b0d4871dd24607854100a4ef762e25 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.628571 24175 ts_tablet_manager.cc:1398] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:00.628854 24175 tablet_bootstrap.cc:492] T 7fe583cbf4af493e82d3ec0508ea0f0b P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:00.630115 24175 tablet_bootstrap.cc:654] T 7fe583cbf4af493e82d3ec0508ea0f0b P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:00.630786 25012 log.cc:1196] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/45f02efa659e457ab57bc32ed3f66bb9
I20240502 08:17:00.631166 25012 ts_tablet_manager.cc:1909] T 45f02efa659e457ab57bc32ed3f66bb9 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:00.631465 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1019) took 4319 ms. Trace:
I20240502 08:17:00.631511 24173 ts_tablet_manager.cc:1398] T e0b0d4871dd24607854100a4ef762e25 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:17:00.631844 24173 tablet_bootstrap.cc:492] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:00.631949 25012 rpcz_store.cc:276] 0502 08:16:56.312473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.313317 (+ 844us) service_pool.cc:225] Handling call
0502 08:17:00.598159 (+4284842us) ts_tablet_manager.cc:1120] Deleting tablet 45f02efa659e457ab57bc32ed3f66bb9
0502 08:17:00.598167 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.622343 (+ 24176us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.625154 (+ 2811us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.631449 (+ 6295us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4284699}
I20240502 08:17:00.633343 24173 tablet_bootstrap.cc:654] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:00.633589 25012 tablet_replica.cc:330] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:00.633600 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79294fbe9daa4789b270c0a0ea9ed37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.633809 25012 raft_consensus.cc:2229] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.633982 25012 pending_rounds.cc:62] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b: Trying to abort 1 pending ops.
I20240502 08:17:00.634099 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ed4b6aa1884d08ae08d8dcb9be38bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.634065 25012 pending_rounds.cc:68] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156493874683904 op_type: NO_OP noop_request { }
I20240502 08:17:00.634240 25012 raft_consensus.cc:2866] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:00.634356 25012 raft_consensus.cc:2258] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.634855 25012 ts_tablet_manager.cc:1875] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.636739 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 1b175e1b55bf4102804666b6fffccead
I20240502 08:17:00.638293 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5556d39f8aa64dd8bfb68ace348e406d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:00.638782 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5556d39f8aa64dd8bfb68ace348e406d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:17:00.641917 23858 leader_election.cc:341] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: df08f0d3af4c4c7e8abce656714b1b7f
W20240502 08:17:00.642000 23723 leader_election.cc:341] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:00.642024 23858 leader_election.cc:302] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:00.642086 23723 leader_election.cc:302] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:00.642222 25649 raft_consensus.cc:2726] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:17:00.642370 23858 leader_election.cc:341] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 592ab8523b8d479d9caead321d5888c3
I20240502 08:17:00.642465 23858 leader_election.cc:302] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:00.642673 25649 raft_consensus.cc:2726] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:17:00.642856 25649 consensus_queue.cc:1035] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:00.642990 25357 raft_consensus.cc:482] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:00.643112 25357 raft_consensus.cc:504] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:00.643674 25357 leader_election.cc:288] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:00.645422 25745 consensus_queue.cc:1035] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:00.646881 24175 tablet_bootstrap.cc:492] T 7fe583cbf4af493e82d3ec0508ea0f0b P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:00.647049 24175 ts_tablet_manager.cc:1367] T 7fe583cbf4af493e82d3ec0508ea0f0b P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240502 08:17:00.647729 24175 raft_consensus.cc:348] T 7fe583cbf4af493e82d3ec0508ea0f0b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.647958 24175 raft_consensus.cc:374] T 7fe583cbf4af493e82d3ec0508ea0f0b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:00.648049 24175 raft_consensus.cc:729] T 7fe583cbf4af493e82d3ec0508ea0f0b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:00.648288 24175 consensus_queue.cc:260] T 7fe583cbf4af493e82d3ec0508ea0f0b P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.648893 24175 ts_tablet_manager.cc:1398] T 7fe583cbf4af493e82d3ec0508ea0f0b P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:00.649185 24175 tablet_bootstrap.cc:492] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:00.644316 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0501575a84ec454780f48574bf3155fe" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:00.649595 24174 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.823s user 0.010s sys 0.009s
W20240502 08:17:00.649693 24174 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.823s user 0.010s sys 0.010s
I20240502 08:17:00.650496 24175 tablet_bootstrap.cc:654] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:17:00.651124 23861 leader_election.cc:341] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:00.651150 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet c00f29fb76b44d6ea4f76963d2d0751c (table default.Table_c2a25d0d51cd40769e5045ff80c9c3fc [id=a1004106109d47e7a1f9c6337480d3c0]) successfully deleted
I20240502 08:17:00.651952 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 5d04312bd81f46ed8c49c42b1af1aec9 (table default.Table_9741459a96e649fb8151cc28b642523e [id=2b9169f285754bfda0f4f8dc7f5cb653]) successfully deleted
I20240502 08:17:00.652760 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 955e7bd2216646c6aef5acce8d5b0d77 (table default.Table_b3278c5d6d0b4593ac33bc9977f5c3b1 [id=e86031550cec46a793785c0b3859fa92]) successfully deleted
I20240502 08:17:00.652980 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 45f02efa659e457ab57bc32ed3f66bb9 (table default.Table_54b9e5042d19438ab2690d06917ec10c [id=27f4152aaee24a3bad52e0390b8fa649]) successfully deleted
I20240502 08:17:00.655232 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b175e1b55bf4102804666b6fffccead with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:00.662878 25012 ts_tablet_manager.cc:1888] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:00.663069 25012 log.cc:1196] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/33c8131846dc41678827b86e87be18d3
I20240502 08:17:00.663445 25012 ts_tablet_manager.cc:1909] T 33c8131846dc41678827b86e87be18d3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:00.663933 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1021) took 4351 ms. Trace:
I20240502 08:17:00.664395 25012 rpcz_store.cc:276] 0502 08:16:56.312672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.313487 (+ 815us) service_pool.cc:225] Handling call
0502 08:17:00.633533 (+4320046us) ts_tablet_manager.cc:1120] Deleting tablet 33c8131846dc41678827b86e87be18d3
0502 08:17:00.633541 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.640126 (+ 6585us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.662779 (+ 22653us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.663913 (+ 1134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4319884}
I20240502 08:17:00.665071 25012 tablet_replica.cc:330] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:00.666044 25012 raft_consensus.cc:2229] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.668653 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 49acc2af1a064dd2b0b60120beed69ba
I20240502 08:17:00.669402 23608 master_service.cc:946] ReplaceTablet: received request to replace tablet 5027fa2c80b544b8b1e2a4694bec7454 from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:00.669756 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 14f54d6ded85434da35c1dfcff52763a
I20240502 08:17:00.665799 24175 tablet_bootstrap.cc:492] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:00.673753 24175 ts_tablet_manager.cc:1367] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240502 08:17:00.674599 24175 raft_consensus.cc:348] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.674877 24175 raft_consensus.cc:374] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:00.674937 24175 raft_consensus.cc:729] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:00.675088 25012 raft_consensus.cc:2258] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.675117 24175 consensus_queue.cc:260] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.675571 24175 ts_tablet_manager.cc:1398] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:00.675805 25012 ts_tablet_manager.cc:1875] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.675947 24175 tablet_bootstrap.cc:492] T 8910bc67c6234a538c8160e7c020653c P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:00.677445 24175 tablet_bootstrap.cc:654] T 8910bc67c6234a538c8160e7c020653c P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:00.683593 25012 ts_tablet_manager.cc:1888] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:00.683787 25012 log.cc:1196] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/d6a19bb8705c45c0bfc537d3a6bf8ae2
I20240502 08:17:00.684202 25012 ts_tablet_manager.cc:1909] T d6a19bb8705c45c0bfc537d3a6bf8ae2 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
W20240502 08:17:00.692705 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 50703.6us
I20240502 08:17:00.692963 25738 raft_consensus.cc:2726] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:00.693524 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1015) took 4381 ms. Trace:
I20240502 08:17:00.698329 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0501575a84ec454780f48574bf3155fe" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:00.698509 23783 raft_consensus.cc:2454] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:00.699410 23858 leader_election.cc:302] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:00.699819 25357 raft_consensus.cc:2781] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:00.699898 25357 raft_consensus.cc:482] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:00.699951 25357 raft_consensus.cc:3037] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:17:00.702203 23597 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 48637.4us
I20240502 08:17:00.702306 25357 raft_consensus.cc:504] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:00.702863 25357 leader_election.cc:288] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:00.703483 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0501575a84ec454780f48574bf3155fe" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:17:00.703998 23861 leader_election.cc:341] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:00.693712 25012 rpcz_store.cc:276] 0502 08:16:56.312093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.313886 (+ 1793us) service_pool.cc:225] Handling call
0502 08:17:00.665020 (+4351134us) ts_tablet_manager.cc:1120] Deleting tablet d6a19bb8705c45c0bfc537d3a6bf8ae2
0502 08:17:00.665029 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.680176 (+ 15147us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.683507 (+ 3331us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.693506 (+ 9999us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":635,"tablet-delete.queue_time_us":4351002}
I20240502 08:17:00.705480 25012 tablet_replica.cc:330] T 7fe583cbf4af493e82d3ec0508ea0f0b P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:00.706064 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0501575a84ec454780f48574bf3155fe" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:00.706225 23783 raft_consensus.cc:3037] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:17:00.706733 25743 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.005s sys 0.013s
I20240502 08:17:00.705864 25012 raft_consensus.cc:2229] T 7fe583cbf4af493e82d3ec0508ea0f0b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.707252 25012 raft_consensus.cc:2258] T 7fe583cbf4af493e82d3ec0508ea0f0b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.708550 25012 ts_tablet_manager.cc:1875] T 7fe583cbf4af493e82d3ec0508ea0f0b P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.709198 24174 tablet_bootstrap.cc:492] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:00.709290 24181 ts_tablet_manager.cc:1888] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:00.709328 24174 ts_tablet_manager.cc:1367] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 1.119s user 0.011s sys 0.010s
I20240502 08:17:00.709400 24181 log.cc:1196] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/ec10ff7251a5442a8ecc1881b34db74d
I20240502 08:17:00.709735 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 49acc2af1a064dd2b0b60120beed69ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:00.709830 24181 ts_tablet_manager.cc:1909] T ec10ff7251a5442a8ecc1881b34db74d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:00.710610 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 14f54d6ded85434da35c1dfcff52763a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:00.710639 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1144) took 1371 ms. Trace:
I20240502 08:17:00.710705 24181 rpcz_store.cc:276] 0502 08:16:59.339501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.340464 (+ 963us) service_pool.cc:225] Handling call
0502 08:17:00.578247 (+1237783us) ts_tablet_manager.cc:1120] Deleting tablet ec10ff7251a5442a8ecc1881b34db74d
0502 08:17:00.578255 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.683814 (+105559us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.709191 (+ 25377us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.710609 (+ 1418us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1237605}
I20240502 08:17:00.711592 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1145) took 1287 ms. Trace:
I20240502 08:17:00.711668 24181 rpcz_store.cc:276] 0502 08:16:59.423929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.424218 (+ 289us) service_pool.cc:225] Handling call
0502 08:17:00.711464 (+1287246us) ts_tablet_manager.cc:1120] Deleting tablet 0dae6a96776c477fb0ada8226d68ea25
0502 08:17:00.711476 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.711578 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1287056}
W20240502 08:17:00.706821 25743 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.680s user 0.005s sys 0.013s
I20240502 08:17:00.718065 25012 ts_tablet_manager.cc:1888] T 7fe583cbf4af493e82d3ec0508ea0f0b P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:00.718226 25012 log.cc:1196] T 7fe583cbf4af493e82d3ec0508ea0f0b P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/7fe583cbf4af493e82d3ec0508ea0f0b
I20240502 08:17:00.718551 25012 ts_tablet_manager.cc:1909] T 7fe583cbf4af493e82d3ec0508ea0f0b P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:00.718851 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1014) took 4440 ms. Trace:
I20240502 08:17:00.721329 25012 rpcz_store.cc:276] 0502 08:16:56.278860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.278937 (+ 77us) service_pool.cc:225] Handling call
0502 08:17:00.705437 (+4426500us) ts_tablet_manager.cc:1120] Deleting tablet 7fe583cbf4af493e82d3ec0508ea0f0b
0502 08:17:00.705446 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.712014 (+ 6568us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.717980 (+ 5966us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.718837 (+ 857us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4385742}
I20240502 08:17:00.721783 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1022) took 4067 ms. Trace:
I20240502 08:17:00.721910 25012 rpcz_store.cc:276] 0502 08:16:56.654551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.654617 (+ 66us) service_pool.cc:225] Handling call
0502 08:17:00.721675 (+4067058us) ts_tablet_manager.cc:1120] Deleting tablet 548ab3f9d0d84527b1092ae715ec2bdd
0502 08:17:00.721683 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.721766 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4059996}
I20240502 08:17:00.722347 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1023) took 3959 ms. Trace:
I20240502 08:17:00.722582 25012 rpcz_store.cc:276] 0502 08:16:56.763075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.763466 (+ 391us) service_pool.cc:225] Handling call
0502 08:17:00.722235 (+3958769us) ts_tablet_manager.cc:1120] Deleting tablet 96013fa1fa014b1891eb621cc3a81cdc
0502 08:17:00.722244 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.722333 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3958613}
I20240502 08:17:00.728511 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet b811b81ddf2b4dc9b5e5f80331bff294 (table default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418]) successfully deleted
I20240502 08:17:00.729621 25012 tablet_replica.cc:330] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:00.730034 25012 raft_consensus.cc:2229] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.730238 25012 raft_consensus.cc:2258] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.730902 25012 ts_tablet_manager.cc:1875] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.731825 24174 raft_consensus.cc:348] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.732004 24174 raft_consensus.cc:374] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:00.732069 24174 raft_consensus.cc:729] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
W20240502 08:17:00.732237 23786 thread.cc:636] tablet-copy [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.006s sys 0.015s
W20240502 08:17:00.732354 23786 thread.cc:602] tablet-copy [worker] (thread pool) Time spent starting thread: real 0.619s user 0.006s sys 0.015s
I20240502 08:17:00.738081 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet ec10ff7251a5442a8ecc1881b34db74d (table default.Table_236150d3903d437d95f67b0edec36091 [id=1d7a43116fe74c35b505eccd165f99cd]) successfully deleted
I20240502 08:17:00.738523 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1146) took 1311 ms. Trace:
I20240502 08:17:00.738863 24174 consensus_queue.cc:260] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.739428 24174 ts_tablet_manager.cc:1398] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.030s user 0.001s sys 0.001s
I20240502 08:17:00.739773 24174 tablet_bootstrap.cc:492] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:00.740413 23783 raft_consensus.cc:2454] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:00.740856 23858 leader_election.cc:302] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
W20240502 08:17:00.741052 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 0dae6a96776c477fb0ada8226d68ea25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0dae6a96776c477fb0ada8226d68ea25 already in progress: creating tablet
I20240502 08:17:00.741150 25357 raft_consensus.cc:2781] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:00.741254 25357 raft_consensus.cc:686] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:17:00.741472 25357 consensus_queue.cc:237] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:00.741672 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet a5a5a410b6314818b10927e35032f95d with delete_type TABLET_DATA_DELETED (replaced by tablet 92b830e128ac439eba286f29dbb20ee4) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:00.738674 24181 rpcz_store.cc:276] 0502 08:16:59.427340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.427402 (+ 62us) service_pool.cc:225] Handling call
0502 08:17:00.738373 (+1310971us) ts_tablet_manager.cc:1120] Deleting tablet 22bc17792cf040748ab672cfc6451e3c
0502 08:17:00.738387 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.738499 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1310756}
I20240502 08:17:00.742430 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1147) took 1285 ms. Trace:
I20240502 08:17:00.742508 24181 rpcz_store.cc:276] 0502 08:16:59.456674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.456985 (+ 311us) service_pool.cc:225] Handling call
0502 08:17:00.742327 (+1285342us) ts_tablet_manager.cc:1120] Deleting tablet 4e32319fa9244ed2a323b2b3f5940748
0502 08:17:00.742335 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.742416 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":919191}
I20240502 08:17:00.743057 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet a198c33a3e314b5b81a08244862b281a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:00.743484 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 07344b78894c42809391d0fe2a9cd8f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:52 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:00.743736 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1167) took 1206 ms. Trace:
I20240502 08:17:00.744130 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f1ab141585441db46d3027833932b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:00.743798 24181 rpcz_store.cc:276] 0502 08:16:59.536821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.823169 (+286348us) service_pool.cc:225] Handling call
0502 08:17:00.743650 (+920481us) ts_tablet_manager.cc:1120] Deleting tablet fd32de329009465c89a4917e2716d3e9
0502 08:17:00.743658 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.743724 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":920343}
I20240502 08:17:00.744585 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet a198c33a3e314b5b81a08244862b281a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:00.745676 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 9850e84117044d2f9dc82b6314dac2cc with delete_type TABLET_DATA_DELETED (replaced by tablet 0cff5d6a37ac45a5a9d3cea7a493e601) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:00.746069 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1168) took 1176 ms. Trace:
I20240502 08:17:00.746477 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 9850e84117044d2f9dc82b6314dac2cc with delete_type TABLET_DATA_DELETED (replaced by tablet 0cff5d6a37ac45a5a9d3cea7a493e601) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:00.746908 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b175e1b55bf4102804666b6fffccead with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:00.747179 24181 rpcz_store.cc:276] 0502 08:16:59.569762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.823324 (+253562us) service_pool.cc:225] Handling call
0502 08:17:00.745975 (+922651us) ts_tablet_manager.cc:1120] Deleting tablet 2a3e66ef4a914f009cca6d418f825d4e
0502 08:17:00.745984 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.746054 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":922544}
I20240502 08:17:00.747664 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b175e1b55bf4102804666b6fffccead with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:00.748100 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 49acc2af1a064dd2b0b60120beed69ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:00.748521 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1148) took 1288 ms. Trace:
I20240502 08:17:00.749173 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 14f54d6ded85434da35c1dfcff52763a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:00.748863 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 49acc2af1a064dd2b0b60120beed69ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:00.749608 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 14f54d6ded85434da35c1dfcff52763a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:00.749835 24181 rpcz_store.cc:276] 0502 08:16:59.460559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.460614 (+ 55us) service_pool.cc:225] Handling call
0502 08:17:00.748429 (+1287815us) ts_tablet_manager.cc:1120] Deleting tablet 7ae9c817c7be42da8c689267c7eb5a06
0502 08:17:00.748437 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.748510 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":918282}
I20240502 08:17:00.750363 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 33c8131846dc41678827b86e87be18d3 (table default.Table_bd9708a3da8240b4aa9444c8e9ed30d2 [id=5849a3c8b28e47439b73e2137187fac9]) successfully deleted
I20240502 08:17:00.750998 24181 tablet_replica.cc:330] T ee022e8e86f8477a8fa1e58d52644b03 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:00.751286 24181 raft_consensus.cc:2229] T ee022e8e86f8477a8fa1e58d52644b03 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.751626 24174 tablet_bootstrap.cc:654] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:00.751847 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet d6a19bb8705c45c0bfc537d3a6bf8ae2 (table default.Table_c67035cbec4e42109cb3d466e0a6ad18 [id=b2cb0ed2926d4be7ade0790203b57e2e]) successfully deleted
I20240502 08:17:00.751657 24181 raft_consensus.cc:2258] T ee022e8e86f8477a8fa1e58d52644b03 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.752314 25012 ts_tablet_manager.cc:1888] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:00.752489 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 7fe583cbf4af493e82d3ec0508ea0f0b (table default.Table_3c1db2ea8e3d49efa8439ddf3a30734c [id=8dc5018c97434885a14739188b199dc0]) successfully deleted
I20240502 08:17:00.752534 25012 log.cc:1196] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/edd7244b9f5741a8aa1c6835fe544676
W20240502 08:17:00.752700 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 548ab3f9d0d84527b1092ae715ec2bdd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 548ab3f9d0d84527b1092ae715ec2bdd already in progress: creating tablet
W20240502 08:17:00.752875 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 96013fa1fa014b1891eb621cc3a81cdc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96013fa1fa014b1891eb621cc3a81cdc already in progress: creating tablet
W20240502 08:17:00.753074 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 22bc17792cf040748ab672cfc6451e3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22bc17792cf040748ab672cfc6451e3c already in progress: creating tablet
W20240502 08:17:00.753233 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 4e32319fa9244ed2a323b2b3f5940748 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e32319fa9244ed2a323b2b3f5940748 already in progress: creating tablet
I20240502 08:17:00.753368 25012 ts_tablet_manager.cc:1909] T edd7244b9f5741a8aa1c6835fe544676 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:00.754101 24181 ts_tablet_manager.cc:1875] T ee022e8e86f8477a8fa1e58d52644b03 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.755051 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1024) took 3991 ms. Trace:
I20240502 08:17:00.755156 25012 rpcz_store.cc:276] 0502 08:16:56.763164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.763648 (+ 484us) service_pool.cc:225] Handling call
0502 08:17:00.729558 (+3965910us) ts_tablet_manager.cc:1120] Deleting tablet edd7244b9f5741a8aa1c6835fe544676
0502 08:17:00.729582 (+ 24us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.735070 (+ 5488us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.752048 (+ 16978us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.755037 (+ 2989us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3965813}
W20240502 08:17:00.755537 24029 consensus_meta.cc:287] T 451e499e498c478b8cce9b187362aa4c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent flushing consensus metadata: real 0.815s user 0.021s sys 0.004s
I20240502 08:17:00.755614 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1025) took 3992 ms. Trace:
I20240502 08:17:00.755733 25012 rpcz_store.cc:276] 0502 08:16:56.763255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.763761 (+ 506us) service_pool.cc:225] Handling call
0502 08:17:00.755523 (+3991762us) ts_tablet_manager.cc:1120] Deleting tablet 92b830e128ac439eba286f29dbb20ee4
0502 08:17:00.755531 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.755601 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3991672}
I20240502 08:17:00.756219 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1026) took 3992 ms. Trace:
I20240502 08:17:00.756330 25012 rpcz_store.cc:276] 0502 08:16:56.763417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:56.763879 (+ 462us) service_pool.cc:225] Handling call
0502 08:17:00.756123 (+3992244us) ts_tablet_manager.cc:1120] Deleting tablet e5de31a3db014615989d9e018521bfc9
0502 08:17:00.756132 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.756205 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3992136}
I20240502 08:17:00.756763 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1027) took 3513 ms. Trace:
I20240502 08:17:00.759284 25012 rpcz_store.cc:276] 0502 08:16:57.243255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.243313 (+ 58us) service_pool.cc:225] Handling call
0502 08:17:00.756674 (+3513361us) ts_tablet_manager.cc:1120] Deleting tablet 01143d14aca64adb846c8141ae41e6ac
0502 08:17:00.756684 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.756750 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3513160}
I20240502 08:17:00.759742 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1028) took 3508 ms. Trace:
I20240502 08:17:00.759872 25012 rpcz_store.cc:276] 0502 08:16:57.251603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.251658 (+ 55us) service_pool.cc:225] Handling call
0502 08:17:00.759639 (+3507981us) ts_tablet_manager.cc:1120] Deleting tablet 521581fe87e547ce816b599578bddeea
0502 08:17:00.759647 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.759727 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3507808}
I20240502 08:17:00.760313 25012 tablet_replica.cc:330] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:00.760632 25012 raft_consensus.cc:2229] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.760812 25012 raft_consensus.cc:2258] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.761312 25012 ts_tablet_manager.cc:1875] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:17:00.762066 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet fd32de329009465c89a4917e2716d3e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd32de329009465c89a4917e2716d3e9 already in progress: creating tablet
I20240502 08:17:00.762933 24029 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51782 (request call id 1159) took 1244 ms. Trace:
W20240502 08:17:00.763270 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 2a3e66ef4a914f009cca6d418f825d4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a3e66ef4a914f009cca6d418f825d4e already in progress: creating tablet
I20240502 08:17:00.763628 24029 rpcz_store.cc:276] 0502 08:16:59.518588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.519786 (+ 1198us) service_pool.cc:225] Handling call
0502 08:16:59.937618 (+417832us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 08:16:59.937655 (+ 37us) ts_tablet_manager.cc:655] Creating new metadata...
0502 08:16:59.940092 (+ 2437us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.762805 (+822713us) tablet_replica.cc:205] Creating consensus instance
0502 08:17:00.762918 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 08:17:00.763978 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 7ae9c817c7be42da8c689267c7eb5a06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ae9c817c7be42da8c689267c7eb5a06 already in progress: creating tablet
I20240502 08:17:00.764667 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet edd7244b9f5741a8aa1c6835fe544676 (table default.Table_4c6989f7e6bb4e99b447a3c1188f5b92 [id=9775ed3c642a4d7e81d5abc48acca163]) successfully deleted
I20240502 08:17:00.767460 25012 ts_tablet_manager.cc:1888] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240502 08:17:00.770164 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 92b830e128ac439eba286f29dbb20ee4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92b830e128ac439eba286f29dbb20ee4 already in progress: creating tablet
W20240502 08:17:00.770368 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e5de31a3db014615989d9e018521bfc9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5de31a3db014615989d9e018521bfc9 already in progress: creating tablet
W20240502 08:17:00.770565 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 01143d14aca64adb846c8141ae41e6ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01143d14aca64adb846c8141ae41e6ac already in progress: creating tablet
W20240502 08:17:00.770732 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 521581fe87e547ce816b599578bddeea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 521581fe87e547ce816b599578bddeea already in progress: creating tablet
I20240502 08:17:00.773648 25012 log.cc:1196] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/4b8d81f071d74b218fcfd4f4e8c20bab
I20240502 08:17:00.774070 25012 ts_tablet_manager.cc:1909] T 4b8d81f071d74b218fcfd4f4e8c20bab P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:00.774856 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1029) took 3427 ms. Trace:
I20240502 08:17:00.776281 25012 rpcz_store.cc:276] 0502 08:16:57.346953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.347209 (+ 256us) service_pool.cc:225] Handling call
0502 08:17:00.760266 (+3413057us) ts_tablet_manager.cc:1120] Deleting tablet 4b8d81f071d74b218fcfd4f4e8c20bab
0502 08:17:00.760276 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.764288 (+ 4012us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.767375 (+ 3087us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.774836 (+ 7461us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3412797}
I20240502 08:17:00.776850 25012 tablet_replica.cc:330] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:00.777176 25012 raft_consensus.cc:2229] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.777758 25012 raft_consensus.cc:2258] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.778699 25012 ts_tablet_manager.cc:1875] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:17:00.777417 25752 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.018s sys 0.006s
W20240502 08:17:00.781790 25752 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.608s user 0.018s sys 0.006s
I20240502 08:17:00.783630 25357 raft_consensus.cc:482] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:00.783757 25357 raft_consensus.cc:504] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:00.784317 25357 leader_election.cc:288] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:17:00.785210 23861 leader_election.cc:341] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: fbef606ce7ac499d9bd84b49d804621c
I20240502 08:17:00.785596 25795 tablet.cc:1722] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:17:00.787111 25743 raft_consensus.cc:482] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:00.787238 25743 raft_consensus.cc:504] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.787729 25743 leader_election.cc:288] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:00.788400 25357 raft_consensus.cc:482] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:00.788329 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45aa0dd26524e0fba27e4eaf17d704e" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:00.788511 25357 raft_consensus.cc:504] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.789014 25357 leader_election.cc:288] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:17:00.788995 23723 leader_election.cc:341] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:00.789182 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45aa0dd26524e0fba27e4eaf17d704e" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:17:00.789846 23723 leader_election.cc:341] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:00.789938 23723 leader_election.cc:302] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
W20240502 08:17:00.790192 23861 leader_election.cc:341] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: ba3aa2621fdb43fd8299f81af486ef1e
I20240502 08:17:00.790238 25743 raft_consensus.cc:2726] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:00.790428 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 4b8d81f071d74b218fcfd4f4e8c20bab (table default.Table_dbe385cc3f6b48d39bee298ca9490248 [id=aa825e5ca73c4dafa9585126d5c48df7]) successfully deleted
W20240502 08:17:00.790479 23858 leader_election.cc:341] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: ba3aa2621fdb43fd8299f81af486ef1e
I20240502 08:17:00.790581 23858 leader_election.cc:302] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:00.790750 25357 raft_consensus.cc:2726] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:17:00.791044 24181 ts_tablet_manager.cc:1888] T ee022e8e86f8477a8fa1e58d52644b03 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:00.791143 24181 log.cc:1196] T ee022e8e86f8477a8fa1e58d52644b03 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/ee022e8e86f8477a8fa1e58d52644b03
I20240502 08:17:00.791450 24181 ts_tablet_manager.cc:1909] T ee022e8e86f8477a8fa1e58d52644b03 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
W20240502 08:17:00.793093 23858 leader_election.cc:341] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: fbef606ce7ac499d9bd84b49d804621c
I20240502 08:17:00.793187 23858 leader_election.cc:302] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:00.793396 25357 raft_consensus.cc:2726] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:00.795043 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1150) took 1329 ms. Trace:
I20240502 08:17:00.795135 24181 rpcz_store.cc:276] 0502 08:16:59.465646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.465848 (+ 202us) service_pool.cc:225] Handling call
0502 08:17:00.750958 (+1285110us) ts_tablet_manager.cc:1120] Deleting tablet ee022e8e86f8477a8fa1e58d52644b03
0502 08:17:00.750967 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.767052 (+ 16085us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.790969 (+ 23917us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.795025 (+ 4056us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":919767}
I20240502 08:17:00.795449 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1149) took 1330 ms. Trace:
I20240502 08:17:00.795518 24181 rpcz_store.cc:276] 0502 08:16:59.465475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.467268 (+ 1793us) service_pool.cc:225] Handling call
0502 08:17:00.795378 (+1328110us) ts_tablet_manager.cc:1120] Deleting tablet 592eeecbf5514215b6464bd030081e4b
0502 08:17:00.795385 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.795435 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":963728}
I20240502 08:17:00.795835 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1151) took 1327 ms. Trace:
I20240502 08:17:00.795933 24181 rpcz_store.cc:276] 0502 08:16:59.468088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.468136 (+ 48us) service_pool.cc:225] Handling call
0502 08:17:00.795746 (+1327610us) ts_tablet_manager.cc:1120] Deleting tablet b1a2bcc6937c4a95b71103dbf8738702
0502 08:17:00.795754 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.795822 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":963671}
I20240502 08:17:00.796233 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1175) took 1149 ms. Trace:
I20240502 08:17:00.796254 25012 ts_tablet_manager.cc:1888] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:00.796444 25012 log.cc:1196] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/25c089f0c1b14e419f250d03d4eaad12
I20240502 08:17:00.796780 25012 ts_tablet_manager.cc:1909] T 25c089f0c1b14e419f250d03d4eaad12 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:00.798177 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet d423a92750c94ee1a08688d60dd2168e
I20240502 08:17:00.799284 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1030) took 3452 ms. Trace:
I20240502 08:17:00.799401 25012 rpcz_store.cc:276] 0502 08:16:57.347128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.347530 (+ 402us) service_pool.cc:225] Handling call
0502 08:17:00.776794 (+3429264us) ts_tablet_manager.cc:1120] Deleting tablet 25c089f0c1b14e419f250d03d4eaad12
0502 08:17:00.776806 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.788229 (+ 11423us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.796173 (+ 7944us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.799270 (+ 3097us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3429111}
I20240502 08:17:00.799872 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1031) took 3427 ms. Trace:
I20240502 08:17:00.799999 25012 rpcz_store.cc:276] 0502 08:16:57.372821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.372874 (+ 53us) service_pool.cc:225] Handling call
0502 08:17:00.799778 (+3426904us) ts_tablet_manager.cc:1120] Deleting tablet 01143d14aca64adb846c8141ae41e6ac
0502 08:17:00.799786 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.799858 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3268085}
I20240502 08:17:00.800426 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1032) took 3425 ms. Trace:
I20240502 08:17:00.801609 25012 rpcz_store.cc:276] 0502 08:16:57.375109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.375156 (+ 47us) service_pool.cc:225] Handling call
0502 08:17:00.800338 (+3425182us) ts_tablet_manager.cc:1120] Deleting tablet 82c49bba3df548558123344a13c946ee
0502 08:17:00.800346 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.800411 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3268303}
I20240502 08:17:00.801774 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 25c089f0c1b14e419f250d03d4eaad12 (table default.Table_94efd28ada5b4024ab65e4d5feda0b38 [id=15b42ca7619645c8ac3cff50044f069b]) successfully deleted
W20240502 08:17:00.801982 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 01143d14aca64adb846c8141ae41e6ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01143d14aca64adb846c8141ae41e6ac already in progress: creating tablet
I20240502 08:17:00.802009 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1033) took 3425 ms. Trace:
I20240502 08:17:00.802135 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet ee022e8e86f8477a8fa1e58d52644b03 (table default.Table_d6f5ca82488a40a69121f1b779d8f874 [id=0422356f66a74ad29c2e13e1ad36b654]) successfully deleted
I20240502 08:17:00.802153 25012 rpcz_store.cc:276] 0502 08:16:57.376190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.376253 (+ 63us) service_pool.cc:225] Handling call
0502 08:17:00.801922 (+3425669us) ts_tablet_manager.cc:1120] Deleting tablet 18d82e75f2624e6782f55987cea223cb
0502 08:17:00.801930 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.801994 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3269768}
W20240502 08:17:00.802292 23597 catalog_manager.cc:4797] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 592eeecbf5514215b6464bd030081e4b because the tablet was not found. No further retry: Not found: Tablet not found: 592eeecbf5514215b6464bd030081e4b
W20240502 08:17:00.802464 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet b1a2bcc6937c4a95b71103dbf8738702 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1a2bcc6937c4a95b71103dbf8738702 already in progress: creating tablet
I20240502 08:17:00.802496 25012 tablet_replica.cc:330] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:00.802734 25012 raft_consensus.cc:2229] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.802908 25012 raft_consensus.cc:2258] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:17:00.802960 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 82c49bba3df548558123344a13c946ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82c49bba3df548558123344a13c946ee already in progress: creating tablet
I20240502 08:17:00.802946 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet d423a92750c94ee1a08688d60dd2168e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:17:00.803151 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 18d82e75f2624e6782f55987cea223cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18d82e75f2624e6782f55987cea223cb already in progress: creating tablet
I20240502 08:17:00.803149 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet d423a92750c94ee1a08688d60dd2168e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:00.803612 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet d423a92750c94ee1a08688d60dd2168e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:00.803649 25012 ts_tablet_manager.cc:1875] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.796312 24181 rpcz_store.cc:276] 0502 08:16:59.647081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.930829 (+283748us) service_pool.cc:225] Handling call
0502 08:17:00.796137 (+865308us) ts_tablet_manager.cc:1120] Deleting tablet d9b67c8cd7f34cbba6401da59dec526c
0502 08:17:00.796146 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.796218 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":865167}
W20240502 08:17:00.805708 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet d9b67c8cd7f34cbba6401da59dec526c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9b67c8cd7f34cbba6401da59dec526c already in progress: creating tablet
I20240502 08:17:00.805999 24181 tablet_replica.cc:330] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:00.806190 24181 raft_consensus.cc:2229] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:00.806484 24181 raft_consensus.cc:2258] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.807049 24181 ts_tablet_manager.cc:1875] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.812711 24181 ts_tablet_manager.cc:1888] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:17:00.812806 24181 log.cc:1196] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/a2da8b9d3f374d8a812336057c0ce7d2
I20240502 08:17:00.813165 24181 ts_tablet_manager.cc:1909] T a2da8b9d3f374d8a812336057c0ce7d2 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:00.813840 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1176) took 1164 ms. Trace:
I20240502 08:17:00.813930 24181 rpcz_store.cc:276] 0502 08:16:59.648911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.930981 (+282070us) service_pool.cc:225] Handling call
0502 08:17:00.805955 (+874974us) ts_tablet_manager.cc:1120] Deleting tablet a2da8b9d3f374d8a812336057c0ce7d2
0502 08:17:00.805964 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.810016 (+ 4052us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.812640 (+ 2624us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.813826 (+ 1186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":874867}
I20240502 08:17:00.814227 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet a2da8b9d3f374d8a812336057c0ce7d2 (table default.Table_3edd27da6927478281e76361718425a3 [id=76b58ce209a24b9a95b405c1c50de3ca]) successfully deleted
I20240502 08:17:00.814530 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1188) took 1128 ms. Trace:
I20240502 08:17:00.814606 24181 rpcz_store.cc:276] 0502 08:16:59.686011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:00.014841 (+328830us) service_pool.cc:225] Handling call
0502 08:17:00.814435 (+799594us) ts_tablet_manager.cc:1120] Deleting tablet 8c93c89e1341474eafb6030b95f1c68c
0502 08:17:00.814443 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.814516 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":799343}
W20240502 08:17:00.814924 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 8c93c89e1341474eafb6030b95f1c68c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c93c89e1341474eafb6030b95f1c68c already in progress: creating tablet
I20240502 08:17:00.815209 24181 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51782 (request call id 1189) took 1128 ms. Trace:
I20240502 08:17:00.815281 24181 rpcz_store.cc:276] 0502 08:16:59.686441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:00.015148 (+328707us) service_pool.cc:225] Handling call
0502 08:17:00.815122 (+799974us) ts_tablet_manager.cc:1120] Deleting tablet c44ce35c463f416794edccc7149aa0cf
0502 08:17:00.815129 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.815197 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":799689}
W20240502 08:17:00.815570 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet c44ce35c463f416794edccc7149aa0cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c44ce35c463f416794edccc7149aa0cf already in progress: creating tablet
I20240502 08:17:00.815796 24181 tablet_replica.cc:330] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:00.816010 24181 raft_consensus.cc:2229] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.816174 24181 raft_consensus.cc:2258] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.816670 24181 ts_tablet_manager.cc:1875] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.816828 25012 ts_tablet_manager.cc:1888] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:00.816982 25012 log.cc:1196] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/af6efeebaebc495aba017a922e243c39
W20240502 08:17:00.817200 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a238b67dd6e84aa58d14a7802b6ba0ee consensus state has no leader
I20240502 08:17:00.817957 25012 ts_tablet_manager.cc:1909] T af6efeebaebc495aba017a922e243c39 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:00.818342 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1040) took 3144 ms. Trace:
I20240502 08:17:00.818667 23601 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_2ee0f7ddd41e41ea84e25359cd3ed0a4 [id=a9c009feee0e486794a8500121fe5487] alter complete (version 2)
I20240502 08:17:00.818895 25012 rpcz_store.cc:276] 0502 08:16:57.674025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.674075 (+ 50us) service_pool.cc:225] Handling call
0502 08:17:00.802460 (+3128385us) ts_tablet_manager.cc:1120] Deleting tablet af6efeebaebc495aba017a922e243c39
0502 08:17:00.802467 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.813106 (+ 10639us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.816694 (+ 3588us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.818323 (+ 1629us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3128211}
I20240502 08:17:00.819288 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet af6efeebaebc495aba017a922e243c39 (table default.Table_82f82cd6e702420283c288c10f905d90 [id=f293b22818344aea9eb6232141432183]) successfully deleted
I20240502 08:17:00.819823 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet ee022e8e86f8477a8fa1e58d52644b03 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:39 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:00.821748 24181 ts_tablet_manager.cc:1888] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:17:00.821887 24181 log.cc:1196] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/a5768cb928e244d2914f10f4eda3c1cd
I20240502 08:17:00.822238 24181 ts_tablet_manager.cc:1909] T a5768cb928e244d2914f10f4eda3c1cd P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:00.823037 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet a5768cb928e244d2914f10f4eda3c1cd (table default.Table_a394560ca3454cb88c3cbc2e87ab4fa7 [id=09f6a10417fb4639a9b97f5bb9efee6b]) successfully deleted
I20240502 08:17:00.823356 24181 tablet_replica.cc:330] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:00.823546 24181 raft_consensus.cc:2229] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:00.823828 24181 pending_rounds.cc:62] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5: Trying to abort 1 pending ops.
I20240502 08:17:00.823927 24181 pending_rounds.cc:68] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156507021963264 op_type: NO_OP noop_request { }
I20240502 08:17:00.824075 24181 raft_consensus.cc:2866] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20240502 08:17:00.824079 23723 consensus_peers.cc:485] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5768cb928e244d2914f10f4eda3c1cd. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:00.824214 24181 raft_consensus.cc:2258] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.824726 24181 ts_tablet_manager.cc:1875] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.825155 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1041) took 3150 ms. Trace:
I20240502 08:17:00.825268 25012 rpcz_store.cc:276] 0502 08:16:57.674915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.674951 (+ 36us) service_pool.cc:225] Handling call
0502 08:17:00.825063 (+3150112us) ts_tablet_manager.cc:1120] Deleting tablet b054783710824dbc8f7bc3ea38f605b0
0502 08:17:00.825069 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.825139 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3149865}
I20240502 08:17:00.825738 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1055) took 3047 ms. Trace:
W20240502 08:17:00.825816 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet b054783710824dbc8f7bc3ea38f605b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b054783710824dbc8f7bc3ea38f605b0 already in progress: creating tablet
I20240502 08:17:00.825850 25012 rpcz_store.cc:276] 0502 08:16:57.778629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.778682 (+ 53us) service_pool.cc:225] Handling call
0502 08:17:00.825651 (+3046969us) ts_tablet_manager.cc:1120] Deleting tablet c84b27ed8cf7429ca720e4cb8d396342
0502 08:17:00.825658 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.825724 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2964990}
W20240502 08:17:00.826459 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet c84b27ed8cf7429ca720e4cb8d396342 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c84b27ed8cf7429ca720e4cb8d396342 already in progress: creating tablet
I20240502 08:17:00.826666 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1056) took 3047 ms. Trace:
I20240502 08:17:00.826793 25012 rpcz_store.cc:276] 0502 08:16:57.778904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.778982 (+ 78us) service_pool.cc:225] Handling call
0502 08:17:00.826484 (+3047502us) ts_tablet_manager.cc:1120] Deleting tablet 9a9676602d1c4b37b3f4126a29f5456f
0502 08:17:00.826495 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.826648 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2965758}
W20240502 08:17:00.827262 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 9a9676602d1c4b37b3f4126a29f5456f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a9676602d1c4b37b3f4126a29f5456f already in progress: creating tablet
I20240502 08:17:00.827402 25012 tablet_replica.cc:330] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:00.827675 25012 raft_consensus.cc:2229] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.827870 25012 raft_consensus.cc:2258] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.828459 25012 ts_tablet_manager.cc:1875] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.825227 25779 tablet.cc:1722] T 7ed919909289410e867881baab841f53 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:17:00.832523 24181 ts_tablet_manager.cc:1888] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:00.832620 24181 log.cc:1196] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/a5a5a410b6314818b10927e35032f95d
I20240502 08:17:00.832922 24181 ts_tablet_manager.cc:1909] T a5a5a410b6314818b10927e35032f95d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:00.833729 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet a5a5a410b6314818b10927e35032f95d (table default.Table_79bb10e705c047138e1c4363673a65e6 [id=e5b9ab35a16943af94e5108dc0d29426]) successfully deleted
I20240502 08:17:00.833966 24181 tablet_replica.cc:330] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:00.834120 24181 raft_consensus.cc:2229] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:00.834286 24181 pending_rounds.cc:62] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5: Trying to abort 1 pending ops.
I20240502 08:17:00.834323 24181 pending_rounds.cc:68] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156511197306880 op_type: NO_OP noop_request { }
I20240502 08:17:00.834403 24181 raft_consensus.cc:2866] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:00.834468 24181 raft_consensus.cc:2258] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.835207 24181 ts_tablet_manager.cc:1875] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.836953 25012 ts_tablet_manager.cc:1888] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240502 08:17:00.840024 23594 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 3519b9c0ff2546c882ac9a92add72b40 (table default.Table_3a2e4270c3d04bb39c94940fb2adb74a [id=3834a986d4d844bf919e00778ec96047], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3519b9c0ff2546c882ac9a92add72b40 consensus state has no leader
I20240502 08:17:00.841805 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_54b9e5042d19438ab2690d06917ec10c" } modify_external_catalogs: true
I20240502 08:17:00.841950 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_54b9e5042d19438ab2690d06917ec10c" } modify_external_catalogs: true
I20240502 08:17:00.844646 23603 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 156c7356ece342aea0d8db667ebad89c (table default.Table_d8c6955b89724769bbfd81db2b5be455 [id=15b52f0688c24df2b6f3fda3732ceeb4]). Expected version 1 got 0
I20240502 08:17:00.844988 25012 log.cc:1196] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/e47459b5f5bb4ecb96f76240fefa1f5d
I20240502 08:17:00.845340 25012 ts_tablet_manager.cc:1909] T e47459b5f5bb4ecb96f76240fefa1f5d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:00.845409 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_47192b76b10e4c79a75072b2846ac5af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:00.845623 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1057) took 3066 ms. Trace:
I20240502 08:17:00.845683 25012 rpcz_store.cc:276] 0502 08:16:57.779026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.779072 (+ 46us) service_pool.cc:225] Handling call
0502 08:17:00.827292 (+3048220us) ts_tablet_manager.cc:1120] Deleting tablet e47459b5f5bb4ecb96f76240fefa1f5d
0502 08:17:00.827304 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.832996 (+ 5692us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.836876 (+ 3880us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.845604 (+ 8728us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2966525}
W20240502 08:17:00.845889 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47192b76b10e4c79a75072b2846ac5af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:00.846087 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet e47459b5f5bb4ecb96f76240fefa1f5d (table default.Table_922974ba60ab43e5af71756d9209fd0d [id=51cd52edc7ae46738d9ffa2bd8083fdb]) successfully deleted
I20240502 08:17:00.846387 25012 tablet_replica.cc:330] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:00.846573 25012 raft_consensus.cc:2229] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.846704 25012 raft_consensus.cc:2258] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.847193 25012 ts_tablet_manager.cc:1875] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.847303 24175 tablet_bootstrap.cc:492] T 8910bc67c6234a538c8160e7c020653c P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:00.847245 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_37c06b36010d4694830aa2c23281c13c" } new_table_name: "default.Table_75e1fe3b13194c87a96db2e77dff9da9" modify_external_catalogs: true
I20240502 08:17:00.847431 24175 ts_tablet_manager.cc:1367] T 8910bc67c6234a538c8160e7c020653c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.172s user 0.010s sys 0.015s
I20240502 08:17:00.848069 24175 raft_consensus.cc:348] T 8910bc67c6234a538c8160e7c020653c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:00.848214 24181 ts_tablet_manager.cc:1888] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:00.848230 24175 raft_consensus.cc:374] T 8910bc67c6234a538c8160e7c020653c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:00.848291 24175 raft_consensus.cc:729] T 8910bc67c6234a538c8160e7c020653c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:00.848332 24181 log.cc:1196] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/a198c33a3e314b5b81a08244862b281a
I20240502 08:17:00.848641 24181 ts_tablet_manager.cc:1909] T a198c33a3e314b5b81a08244862b281a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:00.849079 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1395c9cd077b435d927da88d7bf720fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:00.849396 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet a198c33a3e314b5b81a08244862b281a (table default.Table_22b226147b894b50a865f33b9f41491c [id=337cf81e0af74e8ca4333e4b1bfe4065]) successfully deleted
W20240502 08:17:00.849529 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1395c9cd077b435d927da88d7bf720fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:00.849745 24181 tablet_replica.cc:330] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:00.849895 24181 raft_consensus.cc:2229] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.850015 24181 raft_consensus.cc:2258] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.850418 24181 ts_tablet_manager.cc:1875] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.850898 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5b4382ee4fc14029864a8f4b11589da4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:00.851251 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b4382ee4fc14029864a8f4b11589da4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:00.852427 25012 ts_tablet_manager.cc:1888] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:00.852557 25012 log.cc:1196] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/090bec30f5664421b9bfe3f2810afd14
I20240502 08:17:00.852843 25012 ts_tablet_manager.cc:1909] T 090bec30f5664421b9bfe3f2810afd14 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:00.853132 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1058) took 3073 ms. Trace:
I20240502 08:17:00.853679 25012 rpcz_store.cc:276] 0502 08:16:57.779281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.779323 (+ 42us) service_pool.cc:225] Handling call
0502 08:17:00.846347 (+3067024us) ts_tablet_manager.cc:1120] Deleting tablet 090bec30f5664421b9bfe3f2810afd14
0502 08:17:00.846355 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.850133 (+ 3778us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.852368 (+ 2235us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.853117 (+ 749us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2985544}
I20240502 08:17:00.853817 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_606b0e91907c4c5384fab7ba4980f048" } new_table_name: "default.Table_a889270ae3d442d29f570221d6d5494f" modify_external_catalogs: true
I20240502 08:17:00.855933 24181 ts_tablet_manager.cc:1888] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:00.856505 24181 log.cc:1196] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/07344b78894c42809391d0fe2a9cd8f7
I20240502 08:17:00.856786 24181 ts_tablet_manager.cc:1909] T 07344b78894c42809391d0fe2a9cd8f7 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:00.857550 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 07344b78894c42809391d0fe2a9cd8f7 (table default.Table_4c4a02d4a6eb4c87971a03d670f5fdaf [id=6b6ffe47b9534a618f0912d78e7bb92a]) successfully deleted
I20240502 08:17:00.857612 24181 tablet_replica.cc:330] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:00.857759 24181 raft_consensus.cc:2229] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.857878 24181 raft_consensus.cc:2258] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.857909 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 090bec30f5664421b9bfe3f2810afd14 (table default.Table_70034c04442f49f6a42cb66dccfd1cc1 [id=74be9e9508024ebeb4623c488e21faf9]) successfully deleted
I20240502 08:17:00.858151 25012 tablet_replica.cc:330] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:00.858275 24181 ts_tablet_manager.cc:1875] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.858383 25012 raft_consensus.cc:2229] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.858539 25012 raft_consensus.cc:2258] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.858969 25012 ts_tablet_manager.cc:1875] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.860157 24180 ts_tablet_manager.cc:1888] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:00.860247 24180 log.cc:1196] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/6746c25111d04f20a6f30ce0838d71f5
I20240502 08:17:00.860571 24180 ts_tablet_manager.cc:1909] T 6746c25111d04f20a6f30ce0838d71f5 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:00.861207 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1193) took 2030 ms. Trace:
I20240502 08:17:00.861274 24180 rpcz_store.cc:276] 0502 08:16:58.830525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.848798 (+ 18273us) service_pool.cc:225] Handling call
0502 08:16:59.915697 (+1066899us) ts_tablet_manager.cc:1120] Deleting tablet 6746c25111d04f20a6f30ce0838d71f5
0502 08:16:59.915705 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.767647 (+851942us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.860089 (+ 92442us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.861188 (+ 1099us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1066686}
I20240502 08:17:00.861711 24175 consensus_queue.cc:260] T 8910bc67c6234a538c8160e7c020653c P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:00.863037 24181 ts_tablet_manager.cc:1888] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:00.863116 24181 log.cc:1196] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/91f1ab141585441db46d3027833932b0
I20240502 08:17:00.863332 24181 ts_tablet_manager.cc:1909] T 91f1ab141585441db46d3027833932b0 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:00.864038 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 91f1ab141585441db46d3027833932b0 (table default.Table_b141c2b67ffe43ff97d32c9fff1d1fe0 [id=b2163faeeed146d9936a6e1a5cfbc46e]) successfully deleted
W20240502 08:17:00.864351 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 9850e84117044d2f9dc82b6314dac2cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9850e84117044d2f9dc82b6314dac2cc already in progress: creating tablet
W20240502 08:17:00.864553 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 1b175e1b55bf4102804666b6fffccead because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b175e1b55bf4102804666b6fffccead already in progress: creating tablet
W20240502 08:17:00.864746 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 49acc2af1a064dd2b0b60120beed69ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49acc2af1a064dd2b0b60120beed69ba already in progress: creating tablet
W20240502 08:17:00.864939 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 14f54d6ded85434da35c1dfcff52763a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14f54d6ded85434da35c1dfcff52763a already in progress: creating tablet
W20240502 08:17:00.865167 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet d423a92750c94ee1a08688d60dd2168e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d423a92750c94ee1a08688d60dd2168e already in progress: creating tablet
W20240502 08:17:00.865339 23597 catalog_manager.cc:4797] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet ee022e8e86f8477a8fa1e58d52644b03 because the tablet was not found. No further retry: Not found: Tablet not found: ee022e8e86f8477a8fa1e58d52644b03
I20240502 08:17:00.866382 24175 ts_tablet_manager.cc:1398] T 8910bc67c6234a538c8160e7c020653c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240502 08:17:00.867960 24173 tablet_bootstrap.cc:492] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:00.868068 24173 ts_tablet_manager.cc:1367] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.236s user 0.010s sys 0.015s
I20240502 08:17:00.868080 25012 ts_tablet_manager.cc:1888] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:00.868167 25012 log.cc:1196] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/ba3aa2621fdb43fd8299f81af486ef1e
I20240502 08:17:00.868422 25012 ts_tablet_manager.cc:1909] T ba3aa2621fdb43fd8299f81af486ef1e P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:00.868640 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1059) took 3089 ms. Trace:
I20240502 08:17:00.868692 25012 rpcz_store.cc:276] 0502 08:16:57.779522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.779568 (+ 46us) service_pool.cc:225] Handling call
0502 08:17:00.858114 (+3078546us) ts_tablet_manager.cc:1120] Deleting tablet ba3aa2621fdb43fd8299f81af486ef1e
0502 08:17:00.858123 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.861516 (+ 3393us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.868011 (+ 6495us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.868626 (+ 615us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2997273}
I20240502 08:17:00.868674 24173 raft_consensus.cc:348] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.868877 24173 raft_consensus.cc:374] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:00.868924 24173 raft_consensus.cc:729] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:00.869295 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1197) took 2001 ms. Trace:
I20240502 08:17:00.869518 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet ba3aa2621fdb43fd8299f81af486ef1e (table default.Table_4481e3bc8b3548b0a7ecef01d39f80e8 [id=f8b12b3bb4904c18be60b567cb05b739]) successfully deleted
I20240502 08:17:00.869020 24175 tablet_bootstrap.cc:492] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:00.871908 24175 tablet_bootstrap.cc:654] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:00.875614 25012 tablet_replica.cc:330] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:00.875815 25012 raft_consensus.cc:2229] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.876015 25012 raft_consensus.cc:2258] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.876495 25012 ts_tablet_manager.cc:1875] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.869125 24173 consensus_queue.cc:260] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:00.881373 24173 ts_tablet_manager.cc:1398] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240502 08:17:00.882220 25012 ts_tablet_manager.cc:1888] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:00.882349 25012 log.cc:1196] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/fbef606ce7ac499d9bd84b49d804621c
I20240502 08:17:00.882625 25012 ts_tablet_manager.cc:1909] T fbef606ce7ac499d9bd84b49d804621c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:00.882915 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1060) took 3103 ms. Trace:
I20240502 08:17:00.883016 25012 rpcz_store.cc:276] 0502 08:16:57.779804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.779856 (+ 52us) service_pool.cc:225] Handling call
0502 08:17:00.875576 (+3095720us) ts_tablet_manager.cc:1120] Deleting tablet fbef606ce7ac499d9bd84b49d804621c
0502 08:17:00.875583 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.879213 (+ 3630us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.882163 (+ 2950us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.882901 (+ 738us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3014682}
I20240502 08:17:00.883383 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet fbef606ce7ac499d9bd84b49d804621c (table default.Table_1e7f2516f34b483d913cf5080a8febec [id=d5c976b2debc472fa1d7c28ae73ca32a]) successfully deleted
I20240502 08:17:00.883648 25012 tablet_replica.cc:330] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:00.883829 25012 raft_consensus.cc:2229] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.884014 25012 raft_consensus.cc:2258] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.884594 25012 ts_tablet_manager.cc:1875] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.885457 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 6746c25111d04f20a6f30ce0838d71f5 (table default.Table_02f448a8a9844def8ecda8716fdc352e [id=79a03d22b7e34f76bb3242c640129a13]) successfully deleted
I20240502 08:17:00.889667 24173 tablet_bootstrap.cc:492] T 9467974d67c64c7eb5e488ac51f1240d P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:00.891393 24173 tablet_bootstrap.cc:654] T 9467974d67c64c7eb5e488ac51f1240d P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:00.899034 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ce67caa9ca8a4bd092ecf98b26fd3afb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:00.899641 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce67caa9ca8a4bd092ecf98b26fd3afb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:00.902271 25012 ts_tablet_manager.cc:1888] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:00.902369 25012 log.cc:1196] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/df08f0d3af4c4c7e8abce656714b1b7f
I20240502 08:17:00.902706 25012 ts_tablet_manager.cc:1909] T df08f0d3af4c4c7e8abce656714b1b7f P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:00.902974 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1061) took 3122 ms. Trace:
I20240502 08:17:00.903046 25012 rpcz_store.cc:276] 0502 08:16:57.780175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.780220 (+ 45us) service_pool.cc:225] Handling call
0502 08:17:00.883612 (+3103392us) ts_tablet_manager.cc:1120] Deleting tablet df08f0d3af4c4c7e8abce656714b1b7f
0502 08:17:00.883619 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.887535 (+ 3916us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.902195 (+ 14660us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.902959 (+ 764us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3022662}
I20240502 08:17:00.904026 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 156c7356ece342aea0d8db667ebad89c (table default.Table_d8c6955b89724769bbfd81db2b5be455 [id=15b52f0688c24df2b6f3fda3732ceeb4]). Expected version 1 got 0
I20240502 08:17:00.904296 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 55f3f09fc1f94c00a47b9087b98e221d (table default.Table_7c74c34063104d54b3ab77c878892069 [id=106dd88de5eb49e89f0042beb7920736]). Expected version 2 got 0
I20240502 08:17:00.904515 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet e0b0d4871dd24607854100a4ef762e25 (table default.Table_5301ffccbac743618a78289c2c1415e1 [id=910c7cae36264e39bbfca0be919538e0]). Expected version 1 got 0
I20240502 08:17:00.905644 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 156c7356ece342aea0d8db667ebad89c (table default.Table_d8c6955b89724769bbfd81db2b5be455 [id=15b52f0688c24df2b6f3fda3732ceeb4]). Expected version 1 got 0
I20240502 08:17:00.905833 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 96b8f21c64e343d88d84e9acd0bc69b4 (table default.Table_1b91eb0181b74d5ea0d24f2b9731c155 [id=9ff8ea75a9bc4de194a70fe2088a94bc]). Expected version 2 got 0
I20240502 08:17:00.906328 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet df08f0d3af4c4c7e8abce656714b1b7f (table default.Table_59f82dbde55b406ba5271520c5bab104 [id=8e9acf4c0dbf4590b82aaee07243ed0c]) successfully deleted
I20240502 08:17:00.906206 23609 catalog_manager.cc:5484] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66). New cstate: current_term: 1 leader_uuid: "5b89a1f0485a44b7ac1710e2369a809b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:00.906723 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 0501575a84ec454780f48574bf3155fe (table default.Table_540a424c4bfa4484a65f0b24140a4bbb [id=5cee878b3a31426c879a1bf95045bb09]). Expected version 4 got 0
I20240502 08:17:00.907347 23903 tablet_service.cc:1467] Processing CreateTablet for tablet d607bb20596946e495cf749fb7e2552d (DEFAULT_TABLE table=default.Table_9edd6347f7cb47229111f08382effee5 [id=0f561a70639e4e27a7d192270e8d608f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.907605 23901 tablet_service.cc:1467] Processing CreateTablet for tablet e39139d289f94943b038c15cdde223c2 (DEFAULT_TABLE table=default.Table_9ff802fb6dbc44b7bd29d961c12e0d3c [id=cbbf1df66e9d4e4bac125e8568317771]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.907712 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d607bb20596946e495cf749fb7e2552d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.907963 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e39139d289f94943b038c15cdde223c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.908178 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1f57ce662044b385369d95952cb0ac (DEFAULT_TABLE table=default.Table_04f57727640b4aceabd37ba02a21237a [id=feb6a464a40c4d68acfe133056a62285]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.908463 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e1f57ce662044b385369d95952cb0ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.909792 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae6415f12f4429991153ba9849da301 (DEFAULT_TABLE table=default.Table_3da446d477714e8cbb0ac5817ec962a6 [id=99713b34147a4ff1ad7a74f8a4142fa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.910270 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae6415f12f4429991153ba9849da301. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.911211 23902 tablet_service.cc:1467] Processing CreateTablet for tablet b06516a2288e4c13869f1931e7b70965 (DEFAULT_TABLE table=default.Table_77418fa37d9c4f4fbe5b9dc2120df056 [id=3178c0b3aaab4bccadeb837fd0474e31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.911532 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b06516a2288e4c13869f1931e7b70965. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.913787 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 9b712bf11984497b9115abdba0a4d83d (DEFAULT_TABLE table=default.Table_498fd09f968c4a7a944e5ad1e5326439 [id=be19abcc69a94bcaa0df5a4de4102eea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.914212 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b712bf11984497b9115abdba0a4d83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.921088 25012 tablet_replica.cc:330] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:00.921281 25012 raft_consensus.cc:2229] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.921409 25012 raft_consensus.cc:2258] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.921877 25012 ts_tablet_manager.cc:1875] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.869364 24180 rpcz_store.cc:276] 0502 08:16:58.867432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.867651 (+ 219us) service_pool.cc:225] Handling call
0502 08:17:00.861543 (+1993892us) ts_tablet_manager.cc:1120] Deleting tablet 990d0cc863ee4f49a2fc2fffb217c4d7
0502 08:17:00.861550 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.869280 (+ 7730us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1993714}
I20240502 08:17:00.924465 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1196) took 2060 ms. Trace:
I20240502 08:17:00.924566 24180 rpcz_store.cc:276] 0502 08:16:58.863647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.866836 (+ 3189us) service_pool.cc:225] Handling call
0502 08:17:00.924303 (+2057467us) ts_tablet_manager.cc:1120] Deleting tablet e2792c45255c42adaeefb7aef4e25715
0502 08:17:00.924315 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.924443 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2056217}
I20240502 08:17:00.924916 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1198) took 2056 ms. Trace:
I20240502 08:17:00.924988 24180 rpcz_store.cc:276] 0502 08:16:58.868890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.868944 (+ 54us) service_pool.cc:225] Handling call
0502 08:17:00.924819 (+2055875us) ts_tablet_manager.cc:1120] Deleting tablet 3ad53633ce57478194437f95135d81de
0502 08:17:00.924828 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.924901 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2055705}
W20240502 08:17:00.925108 23429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2vVFYb/build/asan/bin/kudu pid 24241: Timed out: CheckLeaks RPC to 127.22.225.126:43265 timed out after 1.000s (SENT)
I20240502 08:17:00.925290 23429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2vVFYb/build/asan/bin/kudu with pid 24241
I20240502 08:17:00.925292 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1199) took 1948 ms. Trace:
I20240502 08:17:00.925357 24180 rpcz_store.cc:276] 0502 08:16:58.976864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.980041 (+ 3177us) service_pool.cc:225] Handling call
0502 08:17:00.925183 (+1945142us) ts_tablet_manager.cc:1120] Deleting tablet 58f538766e934282932fd4ab41b49831
0502 08:17:00.925190 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.925264 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1944891}
I20240502 08:17:00.926271 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1200) took 1816 ms. Trace:
I20240502 08:17:00.926411 24180 rpcz_store.cc:276] 0502 08:16:59.109354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.109406 (+ 52us) service_pool.cc:225] Handling call
0502 08:17:00.926166 (+1816760us) ts_tablet_manager.cc:1120] Deleting tablet f03473f7db3e411186f0db827a6630f7
0502 08:17:00.926176 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.926255 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1816566}
I20240502 08:17:00.926748 24180 tablet_replica.cc:330] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:00.926986 24180 raft_consensus.cc:2229] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.927242 24180 raft_consensus.cc:2258] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.927994 24180 ts_tablet_manager.cc:1875] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.933753 23750 tablet_service.cc:1467] Processing CreateTablet for tablet e39139d289f94943b038c15cdde223c2 (DEFAULT_TABLE table=default.Table_9ff802fb6dbc44b7bd29d961c12e0d3c [id=cbbf1df66e9d4e4bac125e8568317771]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.934213 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e39139d289f94943b038c15cdde223c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.934840 23760 tablet_service.cc:1467] Processing CreateTablet for tablet b06516a2288e4c13869f1931e7b70965 (DEFAULT_TABLE table=default.Table_77418fa37d9c4f4fbe5b9dc2120df056 [id=3178c0b3aaab4bccadeb837fd0474e31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.935149 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b06516a2288e4c13869f1931e7b70965. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:00.935300 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 990d0cc863ee4f49a2fc2fffb217c4d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 990d0cc863ee4f49a2fc2fffb217c4d7 already in progress: creating tablet
W20240502 08:17:00.935515 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e2792c45255c42adaeefb7aef4e25715 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2792c45255c42adaeefb7aef4e25715 already in progress: creating tablet
W20240502 08:17:00.935672 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 3ad53633ce57478194437f95135d81de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ad53633ce57478194437f95135d81de already in progress: creating tablet
W20240502 08:17:00.935789 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 58f538766e934282932fd4ab41b49831 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58f538766e934282932fd4ab41b49831 already in progress: creating tablet
W20240502 08:17:00.935914 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet f03473f7db3e411186f0db827a6630f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f03473f7db3e411186f0db827a6630f7 already in progress: creating tablet
I20240502 08:17:00.936630 23758 tablet_service.cc:1467] Processing CreateTablet for tablet d607bb20596946e495cf749fb7e2552d (DEFAULT_TABLE table=default.Table_9edd6347f7cb47229111f08382effee5 [id=0f561a70639e4e27a7d192270e8d608f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.936929 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d607bb20596946e495cf749fb7e2552d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.940760 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1f57ce662044b385369d95952cb0ac (DEFAULT_TABLE table=default.Table_04f57727640b4aceabd37ba02a21237a [id=feb6a464a40c4d68acfe133056a62285]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.941020 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e1f57ce662044b385369d95952cb0ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.945165 25012 ts_tablet_manager.cc:1888] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:00.945902 25012 log.cc:1196] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/592ab8523b8d479d9caead321d5888c3
I20240502 08:17:00.946435 25012 ts_tablet_manager.cc:1909] T 592ab8523b8d479d9caead321d5888c3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:00.947206 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1062) took 3166 ms. Trace:
I20240502 08:17:00.947716 25012 rpcz_store.cc:276] 0502 08:16:57.780438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.780484 (+ 46us) service_pool.cc:225] Handling call
0502 08:17:00.921044 (+3140560us) ts_tablet_manager.cc:1120] Deleting tablet 592ab8523b8d479d9caead321d5888c3
0502 08:17:00.921054 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.938074 (+ 17020us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.945081 (+ 7007us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.947186 (+ 2105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3060054}
I20240502 08:17:00.948386 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 592ab8523b8d479d9caead321d5888c3 (table default.Table_af7c06dd8b3f4085b62eaef9ff9b57cb [id=11e47f3f89ff4bbbbb32ab105efe686c]) successfully deleted
I20240502 08:17:00.948758 25012 tablet_replica.cc:330] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:00.949116 25012 raft_consensus.cc:2229] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.949462 25012 pending_rounds.cc:62] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b: Trying to abort 1 pending ops.
I20240502 08:17:00.949645 25012 pending_rounds.cc:68] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156502552535040 op_type: NO_OP noop_request { }
I20240502 08:17:00.949872 25012 raft_consensus.cc:2866] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:00.950050 25012 raft_consensus.cc:2258] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.950637 25012 ts_tablet_manager.cc:1875] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.950721 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae6415f12f4429991153ba9849da301 (DEFAULT_TABLE table=default.Table_3da446d477714e8cbb0ac5817ec962a6 [id=99713b34147a4ff1ad7a74f8a4142fa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.951071 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae6415f12f4429991153ba9849da301. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.956575 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 9b712bf11984497b9115abdba0a4d83d (DEFAULT_TABLE table=default.Table_498fd09f968c4a7a944e5ad1e5326439 [id=be19abcc69a94bcaa0df5a4de4102eea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.956971 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b712bf11984497b9115abdba0a4d83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.959079 25012 ts_tablet_manager.cc:1888] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:00.902746 24051 raft_consensus.cc:3037] T e0b0d4871dd24607854100a4ef762e25 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:00.908243 24031 tablet_service.cc:1467] Processing CreateTablet for tablet e39139d289f94943b038c15cdde223c2 (DEFAULT_TABLE table=default.Table_9ff802fb6dbc44b7bd29d961c12e0d3c [id=cbbf1df66e9d4e4bac125e8568317771]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.965327 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e39139d289f94943b038c15cdde223c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.908605 24029 tablet_service.cc:1467] Processing CreateTablet for tablet b06516a2288e4c13869f1931e7b70965 (DEFAULT_TABLE table=default.Table_77418fa37d9c4f4fbe5b9dc2120df056 [id=3178c0b3aaab4bccadeb837fd0474e31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.969959 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b06516a2288e4c13869f1931e7b70965. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.908838 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 7e1f57ce662044b385369d95952cb0ac (DEFAULT_TABLE table=default.Table_04f57727640b4aceabd37ba02a21237a [id=feb6a464a40c4d68acfe133056a62285]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.909068 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae6415f12f4429991153ba9849da301 (DEFAULT_TABLE table=default.Table_3da446d477714e8cbb0ac5817ec962a6 [id=99713b34147a4ff1ad7a74f8a4142fa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.909312 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 9b712bf11984497b9115abdba0a4d83d (DEFAULT_TABLE table=default.Table_498fd09f968c4a7a944e5ad1e5326439 [id=be19abcc69a94bcaa0df5a4de4102eea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.917429 24027 tablet_service.cc:1467] Processing CreateTablet for tablet d607bb20596946e495cf749fb7e2552d (DEFAULT_TABLE table=default.Table_9edd6347f7cb47229111f08382effee5 [id=0f561a70639e4e27a7d192270e8d608f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:00.973436 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e1f57ce662044b385369d95952cb0ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.974979 25012 log.cc:1196] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/89f57e3a39d54b7ab372d4b145b7a696
I20240502 08:17:00.975281 25012 ts_tablet_manager.cc:1909] T 89f57e3a39d54b7ab372d4b145b7a696 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:00.975543 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1063) took 3195 ms. Trace:
I20240502 08:17:00.977777 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d607bb20596946e495cf749fb7e2552d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.977977 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae6415f12f4429991153ba9849da301. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.975612 25012 rpcz_store.cc:276] 0502 08:16:57.780533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.780573 (+ 40us) service_pool.cc:225] Handling call
0502 08:17:00.948717 (+3168144us) ts_tablet_manager.cc:1120] Deleting tablet 89f57e3a39d54b7ab372d4b145b7a696
0502 08:17:00.948726 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.956386 (+ 7660us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.959006 (+ 2620us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.975524 (+ 16518us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3087692}
I20240502 08:17:00.978791 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1075) took 3018 ms. Trace:
I20240502 08:17:00.978857 25012 rpcz_store.cc:276] 0502 08:16:57.960365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.960418 (+ 53us) service_pool.cc:225] Handling call
0502 08:17:00.978677 (+3018259us) ts_tablet_manager.cc:1120] Deleting tablet d2886233097a4d19b45a4f139d82e68a
0502 08:17:00.978685 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.978777 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3017851}
I20240502 08:17:00.979142 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1073) took 3019 ms. Trace:
I20240502 08:17:00.979200 25012 rpcz_store.cc:276] 0502 08:16:57.960128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.960878 (+ 750us) service_pool.cc:225] Handling call
0502 08:17:00.979051 (+3018173us) ts_tablet_manager.cc:1120] Deleting tablet e4a87654d97249418e5df3a99b23c557
0502 08:17:00.979058 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.979129 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3017972}
I20240502 08:17:00.979456 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1074) took 3019 ms. Trace:
I20240502 08:17:00.979512 25012 rpcz_store.cc:276] 0502 08:16:57.960238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:57.961124 (+ 886us) service_pool.cc:225] Handling call
0502 08:17:00.979374 (+3018250us) ts_tablet_manager.cc:1120] Deleting tablet 63a595654cff4565bcf21a7c8fabc2c4
0502 08:17:00.979382 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.979443 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3018084}
I20240502 08:17:00.979776 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1078) took 2957 ms. Trace:
I20240502 08:17:00.979832 25012 rpcz_store.cc:276] 0502 08:16:58.022748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.022901 (+ 153us) service_pool.cc:225] Handling call
0502 08:17:00.979687 (+2956786us) ts_tablet_manager.cc:1120] Deleting tablet 876d7dc6291f47e4b0e671b50f2fa8c7
0502 08:17:00.979694 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.979761 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2956606}
I20240502 08:17:00.980090 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1079) took 2957 ms. Trace:
I20240502 08:17:00.980149 25012 rpcz_store.cc:276] 0502 08:16:58.023117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.023924 (+ 807us) service_pool.cc:225] Handling call
0502 08:17:00.980008 (+2956084us) ts_tablet_manager.cc:1120] Deleting tablet c4f9e61686dc417bb5a88f96b4231f2e
0502 08:17:00.980016 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.980077 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2955912}
I20240502 08:17:00.980392 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1077) took 2957 ms. Trace:
I20240502 08:17:00.980448 25012 rpcz_store.cc:276] 0502 08:16:58.022635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.029832 (+ 7197us) service_pool.cc:225] Handling call
0502 08:17:00.980311 (+2950479us) ts_tablet_manager.cc:1120] Deleting tablet 1b2aa20459484e4b98e0ae372418020f
0502 08:17:00.980318 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.980381 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2949772}
I20240502 08:17:00.980710 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1076) took 2958 ms. Trace:
I20240502 08:17:00.980798 25012 rpcz_store.cc:276] 0502 08:16:58.022514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.077676 (+ 55162us) service_pool.cc:225] Handling call
0502 08:17:00.980620 (+2902944us) ts_tablet_manager.cc:1120] Deleting tablet 3af208d6231a4cf18823e9fe367ca46d
0502 08:17:00.980627 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.980696 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2902651}
I20240502 08:17:00.981077 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1089) took 2824 ms. Trace:
I20240502 08:17:00.981148 25012 rpcz_store.cc:276] 0502 08:16:58.156141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.156202 (+ 61us) service_pool.cc:225] Handling call
0502 08:17:00.980991 (+2824789us) ts_tablet_manager.cc:1120] Deleting tablet 4bc133adb9a44cc7b56a3746d1eb76af
0502 08:17:00.980998 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.981063 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2796342}
I20240502 08:17:00.981389 25012 tablet_replica.cc:330] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:00.981876 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 89f57e3a39d54b7ab372d4b145b7a696 (table default.Table_28b3620523074c23866e772b117bf0ee [id=b09ad60d787b4740a493cb73c22b94a9]) successfully deleted
W20240502 08:17:00.982177 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet d2886233097a4d19b45a4f139d82e68a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2886233097a4d19b45a4f139d82e68a already in progress: creating tablet
W20240502 08:17:00.982362 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e4a87654d97249418e5df3a99b23c557 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4a87654d97249418e5df3a99b23c557 already in progress: creating tablet
W20240502 08:17:00.982522 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 63a595654cff4565bcf21a7c8fabc2c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63a595654cff4565bcf21a7c8fabc2c4 already in progress: creating tablet
I20240502 08:17:00.982609 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b712bf11984497b9115abdba0a4d83d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:00.982671 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 876d7dc6291f47e4b0e671b50f2fa8c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 876d7dc6291f47e4b0e671b50f2fa8c7 already in progress: creating tablet
W20240502 08:17:00.982825 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet c4f9e61686dc417bb5a88f96b4231f2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4f9e61686dc417bb5a88f96b4231f2e already in progress: creating tablet
W20240502 08:17:00.982969 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 1b2aa20459484e4b98e0ae372418020f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b2aa20459484e4b98e0ae372418020f already in progress: creating tablet
W20240502 08:17:00.983103 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 3af208d6231a4cf18823e9fe367ca46d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3af208d6231a4cf18823e9fe367ca46d already in progress: creating tablet
W20240502 08:17:00.983278 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 4bc133adb9a44cc7b56a3746d1eb76af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4bc133adb9a44cc7b56a3746d1eb76af already in progress: creating tablet
I20240502 08:17:00.981559 25012 raft_consensus.cc:2229] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:00.984292 25012 raft_consensus.cc:2258] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:00.984920 25012 ts_tablet_manager.cc:1875] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:00.994468 25012 ts_tablet_manager.cc:1888] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:00.997022 25012 log.cc:1196] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/18d284f71937459db0c0676ce4568175
I20240502 08:17:00.997439 25012 ts_tablet_manager.cc:1909] T 18d284f71937459db0c0676ce4568175 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:00.998562 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1094) took 2813 ms. Trace:
I20240502 08:17:00.999370 25012 rpcz_store.cc:276] 0502 08:16:58.185443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.185609 (+ 166us) service_pool.cc:225] Handling call
0502 08:17:00.981349 (+2795740us) ts_tablet_manager.cc:1120] Deleting tablet 18d284f71937459db0c0676ce4568175
0502 08:17:00.981358 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:00.988412 (+ 7054us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.994396 (+ 5984us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:00.998546 (+ 4150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2795503}
I20240502 08:17:00.999877 25012 tablet_replica.cc:330] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:01.000306 25012 raft_consensus.cc:2229] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:01.000317 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 18d284f71937459db0c0676ce4568175 (table default.Table_7af51f5f2feb42ca8dbe067300bbbfd9 [id=7eff9f0e4b304a01b8d8c521bfdd1563]) successfully deleted
I20240502 08:17:01.000720 25012 raft_consensus.cc:2258] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:01.001291 25012 ts_tablet_manager.cc:1875] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:01.002051 24175 tablet_bootstrap.cc:492] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:01.002207 24175 ts_tablet_manager.cc:1367] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.133s user 0.016s sys 0.014s
I20240502 08:17:01.002960 24175 raft_consensus.cc:348] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.003237 24175 raft_consensus.cc:374] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.003320 24175 raft_consensus.cc:729] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.003538 24175 consensus_queue.cc:260] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.005890 24175 ts_tablet_manager.cc:1398] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:17:01.006188 24175 tablet_bootstrap.cc:492] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:01.008219 24175 tablet_bootstrap.cc:654] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:17:01.012073 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da91c16f5cc3474ea21dbab9ea1dd5be consensus state has no leader
I20240502 08:17:01.021185 24174 tablet_bootstrap.cc:492] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:01.021517 24174 ts_tablet_manager.cc:1367] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.282s user 0.008s sys 0.008s
I20240502 08:17:01.024036 24174 raft_consensus.cc:348] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.024514 24174 raft_consensus.cc:374] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.024740 24174 raft_consensus.cc:729] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.025077 24174 consensus_queue.cc:260] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.026963 25649 raft_consensus.cc:482] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:01.027122 25649 raft_consensus.cc:504] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:01.027765 25649 leader_election.cc:288] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:01.030030 24175 tablet_bootstrap.cc:492] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:01.030167 24174 ts_tablet_manager.cc:1398] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240502 08:17:01.030328 24175 ts_tablet_manager.cc:1367] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.002s
I20240502 08:17:01.030980 24175 raft_consensus.cc:348] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.031193 24175 raft_consensus.cc:374] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.031280 24175 raft_consensus.cc:729] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.031504 24175 consensus_queue.cc:260] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.033959 24175 ts_tablet_manager.cc:1398] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:17:01.034207 24175 tablet_bootstrap.cc:492] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:01.034263 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b8f21c64e343d88d84e9acd0bc69b4" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:01.034440 24174 tablet_bootstrap.cc:492] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
W20240502 08:17:01.034726 23861 leader_election.cc:341] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:01.035799 24175 tablet_bootstrap.cc:654] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.035832 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b8f21c64e343d88d84e9acd0bc69b4" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:01.036075 23784 raft_consensus.cc:2454] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:01.036320 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0e95f8c694d7437c8aa8cb258bfcbe79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:01.036474 23858 leader_election.cc:302] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:01.036370 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0a2eb6adda7b49aeb8d231cd2306f093"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:01.036826 25649 raft_consensus.cc:2781] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:01.036913 25649 raft_consensus.cc:482] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240502 08:17:01.036948 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e95f8c694d7437c8aa8cb258bfcbe79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:01.036993 25649 raft_consensus.cc:3037] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:17:01.038276 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a2eb6adda7b49aeb8d231cd2306f093 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:01.039361 25649 raft_consensus.cc:504] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:01.042918 25649 leader_election.cc:288] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:01.044217 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b61b626a7be44b199bf2359795269a93" } new_table_name: "default.Table_ae675fd24d2049eeb39523c183bf2d4c" modify_external_catalogs: true
I20240502 08:17:01.048691 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b8f21c64e343d88d84e9acd0bc69b4" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:17:01.049322 23861 leader_election.cc:341] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:01.049634 24175 tablet_bootstrap.cc:492] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:01.049823 24175 ts_tablet_manager.cc:1367] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20240502 08:17:01.050487 24174 tablet_bootstrap.cc:654] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.050474 24175 raft_consensus.cc:348] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.050745 24175 raft_consensus.cc:374] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.050853 24175 raft_consensus.cc:729] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.051110 24175 consensus_queue.cc:260] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.051170 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b8f21c64e343d88d84e9acd0bc69b4" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:01.051319 23784 raft_consensus.cc:3037] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.051645 24175 ts_tablet_manager.cc:1398] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:01.053704 24175 tablet_bootstrap.cc:492] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:01.055215 23784 raft_consensus.cc:2454] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:01.056277 25649 raft_consensus.cc:482] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:01.056464 25649 raft_consensus.cc:504] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.057365 25649 leader_election.cc:288] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:01.057744 24175 tablet_bootstrap.cc:654] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:17:01.060122 23861 leader_election.cc:341] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: ec10ff7251a5442a8ecc1881b34db74d
I20240502 08:17:01.069463 24175 tablet_bootstrap.cc:492] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:01.069649 24175 ts_tablet_manager.cc:1367] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
I20240502 08:17:01.057830 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec10ff7251a5442a8ecc1881b34db74d" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:01.070210 24175 raft_consensus.cc:348] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.070430 24175 raft_consensus.cc:374] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.070369 23858 leader_election.cc:302] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:01.070819 23858 leader_election.cc:302] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:01.071101 25494 raft_consensus.cc:2781] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:01.071200 25494 raft_consensus.cc:686] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:17:01.071234 25357 raft_consensus.cc:2726] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:01.071395 25494 consensus_queue.cc:237] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:01.075358 24175 raft_consensus.cc:729] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.075817 24175 consensus_queue.cc:260] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.076982 24175 ts_tablet_manager.cc:1398] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 08:17:01.077437 24175 tablet_bootstrap.cc:492] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:01.079123 24174 tablet_bootstrap.cc:492] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:01.079241 24174 ts_tablet_manager.cc:1367] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.045s user 0.001s sys 0.007s
I20240502 08:17:01.079790 24174 raft_consensus.cc:348] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.079968 24174 raft_consensus.cc:374] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.080022 24174 raft_consensus.cc:729] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.081698 24174 consensus_queue.cc:260] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.082192 24174 ts_tablet_manager.cc:1398] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:17:01.082432 24174 tablet_bootstrap.cc:492] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:01.083390 24175 tablet_bootstrap.cc:654] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.083554 24174 tablet_bootstrap.cc:654] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:17:01.084354 23595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.225.126:43265 (error 108) [suppressed 5 similar messages]
W20240502 08:17:01.084642 23721 connection.cc:535] client connection to 127.22.225.126:43265 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:17:01.084690 23859 connection.cc:535] client connection to 127.22.225.126:43265 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:17:01.091163 23994 connection.cc:535] client connection to 127.22.225.126:43265 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:01.092844 24173 tablet_bootstrap.cc:492] T 9467974d67c64c7eb5e488ac51f1240d P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:01.093107 24173 ts_tablet_manager.cc:1367] T 9467974d67c64c7eb5e488ac51f1240d P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.203s user 0.004s sys 0.012s
I20240502 08:17:01.093923 24173 raft_consensus.cc:348] T 9467974d67c64c7eb5e488ac51f1240d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.094182 24173 raft_consensus.cc:374] T 9467974d67c64c7eb5e488ac51f1240d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.094271 24173 raft_consensus.cc:729] T 9467974d67c64c7eb5e488ac51f1240d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.094591 24173 consensus_queue.cc:260] T 9467974d67c64c7eb5e488ac51f1240d P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.095302 24173 ts_tablet_manager.cc:1398] T 9467974d67c64c7eb5e488ac51f1240d P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:01.095681 24173 tablet_bootstrap.cc:492] T 7fe583cbf4af493e82d3ec0508ea0f0b P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:01.097267 24173 tablet_bootstrap.cc:654] T 7fe583cbf4af493e82d3ec0508ea0f0b P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.064576 24180 ts_tablet_manager.cc:1888] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.3
I20240502 08:17:01.098285 24180 log.cc:1196] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/b811b81ddf2b4dc9b5e5f80331bff294
I20240502 08:17:01.098723 24180 ts_tablet_manager.cc:1909] T b811b81ddf2b4dc9b5e5f80331bff294 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:01.099553 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1201) took 1977 ms. Trace:
I20240502 08:17:01.099676 24180 rpcz_store.cc:276] 0502 08:16:59.121694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.121749 (+ 55us) service_pool.cc:225] Handling call
0502 08:17:00.926702 (+1804953us) ts_tablet_manager.cc:1120] Deleting tablet b811b81ddf2b4dc9b5e5f80331bff294
0502 08:17:00.926712 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.021221 (+ 94509us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.064457 (+ 43236us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.099531 (+ 35074us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1804737}
I20240502 08:17:01.100248 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet b811b81ddf2b4dc9b5e5f80331bff294 (table default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418]) successfully deleted
I20240502 08:17:01.111541 24180 tablet_replica.cc:330] T ec10ff7251a5442a8ecc1881b34db74d P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:01.111785 24180 raft_consensus.cc:2229] T ec10ff7251a5442a8ecc1881b34db74d P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240502 08:17:01.115613 25812 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:17:01.085559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.126:43265 (local address 127.22.225.66:51561)
0502 08:17:01.115390 (+ 29831us) negotiation.cc:107] Waiting for socket to connect
0502 08:17:01.115480 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.126:43265: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":29689,"threads_started":1}
I20240502 08:17:01.115865 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 5027fa2c80b544b8b1e2a4694bec7454
I20240502 08:17:01.116422 23608 catalog_manager.cc:6554] ReplaceTablet: tablet 5027fa2c80b544b8b1e2a4694bec7454 deleted and replaced by tablet 5615e8bdcbfe478c8c9fb029f95ec8ad
I20240502 08:17:01.116655 23608 master_service.cc:946] ReplaceTablet: received request to replace tablet ff4a333c6dd44a13bc267e885bb9e487 from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:01.117285 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet df06e0edb57f45e5ac71b568c14769cd
I20240502 08:17:01.117843 23605 master_service.cc:946] ReplaceTablet: received request to replace tablet 711a38d7c47749f8a7f59e52b81b51ee from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:01.118600 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet 5027fa2c80b544b8b1e2a4694bec7454 with delete_type TABLET_DATA_DELETED (replaced by tablet 5615e8bdcbfe478c8c9fb029f95ec8ad) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:01.118898 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet df06e0edb57f45e5ac71b568c14769cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:00 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:17:01.119138 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3922e528a97a4287a0122a5b51035dcb consensus state has no leader
I20240502 08:17:01.122025 23602 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 63afd6e4fee24edeaba1209105f1c3e6 (table default.Table_9f3ea2fdcdc2477b89d509e868ffe4de [id=84e36915119b4125a1b0d64b2d1647fa]). Expected version 1 got 0
I20240502 08:17:01.122210 23602 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet c45aa0dd26524e0fba27e4eaf17d704e (table default.Table_cbe61670c678416995bf27144ab702d9 [id=589deb067c2b4cd086d0277ecfed5126]). Expected version 2 got 0
I20240502 08:17:01.123476 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_28e9d1b5045f4818a01c228a3d25994d" } modify_external_catalogs: true
I20240502 08:17:01.123672 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_28e9d1b5045f4818a01c228a3d25994d" } modify_external_catalogs: true
I20240502 08:17:01.124739 25012 ts_tablet_manager.cc:1888] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:01.124819 25012 log.cc:1196] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/ee022e8e86f8477a8fa1e58d52644b03
I20240502 08:17:01.125530 24175 tablet_bootstrap.cc:492] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:01.125577 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0a76b688246e4aaaabcbc5ca97aaa72e" } new_table_name: "default.Table_50c546a17f484d3cb98a5ca4b3ed63e8" modify_external_catalogs: true
I20240502 08:17:01.125746 24175 ts_tablet_manager.cc:1367] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.001s
I20240502 08:17:01.126358 24175 raft_consensus.cc:348] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.126739 24175 raft_consensus.cc:374] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.126801 24175 raft_consensus.cc:729] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.127058 24175 consensus_queue.cc:260] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.127146 23601 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_2edad641920b44ce9a4d5b7503f0279a [id=ee3912407a934444ad2c62f7b83d93ac] alter complete (version 2)
I20240502 08:17:01.127940 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_db86cd2cf86e48ffbc6a513b5c2bd7ab" } modify_external_catalogs: true
I20240502 08:17:01.128108 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_db86cd2cf86e48ffbc6a513b5c2bd7ab" } modify_external_catalogs: true
I20240502 08:17:01.128670 24175 ts_tablet_manager.cc:1398] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:01.129652 24175 tablet_bootstrap.cc:492] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:01.130708 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fe583cbf4af493e82d3ec0508ea0f0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:01.130826 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet b6bb4a4ccd574f4cadc0b9d81ef4dd5b from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:01.130857 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet a4d151770ee44130adfd822518b9b017 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:01.131266 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet a5a5a410b6314818b10927e35032f95d with delete_type TABLET_DATA_DELETED (replaced by tablet 92b830e128ac439eba286f29dbb20ee4) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:01.132479 24175 tablet_bootstrap.cc:654] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.132768 25012 ts_tablet_manager.cc:1909] T ee022e8e86f8477a8fa1e58d52644b03 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
W20240502 08:17:01.135304 23595 consensus_peers.cc:485] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 -> Peer bcc8a323e6b9429eaaf4e0c1698ad077 (127.22.225.126:43265): Couldn't send request to peer bcc8a323e6b9429eaaf4e0c1698ad077. Status: Network error: Client connection negotiation failed: client connection to 127.22.225.126:43265: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:01.136425 24175 tablet_bootstrap.cc:492] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:01.136608 24175 ts_tablet_manager.cc:1367] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 08:17:01.136933 25494 raft_consensus.cc:482] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:01.137068 25494 raft_consensus.cc:504] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.137614 25494 leader_election.cc:288] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:01.138146 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4dd883502bdd4291ac0d3f15a4548db5" } modify_external_catalogs: true
I20240502 08:17:01.138274 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4dd883502bdd4291ac0d3f15a4548db5" } modify_external_catalogs: true
I20240502 08:17:01.140360 24175 raft_consensus.cc:348] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.140563 24175 raft_consensus.cc:374] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.140637 24175 raft_consensus.cc:729] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.140862 24175 consensus_queue.cc:260] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.142086 24175 ts_tablet_manager.cc:1398] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:01.149497 25494 raft_consensus.cc:482] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:01.149657 25494 raft_consensus.cc:504] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.150326 25494 leader_election.cc:288] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:17:01.151907 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4191e0f787d948bda654f8bbb6d8d24e" } modify_external_catalogs: true
I20240502 08:17:01.152094 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1095) took 2943 ms. Trace:
I20240502 08:17:01.152153 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4191e0f787d948bda654f8bbb6d8d24e" } modify_external_catalogs: true
I20240502 08:17:01.152220 25012 rpcz_store.cc:276] 0502 08:16:58.208881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.209145 (+ 264us) service_pool.cc:225] Handling call
0502 08:17:00.999836 (+2790691us) ts_tablet_manager.cc:1120] Deleting tablet ee022e8e86f8477a8fa1e58d52644b03
0502 08:17:00.999845 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.015330 (+ 15485us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.124643 (+109313us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.152076 (+ 27433us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2790484}
I20240502 08:17:01.152709 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet ee022e8e86f8477a8fa1e58d52644b03 (table default.Table_d6f5ca82488a40a69121f1b779d8f874 [id=0422356f66a74ad29c2e13e1ad36b654]) successfully deleted
I20240502 08:17:01.153215 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1096) took 2944 ms. Trace:
I20240502 08:17:01.153288 25012 rpcz_store.cc:276] 0502 08:16:58.210078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.210241 (+ 163us) service_pool.cc:225] Handling call
0502 08:17:01.152580 (+2942339us) ts_tablet_manager.cc:1120] Deleting tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9
0502 08:17:01.153090 (+ 510us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.153200 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2942107}
W20240502 08:17:01.153717 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cd2b85626f24f3f8c46b4b81f9ecdd9 already in progress: creating tablet
I20240502 08:17:01.153888 25012 tablet_replica.cc:330] T 592eeecbf5514215b6464bd030081e4b P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:01.154045 25012 raft_consensus.cc:2229] T 592eeecbf5514215b6464bd030081e4b P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:01.154181 25012 raft_consensus.cc:2258] T 592eeecbf5514215b6464bd030081e4b P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:01.154597 25012 ts_tablet_manager.cc:1875] T 592eeecbf5514215b6464bd030081e4b P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:01.160008 25012 ts_tablet_manager.cc:1888] T 592eeecbf5514215b6464bd030081e4b P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:01.160094 25012 log.cc:1196] T 592eeecbf5514215b6464bd030081e4b P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/592eeecbf5514215b6464bd030081e4b
I20240502 08:17:01.160342 25012 ts_tablet_manager.cc:1909] T 592eeecbf5514215b6464bd030081e4b P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:01.161000 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1101) took 2931 ms. Trace:
I20240502 08:17:01.161072 25012 rpcz_store.cc:276] 0502 08:16:58.229317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.229814 (+ 497us) service_pool.cc:225] Handling call
0502 08:17:01.153499 (+2923685us) ts_tablet_manager.cc:1120] Deleting tablet 592eeecbf5514215b6464bd030081e4b
0502 08:17:01.153840 (+ 341us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.157523 (+ 3683us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.159941 (+ 2418us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.160982 (+ 1041us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2922525}
I20240502 08:17:01.161415 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1109) took 2749 ms. Trace:
I20240502 08:17:01.161450 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 592eeecbf5514215b6464bd030081e4b (table default.Table_4191e0f787d948bda654f8bbb6d8d24e [id=07050bd496e84a97a86fdc4f0b7811ac]) successfully deleted
I20240502 08:17:01.162549 24175 tablet_bootstrap.cc:492] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
W20240502 08:17:01.121488 25811 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:17:01.085483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.225.126:43265 (local address 127.22.225.65:33727)
0502 08:17:01.121272 (+ 35789us) negotiation.cc:107] Waiting for socket to connect
0502 08:17:01.121355 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.225.126:43265: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":35689,"threads_started":1}
I20240502 08:17:01.134881 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet 5027fa2c80b544b8b1e2a4694bec7454 with delete_type TABLET_DATA_DELETED (replaced by tablet 5615e8bdcbfe478c8c9fb029f95ec8ad) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:01.138271 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b4c563d6bd4a44bb3bab58a3fcdc07" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:01.150808 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f2e98ef2884b05994d02114e992e80" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:01.163820 24175 tablet_bootstrap.cc:654] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.153218 23754 tablet_service.cc:1514] Processing DeleteTablet for tablet df06e0edb57f45e5ac71b568c14769cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:00 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:01.163939 23783 raft_consensus.cc:2454] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:01.164028 24180 raft_consensus.cc:2258] T ec10ff7251a5442a8ecc1881b34db74d P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:01.164731 23858 leader_election.cc:302] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:17:01.165191 25494 raft_consensus.cc:2781] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:01.165261 25494 raft_consensus.cc:482] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:01.165313 25494 raft_consensus.cc:3037] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.167169 24180 ts_tablet_manager.cc:1875] T ec10ff7251a5442a8ecc1881b34db74d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:01.167928 25494 raft_consensus.cc:504] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.169597 25494 leader_election.cc:288] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:17:01.169656 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f2e98ef2884b05994d02114e992e80" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:01.169852 23783 raft_consensus.cc:3037] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.172333 23783 raft_consensus.cc:2454] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:01.107698 25814 tablet.cc:1722] T e0b0d4871dd24607854100a4ef762e25 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:01.172683 23858 leader_election.cc:302] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:17:01.119910 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 5027fa2c80b544b8b1e2a4694bec7454 with delete_type TABLET_DATA_DELETED (replaced by tablet 5615e8bdcbfe478c8c9fb029f95ec8ad) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:01.172892 25494 raft_consensus.cc:2781] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:01.120934 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet df06e0edb57f45e5ac71b568c14769cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:00 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:01.138176 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b4c563d6bd4a44bb3bab58a3fcdc07" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:01.151250 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f2e98ef2884b05994d02114e992e80" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:01.173796 25494 raft_consensus.cc:686] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
W20240502 08:17:01.174063 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 5027fa2c80b544b8b1e2a4694bec7454 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5027fa2c80b544b8b1e2a4694bec7454 already in progress: creating tablet
I20240502 08:17:01.174100 25494 consensus_queue.cc:237] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:17:01.174289 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet df06e0edb57f45e5ac71b568c14769cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df06e0edb57f45e5ac71b568c14769cd already in progress: creating tablet
I20240502 08:17:01.161486 25012 rpcz_store.cc:276] 0502 08:16:58.412081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.412199 (+ 118us) service_pool.cc:225] Handling call
0502 08:17:01.161304 (+2749105us) ts_tablet_manager.cc:1120] Deleting tablet e0bd774640464d2a92b2e71126c83883
0502 08:17:01.161313 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.161392 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2748631}
I20240502 08:17:01.168972 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f2e98ef2884b05994d02114e992e80" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:17:01.174634 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e0bd774640464d2a92b2e71126c83883 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0bd774640464d2a92b2e71126c83883 already in progress: creating tablet
I20240502 08:17:01.174682 23861 leader_election.cc:302] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:01.174952 25649 raft_consensus.cc:2726] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:17:01.175093 23861 leader_election.cc:341] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:01.175078 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1110) took 2762 ms. Trace:
I20240502 08:17:01.175374 25012 rpcz_store.cc:276] 0502 08:16:58.413002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.413077 (+ 75us) service_pool.cc:225] Handling call
0502 08:17:01.174964 (+2761887us) ts_tablet_manager.cc:1120] Deleting tablet 11ebacd4c41941cd8a6ec634e3aeeaf5
0502 08:17:01.174974 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.175053 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2761544}
W20240502 08:17:01.175808 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 11ebacd4c41941cd8a6ec634e3aeeaf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11ebacd4c41941cd8a6ec634e3aeeaf5 already in progress: creating tablet
W20240502 08:17:01.176157 23861 leader_election.cc:341] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:01.176604 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1112) took 2673 ms. Trace:
I20240502 08:17:01.177112 25012 rpcz_store.cc:276] 0502 08:16:58.503223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.503269 (+ 46us) service_pool.cc:225] Handling call
0502 08:17:01.176466 (+2673197us) ts_tablet_manager.cc:1120] Deleting tablet 55fb41b8f8f9487a81e98bae9cddaae1
0502 08:17:01.176480 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.176583 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2672932}
W20240502 08:17:01.177691 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 55fb41b8f8f9487a81e98bae9cddaae1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55fb41b8f8f9487a81e98bae9cddaae1 already in progress: creating tablet
I20240502 08:17:01.178068 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1111) took 2675 ms. Trace:
I20240502 08:17:01.179677 25012 rpcz_store.cc:276] 0502 08:16:58.503105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.577014 (+ 73909us) service_pool.cc:225] Handling call
0502 08:17:01.177960 (+2600946us) ts_tablet_manager.cc:1120] Deleting tablet 73d7633fd23541f3a553af008dd6a354
0502 08:17:01.177970 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.178037 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2600758}
W20240502 08:17:01.180217 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 73d7633fd23541f3a553af008dd6a354 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73d7633fd23541f3a553af008dd6a354 already in progress: creating tablet
I20240502 08:17:01.180532 25012 tablet_replica.cc:330] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:01.180869 25012 raft_consensus.cc:2229] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:01.181226 25012 raft_consensus.cc:2258] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:01.182047 25012 ts_tablet_manager.cc:1875] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:01.184196 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet d1be2c29d3154d9189af3266191f0b3e
W20240502 08:17:01.185809 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b3b1e3e2e9514f289725f4afc9adb116 consensus state has no leader
I20240502 08:17:01.185953 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet d1be2c29d3154d9189af3266191f0b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:17:01.187242 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41f2e98ef2884b05994d02114e992e80 consensus state has no leader
I20240502 08:17:01.187356 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet b6bb4a4ccd574f4cadc0b9d81ef4dd5b
I20240502 08:17:01.187803 23609 catalog_manager.cc:6554] ReplaceTablet: tablet b6bb4a4ccd574f4cadc0b9d81ef4dd5b deleted and replaced by tablet 12d4afaeae25429cb469204690a4d3ec
I20240502 08:17:01.188658 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet b6bb4a4ccd574f4cadc0b9d81ef4dd5b with delete_type TABLET_DATA_DELETED (replaced by tablet 12d4afaeae25429cb469204690a4d3ec) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:01.188870 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet f87fe1c2c57748f0a17ee14503890c2f
I20240502 08:17:01.188926 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet b6bb4a4ccd574f4cadc0b9d81ef4dd5b with delete_type TABLET_DATA_DELETED (replaced by tablet 12d4afaeae25429cb469204690a4d3ec) from {username='slave'} at 127.0.0.1:33620
W20240502 08:17:01.189934 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet b6bb4a4ccd574f4cadc0b9d81ef4dd5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6bb4a4ccd574f4cadc0b9d81ef4dd5b already in progress: creating tablet
I20240502 08:17:01.190348 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet f87fe1c2c57748f0a17ee14503890c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:01.190574 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet f87fe1c2c57748f0a17ee14503890c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:17:01.191180 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet f87fe1c2c57748f0a17ee14503890c2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f87fe1c2c57748f0a17ee14503890c2f already in progress: creating tablet
I20240502 08:17:01.191736 24175 tablet_bootstrap.cc:492] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:01.191812 23754 tablet_service.cc:1514] Processing DeleteTablet for tablet d1be2c29d3154d9189af3266191f0b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:01.191915 24175 ts_tablet_manager.cc:1367] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.003s
I20240502 08:17:01.192536 23754 tablet_service.cc:1514] Processing DeleteTablet for tablet b6bb4a4ccd574f4cadc0b9d81ef4dd5b with delete_type TABLET_DATA_DELETED (replaced by tablet 12d4afaeae25429cb469204690a4d3ec) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:01.192767 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet d1be2c29d3154d9189af3266191f0b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:01.192695 24175 raft_consensus.cc:348] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.192888 24175 raft_consensus.cc:374] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.192948 24175 raft_consensus.cc:729] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.193298 24181 tablet_replica.cc:330] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:01.193352 23754 tablet_service.cc:1514] Processing DeleteTablet for tablet f87fe1c2c57748f0a17ee14503890c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:01.193465 24181 raft_consensus.cc:2229] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:01.193661 24181 raft_consensus.cc:2258] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:01.193166 24175 consensus_queue.cc:260] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.194198 24181 ts_tablet_manager.cc:1875] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:01.194655 24175 ts_tablet_manager.cc:1398] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:01.194860 24175 tablet_bootstrap.cc:492] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:01.196167 24175 tablet_bootstrap.cc:654] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.200242 24181 ts_tablet_manager.cc:1888] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:17:01.200340 24181 log.cc:1196] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/d1be2c29d3154d9189af3266191f0b3e
I20240502 08:17:01.200682 24181 ts_tablet_manager.cc:1909] T d1be2c29d3154d9189af3266191f0b3e P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
W20240502 08:17:01.201550 23861 consensus_peers.cc:485] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:01.201638 25012 ts_tablet_manager.cc:1888] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:01.201725 25012 log.cc:1196] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/6746c25111d04f20a6f30ce0838d71f5
I20240502 08:17:01.201987 25012 ts_tablet_manager.cc:1909] T 6746c25111d04f20a6f30ce0838d71f5 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:01.201963 23783 raft_consensus.cc:1261] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Refusing update from remote peer 5b89a1f0485a44b7ac1710e2369a809b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:01.202383 25649 consensus_queue.cc:1035] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:01.202922 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet d1be2c29d3154d9189af3266191f0b3e (table default.Table_db86cd2cf86e48ffbc6a513b5c2bd7ab [id=6103f133bcdd4f768b92b37245bf5356]) successfully deleted
I20240502 08:17:01.215906 24180 ts_tablet_manager.cc:1888] T ec10ff7251a5442a8ecc1881b34db74d P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:01.216923 24180 log.cc:1196] T ec10ff7251a5442a8ecc1881b34db74d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/ec10ff7251a5442a8ecc1881b34db74d
I20240502 08:17:01.218508 24180 ts_tablet_manager.cc:1909] T ec10ff7251a5442a8ecc1881b34db74d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:01.219352 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1202) took 1876 ms. Trace:
I20240502 08:17:01.220170 24173 tablet_bootstrap.cc:492] T 7fe583cbf4af493e82d3ec0508ea0f0b P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:01.220336 24173 ts_tablet_manager.cc:1367] T 7fe583cbf4af493e82d3ec0508ea0f0b P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.125s user 0.003s sys 0.001s
I20240502 08:17:01.221014 24173 raft_consensus.cc:348] T 7fe583cbf4af493e82d3ec0508ea0f0b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.221241 24173 raft_consensus.cc:374] T 7fe583cbf4af493e82d3ec0508ea0f0b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.221246 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1128) took 2388 ms. Trace:
I20240502 08:17:01.221364 24173 raft_consensus.cc:729] T 7fe583cbf4af493e82d3ec0508ea0f0b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.221331 25012 rpcz_store.cc:276] 0502 08:16:58.832722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.832772 (+ 50us) service_pool.cc:225] Handling call
0502 08:17:01.180494 (+2347722us) ts_tablet_manager.cc:1120] Deleting tablet 6746c25111d04f20a6f30ce0838d71f5
0502 08:17:01.180502 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.199192 (+ 18690us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.201551 (+ 2359us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.221226 (+ 19675us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2347568}
I20240502 08:17:01.221747 25012 tablet_replica.cc:330] T 001e3f4f72f64dc69dbd521724912f07 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:01.221968 25012 raft_consensus.cc:2229] T 001e3f4f72f64dc69dbd521724912f07 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:01.221983 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 6746c25111d04f20a6f30ce0838d71f5 (table default.Table_02f448a8a9844def8ecda8716fdc352e [id=79a03d22b7e34f76bb3242c640129a13]) successfully deleted
I20240502 08:17:01.222115 25012 raft_consensus.cc:2258] T 001e3f4f72f64dc69dbd521724912f07 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:01.222559 24175 tablet_bootstrap.cc:492] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:01.222723 24175 ts_tablet_manager.cc:1367] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.001s
I20240502 08:17:01.223349 24175 raft_consensus.cc:348] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.223551 24175 raft_consensus.cc:374] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.223629 24175 raft_consensus.cc:729] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.223851 24175 consensus_queue.cc:260] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.224472 24175 ts_tablet_manager.cc:1398] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:01.224741 24175 tablet_bootstrap.cc:492] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:01.226372 24175 tablet_bootstrap.cc:654] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.221680 24173 consensus_queue.cc:260] T 7fe583cbf4af493e82d3ec0508ea0f0b P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.228021 24173 ts_tablet_manager.cc:1398] T 7fe583cbf4af493e82d3ec0508ea0f0b P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 08:17:01.228504 24173 tablet_bootstrap.cc:492] T 41f2e98ef2884b05994d02114e992e80 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:01.229805 24173 tablet_bootstrap.cc:654] T 41f2e98ef2884b05994d02114e992e80 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.222587 25012 ts_tablet_manager.cc:1875] T 001e3f4f72f64dc69dbd521724912f07 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:01.233831 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1bdb95f894a04d6bbeafc73022a2283c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:01.234340 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1bdb95f894a04d6bbeafc73022a2283c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:17:01.220675 23723 consensus_peers.cc:485] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1be2c29d3154d9189af3266191f0b3e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:01.223462 24174 tablet_bootstrap.cc:492] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:01.236048 24174 ts_tablet_manager.cc:1367] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.154s user 0.007s sys 0.030s
I20240502 08:17:01.236323 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_43f0e7d47b19428e856ead8b62806088"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:01.236738 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43f0e7d47b19428e856ead8b62806088 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:01.236900 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9455a9a5481a40749f263245cfa038ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:01.236884 24174 raft_consensus.cc:348] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.237071 24174 raft_consensus.cc:374] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.237128 24174 raft_consensus.cc:729] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
W20240502 08:17:01.237335 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9455a9a5481a40749f263245cfa038ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:01.237327 24174 consensus_queue.cc:260] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.238353 24174 ts_tablet_manager.cc:1398] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:01.238593 24174 tablet_bootstrap.cc:492] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:01.239651 24174 tablet_bootstrap.cc:654] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.241447 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_36799657d69e43759def365198950d2e" } new_table_name: "default.Table_39b97fc571c04e5eb6c403c080d75a87" modify_external_catalogs: true
I20240502 08:17:01.243350 24180 rpcz_store.cc:276] 0502 08:16:59.342679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.343197 (+ 518us) service_pool.cc:225] Handling call
0502 08:17:01.111497 (+1768300us) ts_tablet_manager.cc:1120] Deleting tablet ec10ff7251a5442a8ecc1881b34db74d
0502 08:17:01.111507 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.204871 (+ 93364us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.215787 (+ 10916us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.219333 (+ 3546us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1768141}
I20240502 08:17:01.243888 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1203) took 1820 ms. Trace:
I20240502 08:17:01.244021 24180 rpcz_store.cc:276] 0502 08:16:59.423331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.423381 (+ 50us) service_pool.cc:225] Handling call
0502 08:17:01.243762 (+1820381us) ts_tablet_manager.cc:1120] Deleting tablet 0dae6a96776c477fb0ada8226d68ea25
0502 08:17:01.243771 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.243871 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1820221}
I20240502 08:17:01.244248 25779 tablet.cc:1722] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240502 08:17:01.244635 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1204) took 1816 ms. Trace:
I20240502 08:17:01.244709 24180 rpcz_store.cc:276] 0502 08:16:59.428271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.428333 (+ 62us) service_pool.cc:225] Handling call
0502 08:17:01.244530 (+1816197us) ts_tablet_manager.cc:1120] Deleting tablet 22bc17792cf040748ab672cfc6451e3c
0502 08:17:01.244539 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.244618 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1815115}
I20240502 08:17:01.245059 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1205) took 1787 ms. Trace:
I20240502 08:17:01.245133 24180 rpcz_store.cc:276] 0502 08:16:59.457663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.457745 (+ 82us) service_pool.cc:225] Handling call
0502 08:17:01.244916 (+1787171us) ts_tablet_manager.cc:1120] Deleting tablet 4e32319fa9244ed2a323b2b3f5940748
0502 08:17:01.244924 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.245042 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1773777}
I20240502 08:17:01.246331 24173 tablet_bootstrap.cc:492] T 41f2e98ef2884b05994d02114e992e80 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:01.246526 24173 ts_tablet_manager.cc:1367] T 41f2e98ef2884b05994d02114e992e80 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240502 08:17:01.246985 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1206) took 1786 ms. Trace:
I20240502 08:17:01.247295 25744 raft_consensus.cc:482] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:01.247272 24173 raft_consensus.cc:348] T 41f2e98ef2884b05994d02114e992e80 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.247457 24173 raft_consensus.cc:374] T 41f2e98ef2884b05994d02114e992e80 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.247359 24180 rpcz_store.cc:276] 0502 08:16:59.460905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.461023 (+ 118us) service_pool.cc:225] Handling call
0502 08:17:01.246864 (+1785841us) ts_tablet_manager.cc:1120] Deleting tablet 7ae9c817c7be42da8c689267c7eb5a06
0502 08:17:01.246873 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.246968 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1775363}
I20240502 08:17:01.247507 24173 raft_consensus.cc:729] T 41f2e98ef2884b05994d02114e992e80 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.247442 25744 raft_consensus.cc:504] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:01.248121 25744 leader_election.cc:288] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:01.248543 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8910bc67c6234a538c8160e7c020653c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:01.248713 24052 raft_consensus.cc:2454] T 8910bc67c6234a538c8160e7c020653c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:01.248711 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8910bc67c6234a538c8160e7c020653c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:01.248909 23920 raft_consensus.cc:2454] T 8910bc67c6234a538c8160e7c020653c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:01.249284 23723 leader_election.cc:302] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:17:01.249558 25744 raft_consensus.cc:2781] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:01.249642 25744 raft_consensus.cc:482] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:01.249706 25744 raft_consensus.cc:3037] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.251986 25012 ts_tablet_manager.cc:1888] T 001e3f4f72f64dc69dbd521724912f07 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:01.252084 25012 log.cc:1196] T 001e3f4f72f64dc69dbd521724912f07 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/001e3f4f72f64dc69dbd521724912f07
I20240502 08:17:01.252154 25744 raft_consensus.cc:504] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:01.252367 25012 ts_tablet_manager.cc:1909] T 001e3f4f72f64dc69dbd521724912f07 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:01.252954 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1129) took 2420 ms. Trace:
I20240502 08:17:01.253221 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1207) took 1784 ms. Trace:
I20240502 08:17:01.253315 24180 rpcz_store.cc:276] 0502 08:16:59.469054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.469238 (+ 184us) service_pool.cc:225] Handling call
0502 08:17:01.253126 (+1783888us) ts_tablet_manager.cc:1120] Deleting tablet b1a2bcc6937c4a95b71103dbf8738702
0502 08:17:01.253134 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.253206 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1776181}
I20240502 08:17:01.253401 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8910bc67c6234a538c8160e7c020653c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:01.253580 24052 raft_consensus.cc:3037] T 8910bc67c6234a538c8160e7c020653c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.253691 24180 tablet_replica.cc:330] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:01.253845 24180 raft_consensus.cc:2229] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:01.253971 25744 leader_election.cc:288] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:01.254128 24180 raft_consensus.cc:2258] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:01.254298 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet ec10ff7251a5442a8ecc1881b34db74d (table default.Table_236150d3903d437d95f67b0edec36091 [id=1d7a43116fe74c35b505eccd165f99cd]) successfully deleted
W20240502 08:17:01.254493 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 0dae6a96776c477fb0ada8226d68ea25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0dae6a96776c477fb0ada8226d68ea25 already in progress: creating tablet
W20240502 08:17:01.254650 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 22bc17792cf040748ab672cfc6451e3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22bc17792cf040748ab672cfc6451e3c already in progress: creating tablet
W20240502 08:17:01.254788 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 4e32319fa9244ed2a323b2b3f5940748 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e32319fa9244ed2a323b2b3f5940748 already in progress: creating tablet
W20240502 08:17:01.254913 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 7ae9c817c7be42da8c689267c7eb5a06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ae9c817c7be42da8c689267c7eb5a06 already in progress: creating tablet
W20240502 08:17:01.255059 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet b1a2bcc6937c4a95b71103dbf8738702 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1a2bcc6937c4a95b71103dbf8738702 already in progress: creating tablet
I20240502 08:17:01.255956 24052 raft_consensus.cc:2454] T 8910bc67c6234a538c8160e7c020653c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:17:01.256359 23723 leader_election.cc:302] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:17:01.256654 25744 raft_consensus.cc:2781] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:01.256778 25744 raft_consensus.cc:686] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:17:01.257058 25744 consensus_queue.cc:237] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:01.257680 24173 consensus_queue.cc:260] T 41f2e98ef2884b05994d02114e992e80 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.257977 24180 ts_tablet_manager.cc:1875] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:01.262147 24173 ts_tablet_manager.cc:1398] T 41f2e98ef2884b05994d02114e992e80 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240502 08:17:01.262450 24173 tablet_bootstrap.cc:492] T edd7244b9f5741a8aa1c6835fe544676 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:01.263020 24180 ts_tablet_manager.cc:1888] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:01.253656 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8910bc67c6234a538c8160e7c020653c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:01.263198 23920 raft_consensus.cc:3037] T 8910bc67c6234a538c8160e7c020653c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.263998 24173 tablet_bootstrap.cc:654] T edd7244b9f5741a8aa1c6835fe544676 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.265504 23920 raft_consensus.cc:2454] T 8910bc67c6234a538c8160e7c020653c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:17:01.254825 24175 tablet_bootstrap.cc:492] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:01.266358 24175 ts_tablet_manager.cc:1367] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
I20240502 08:17:01.267138 24175 raft_consensus.cc:348] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.267372 24175 raft_consensus.cc:374] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.267457 24175 raft_consensus.cc:729] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.267699 24175 consensus_queue.cc:260] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.268261 24175 ts_tablet_manager.cc:1398] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:01.268522 24175 tablet_bootstrap.cc:492] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:01.269857 24175 tablet_bootstrap.cc:654] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.270687 25012 rpcz_store.cc:276] 0502 08:16:58.832855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.832916 (+ 61us) service_pool.cc:225] Handling call
0502 08:17:01.221693 (+2388777us) ts_tablet_manager.cc:1120] Deleting tablet 001e3f4f72f64dc69dbd521724912f07
0502 08:17:01.221702 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.233016 (+ 11314us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.251885 (+ 18869us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.252937 (+ 1052us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2388621}
I20240502 08:17:01.271142 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 001e3f4f72f64dc69dbd521724912f07 (table default.Table_d7cff78803d44e5aba3b50cd8ff11e92 [id=5c21117955d946418c656330984ae747]) successfully deleted
I20240502 08:17:01.271540 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1133) took 2393 ms. Trace:
I20240502 08:17:01.271662 25012 rpcz_store.cc:276] 0502 08:16:58.877682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.877743 (+ 61us) service_pool.cc:225] Handling call
0502 08:17:01.271428 (+2393685us) ts_tablet_manager.cc:1120] Deleting tablet 990d0cc863ee4f49a2fc2fffb217c4d7
0502 08:17:01.271437 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.271525 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2393468}
W20240502 08:17:01.272105 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 990d0cc863ee4f49a2fc2fffb217c4d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 990d0cc863ee4f49a2fc2fffb217c4d7 already in progress: creating tablet
I20240502 08:17:01.272459 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1132) took 2394 ms. Trace:
I20240502 08:17:01.272563 25012 rpcz_store.cc:276] 0502 08:16:58.877545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.877834 (+ 289us) service_pool.cc:225] Handling call
0502 08:17:01.272367 (+2394533us) ts_tablet_manager.cc:1120] Deleting tablet e2792c45255c42adaeefb7aef4e25715
0502 08:17:01.272375 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.272445 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2394297}
W20240502 08:17:01.272944 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet e2792c45255c42adaeefb7aef4e25715 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2792c45255c42adaeefb7aef4e25715 already in progress: creating tablet
I20240502 08:17:01.273286 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1134) took 2395 ms. Trace:
I20240502 08:17:01.273597 25012 rpcz_store.cc:276] 0502 08:16:58.877788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.878985 (+ 1197us) service_pool.cc:225] Handling call
0502 08:17:01.273194 (+2394209us) ts_tablet_manager.cc:1120] Deleting tablet 3ad53633ce57478194437f95135d81de
0502 08:17:01.273205 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.273272 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2394031}
W20240502 08:17:01.273975 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 3ad53633ce57478194437f95135d81de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ad53633ce57478194437f95135d81de already in progress: creating tablet
I20240502 08:17:01.276099 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1135) took 2299 ms. Trace:
I20240502 08:17:01.276212 25012 rpcz_store.cc:276] 0502 08:16:58.976756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.985539 (+ 8783us) service_pool.cc:225] Handling call
0502 08:17:01.276000 (+2290461us) ts_tablet_manager.cc:1120] Deleting tablet 58f538766e934282932fd4ab41b49831
0502 08:17:01.276009 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.276084 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2263863}
W20240502 08:17:01.276731 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 58f538766e934282932fd4ab41b49831 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58f538766e934282932fd4ab41b49831 already in progress: creating tablet
I20240502 08:17:01.277189 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1136) took 2150 ms. Trace:
I20240502 08:17:01.277278 25012 rpcz_store.cc:276] 0502 08:16:59.127075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.129619 (+ 2544us) service_pool.cc:225] Handling call
0502 08:17:01.277077 (+2147458us) ts_tablet_manager.cc:1120] Deleting tablet f03473f7db3e411186f0db827a6630f7
0502 08:17:01.277099 (+ 22us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.277176 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2147310}
W20240502 08:17:01.277770 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet f03473f7db3e411186f0db827a6630f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f03473f7db3e411186f0db827a6630f7 already in progress: creating tablet
I20240502 08:17:01.278088 25012 tablet_replica.cc:330] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:01.278411 25012 raft_consensus.cc:2229] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:01.278754 25012 raft_consensus.cc:2258] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:01.279557 25012 ts_tablet_manager.cc:1875] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:01.271261 25795 tablet.cc:1722] T 63afd6e4fee24edeaba1209105f1c3e6 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:01.283660 24180 log.cc:1196] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/9467974d67c64c7eb5e488ac51f1240d
I20240502 08:17:01.290119 24180 ts_tablet_manager.cc:1909] T 9467974d67c64c7eb5e488ac51f1240d P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:01.290867 24175 tablet_bootstrap.cc:492] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:01.292011 24175 ts_tablet_manager.cc:1367] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240502 08:17:01.293937 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet ff4a333c6dd44a13bc267e885bb9e487
I20240502 08:17:01.294350 23608 catalog_manager.cc:6554] ReplaceTablet: tablet ff4a333c6dd44a13bc267e885bb9e487 deleted and replaced by tablet dba5ed2797d64947a0b1097ea3cecb0a
I20240502 08:17:01.294706 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_497c6963c9c94ff391bc3c47722ebbe4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:01.295116 24175 raft_consensus.cc:348] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:17:01.295213 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_497c6963c9c94ff391bc3c47722ebbe4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:01.295293 24175 raft_consensus.cc:374] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.295354 24175 raft_consensus.cc:729] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.295540 24175 consensus_queue.cc:260] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.296066 24175 ts_tablet_manager.cc:1398] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20240502 08:17:01.299278 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1223) took 1776 ms. Trace:
I20240502 08:17:01.300211 24180 rpcz_store.cc:276] 0502 08:16:59.523014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.529627 (+ 6613us) service_pool.cc:225] Handling call
0502 08:17:01.253655 (+1724028us) ts_tablet_manager.cc:1120] Deleting tablet 9467974d67c64c7eb5e488ac51f1240d
0502 08:17:01.253662 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.260495 (+ 6833us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.262947 (+ 2452us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.299257 (+ 36310us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1723890}
I20240502 08:17:01.300789 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1224) took 1772 ms. Trace:
I20240502 08:17:01.300962 24180 rpcz_store.cc:276] 0502 08:16:59.528258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.529966 (+ 1708us) service_pool.cc:225] Handling call
0502 08:17:01.300665 (+1770699us) ts_tablet_manager.cc:1120] Deleting tablet fd32de329009465c89a4917e2716d3e9
0502 08:17:01.300674 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.300776 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1770302}
I20240502 08:17:01.301326 24180 tablet_replica.cc:330] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:01.301532 24180 raft_consensus.cc:2229] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:01.302348 24173 tablet_bootstrap.cc:492] T edd7244b9f5741a8aa1c6835fe544676 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:01.302541 24173 ts_tablet_manager.cc:1367] T edd7244b9f5741a8aa1c6835fe544676 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.001s
I20240502 08:17:01.303192 24173 raft_consensus.cc:348] T edd7244b9f5741a8aa1c6835fe544676 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.303419 24173 raft_consensus.cc:374] T edd7244b9f5741a8aa1c6835fe544676 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.303503 24173 raft_consensus.cc:729] T edd7244b9f5741a8aa1c6835fe544676 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.303753 24173 consensus_queue.cc:260] T edd7244b9f5741a8aa1c6835fe544676 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.304265 24173 ts_tablet_manager.cc:1398] T edd7244b9f5741a8aa1c6835fe544676 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:01.304558 24173 tablet_bootstrap.cc:492] T 0501575a84ec454780f48574bf3155fe P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:01.307812 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 9467974d67c64c7eb5e488ac51f1240d (table default.Table_c1886293aee349b2b9e7177c1cfd6d26 [id=0853ffc4d48549bc93e251a5dbc55901]) successfully deleted
W20240502 08:17:01.308049 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet fd32de329009465c89a4917e2716d3e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd32de329009465c89a4917e2716d3e9 already in progress: creating tablet
I20240502 08:17:01.310531 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4a333c6dd44a13bc267e885bb9e487 with delete_type TABLET_DATA_DELETED (replaced by tablet dba5ed2797d64947a0b1097ea3cecb0a) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:01.313584 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4a333c6dd44a13bc267e885bb9e487 with delete_type TABLET_DATA_DELETED (replaced by tablet dba5ed2797d64947a0b1097ea3cecb0a) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:01.312662 24173 tablet_bootstrap.cc:654] T 0501575a84ec454780f48574bf3155fe P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.316156 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_312f784e0e4b43ec8666607e58f62ca3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:01.316681 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_312f784e0e4b43ec8666607e58f62ca3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:01.316922 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_01f87df24c614973b564898fe3bfe511" } new_table_name: "default.Table_c8c6e4da2a29415d9d64c129ec768aeb" modify_external_catalogs: true
W20240502 08:17:01.317705 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet ff4a333c6dd44a13bc267e885bb9e487 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff4a333c6dd44a13bc267e885bb9e487 already in progress: creating tablet
W20240502 08:17:01.318231 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1218fd56b8a4e88827f275f952ad782 consensus state has no leader
I20240502 08:17:01.318593 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1dcd211756884e2abc59e20c039a4566"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:01.318961 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1dcd211756884e2abc59e20c039a4566 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:01.319010 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet e0bd774640464d2a92b2e71126c83883 from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:01.319576 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_299008ee28694f2395585f3ce567a641"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:01.319945 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_299008ee28694f2395585f3ce567a641 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:01.327139 25405 raft_consensus.cc:482] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:01.327280 25405 raft_consensus.cc:504] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.327924 25405 leader_election.cc:288] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:01.329735 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63afd6e4fee24edeaba1209105f1c3e6" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:01.341616 24175 tablet_bootstrap.cc:492] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:01.343412 24175 tablet_bootstrap.cc:654] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.363873 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_78dbedcd5bd64c8ba2b168eb29d9ca36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:01.364434 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78dbedcd5bd64c8ba2b168eb29d9ca36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:01.369668 23607 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 63afd6e4fee24edeaba1209105f1c3e6 (table default.Table_9f3ea2fdcdc2477b89d509e868ffe4de [id=84e36915119b4125a1b0d64b2d1647fa]). Expected version 1 got 0
I20240502 08:17:01.370559 23607 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 665bbee2d15b4112b651844431535d0e (table default.Table_64f173e9aa5947069cae45a6cb8dd18f [id=427121bc94c247eaa315b6c830770d61]). Expected version 3 got 0
I20240502 08:17:01.371716 23603 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet e0b0d4871dd24607854100a4ef762e25 (table default.Table_5301ffccbac743618a78289c2c1415e1 [id=910c7cae36264e39bbfca0be919538e0]). Expected version 1 got 0
I20240502 08:17:01.375552 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_15060d38700048cf9cd741f97742efe9" } new_table_name: "default.Table_8d44ca108ddc47fc92c9f49e48304f1b" modify_external_catalogs: true
I20240502 08:17:01.377342 23603 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 63afd6e4fee24edeaba1209105f1c3e6 (table default.Table_9f3ea2fdcdc2477b89d509e868ffe4de [id=84e36915119b4125a1b0d64b2d1647fa]). Expected version 1 got 0
I20240502 08:17:01.378664 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5f11f4a4e19542c3bb6267e4e858d5e9" } modify_external_catalogs: true
I20240502 08:17:01.378806 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5f11f4a4e19542c3bb6267e4e858d5e9" } modify_external_catalogs: true
I20240502 08:17:01.382766 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet 9467974d67c64c7eb5e488ac51f1240d with delete_type TABLET_DATA_DELETED (replaced by tablet 29f6c3cea4b14364b6d073a92171119d) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:01.383031 24181 tablet_replica.cc:330] T 9467974d67c64c7eb5e488ac51f1240d P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:01.383232 24181 raft_consensus.cc:2229] T 9467974d67c64c7eb5e488ac51f1240d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:01.383376 24181 raft_consensus.cc:2258] T 9467974d67c64c7eb5e488ac51f1240d P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:01.384454 24181 ts_tablet_manager.cc:1875] T 9467974d67c64c7eb5e488ac51f1240d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:01.386453 23601 catalog_manager.cc:5484] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66). New cstate: current_term: 1 leader_uuid: "5b89a1f0485a44b7ac1710e2369a809b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:01.386879 23601 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 96b8f21c64e343d88d84e9acd0bc69b4 (table default.Table_1b91eb0181b74d5ea0d24f2b9731c155 [id=9ff8ea75a9bc4de194a70fe2088a94bc]). Expected version 2 got 0
I20240502 08:17:01.387116 23601 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 0501575a84ec454780f48574bf3155fe (table default.Table_540a424c4bfa4484a65f0b24140a4bbb [id=5cee878b3a31426c879a1bf95045bb09]). Expected version 4 got 0
I20240502 08:17:01.387298 23601 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 41f2e98ef2884b05994d02114e992e80 (table default.Table_50c546a17f484d3cb98a5ca4b3ed63e8 [id=b6dee89ecfb443d99267917d4df78e4d]). Expected version 3 got 0
I20240502 08:17:01.387467 23601 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 156c7356ece342aea0d8db667ebad89c (table default.Table_d8c6955b89724769bbfd81db2b5be455 [id=15b52f0688c24df2b6f3fda3732ceeb4]). Expected version 1 got 0
I20240502 08:17:01.387626 23601 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 63afd6e4fee24edeaba1209105f1c3e6 (table default.Table_9f3ea2fdcdc2477b89d509e868ffe4de [id=84e36915119b4125a1b0d64b2d1647fa]). Expected version 1 got 0
I20240502 08:17:01.387732 23601 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 74b4c563d6bd4a44bb3bab58a3fcdc07 (table default.Table_2ee0f7ddd41e41ea84e25359cd3ed0a4 [id=a9c009feee0e486794a8500121fe5487]). Expected version 2 got 0
I20240502 08:17:01.394665 25494 raft_consensus.cc:482] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:01.394671 25755 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8a323e6b9429eaaf4e0c1698ad077" member_type: VOTER last_known_addr { host: "127.22.225.126" port: 43265 } } peers { permanent_uuid: "84f3c07a31414c4f8fb61c44dc9f16e1" member_type: VOTER last_known_addr { host: "127.22.225.125" port: 36189 } } peers { permanent_uuid: "f87df6a45ad94428a9aa6b2796a0fd25" member_type: VOTER last_known_addr { host: "127.22.225.124" port: 41745 } } }
I20240502 08:17:01.394855 25755 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:01.394856 25494 raft_consensus.cc:504] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.395592 25494 leader_election.cc:288] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:17:01.396320 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac50774b2dd241de9cae8096578d183e" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:01.397648 23861 leader_election.cc:341] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:01.401831 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 1d46ced6ac2e439b99fd6fa29e1a5a52 (DEFAULT_TABLE table=default.Table_5d3aa312157c4ed7baafaa4864bcc205 [id=00ecc7cedcb44c7d931bb2d5b9b6fdc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.401993 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 81a8101212df4f3e88bd322c23c17fd1 (DEFAULT_TABLE table=default.Table_1f74b893a9be40b6919d054cc22dbf98 [id=2aa334c9a83c4254893864c0f9f0e3be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.402248 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d46ced6ac2e439b99fd6fa29e1a5a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.402403 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 34c1043bf2684641a2ed13bec9e89de2 (DEFAULT_TABLE table=default.Table_f92f900ce0114ea6ac0e01a049d206e1 [id=47431ccdbb074dbf8f641eb673f9a837]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.402691 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34c1043bf2684641a2ed13bec9e89de2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:01.409497 23594 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.125:36189, user_credentials={real_user=slave}} blocked reactor thread for 47925.4us
I20240502 08:17:01.412629 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 0cff5d6a37ac45a5a9d3cea7a493e601 (DEFAULT_TABLE table=default.Table_8f820382d8484e16834f76af57d0ab45 [id=30a680241cb24e15ab8bad3b6d6b93ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.412997 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cff5d6a37ac45a5a9d3cea7a493e601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.401831 24024 tablet_service.cc:1467] Processing CreateTablet for tablet b292e04df02e4c11beca3c8169068038 (DEFAULT_TABLE table=default.Table_9e9d68e4681a4d8989286fc07bee3791 [id=2b1f87766d4844fd9efa8cb7998afb3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.415813 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b292e04df02e4c11beca3c8169068038. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.426723 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 91f51481a84d454390b4714ce12b882e (DEFAULT_TABLE table=default.Table_5556d39f8aa64dd8bfb68ace348e406d [id=e0ab69b1737d47a7bf944eecfc782962]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.427132 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f51481a84d454390b4714ce12b882e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.429723 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81a8101212df4f3e88bd322c23c17fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.434171 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 91f51481a84d454390b4714ce12b882e (DEFAULT_TABLE table=default.Table_5556d39f8aa64dd8bfb68ace348e406d [id=e0ab69b1737d47a7bf944eecfc782962]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.434382 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 34c1043bf2684641a2ed13bec9e89de2 (DEFAULT_TABLE table=default.Table_f92f900ce0114ea6ac0e01a049d206e1 [id=47431ccdbb074dbf8f641eb673f9a837]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.434553 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f51481a84d454390b4714ce12b882e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.434716 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 81a8101212df4f3e88bd322c23c17fd1 (DEFAULT_TABLE table=default.Table_1f74b893a9be40b6919d054cc22dbf98 [id=2aa334c9a83c4254893864c0f9f0e3be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.434849 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 0cff5d6a37ac45a5a9d3cea7a493e601 (DEFAULT_TABLE table=default.Table_8f820382d8484e16834f76af57d0ab45 [id=30a680241cb24e15ab8bad3b6d6b93ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.435132 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81a8101212df4f3e88bd322c23c17fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.435338 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 1be6de4fae9a4c77a529a76da0b2bddf (DEFAULT_TABLE table=default.Table_5c50ed61204d4468990c37540b069f08 [id=30ee06ad3e344bdb87c7dfcc84479054]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.435621 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be6de4fae9a4c77a529a76da0b2bddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.434171 23898 tablet_service.cc:1467] Processing CreateTablet for tablet b292e04df02e4c11beca3c8169068038 (DEFAULT_TABLE table=default.Table_9e9d68e4681a4d8989286fc07bee3791 [id=2b1f87766d4844fd9efa8cb7998afb3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.437726 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34c1043bf2684641a2ed13bec9e89de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.437951 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b292e04df02e4c11beca3c8169068038. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.439513 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cff5d6a37ac45a5a9d3cea7a493e601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.444770 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 1d46ced6ac2e439b99fd6fa29e1a5a52 (DEFAULT_TABLE table=default.Table_5d3aa312157c4ed7baafaa4864bcc205 [id=00ecc7cedcb44c7d931bb2d5b9b6fdc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.445158 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d46ced6ac2e439b99fd6fa29e1a5a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.450567 24055 raft_consensus.cc:3037] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.454349 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 1be6de4fae9a4c77a529a76da0b2bddf (DEFAULT_TABLE table=default.Table_5c50ed61204d4468990c37540b069f08 [id=30ee06ad3e344bdb87c7dfcc84479054]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.454684 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be6de4fae9a4c77a529a76da0b2bddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.459444 24175 tablet_bootstrap.cc:492] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:01.310585 23754 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4a333c6dd44a13bc267e885bb9e487 with delete_type TABLET_DATA_DELETED (replaced by tablet dba5ed2797d64947a0b1097ea3cecb0a) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:01.328493 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63afd6e4fee24edeaba1209105f1c3e6" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:01.460253 23993 leader_election.cc:302] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:01.460547 24175 ts_tablet_manager.cc:1367] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.164s user 0.001s sys 0.001s
I20240502 08:17:01.397246 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac50774b2dd241de9cae8096578d183e" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:01.460908 23784 raft_consensus.cc:2454] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:01.413486 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet 9467974d67c64c7eb5e488ac51f1240d with delete_type TABLET_DATA_DELETED (replaced by tablet 29f6c3cea4b14364b6d073a92171119d) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:01.416620 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 91f51481a84d454390b4714ce12b882e (DEFAULT_TABLE table=default.Table_5556d39f8aa64dd8bfb68ace348e406d [id=e0ab69b1737d47a7bf944eecfc782962]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.461447 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f51481a84d454390b4714ce12b882e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.461449 23858 leader_election.cc:302] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:01.461712 24175 raft_consensus.cc:348] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.461927 24175 raft_consensus.cc:374] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.462020 25494 raft_consensus.cc:2781] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:01.462112 25494 raft_consensus.cc:482] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:01.462185 25494 raft_consensus.cc:3037] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.464519 25494 raft_consensus.cc:504] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.465377 25494 leader_election.cc:288] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:17:01.418653 23760 tablet_service.cc:1467] Processing CreateTablet for tablet b292e04df02e4c11beca3c8169068038 (DEFAULT_TABLE table=default.Table_9e9d68e4681a4d8989286fc07bee3791 [id=2b1f87766d4844fd9efa8cb7998afb3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.465884 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b292e04df02e4c11beca3c8169068038. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.465906 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac50774b2dd241de9cae8096578d183e" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:17:01.466362 23861 leader_election.cc:341] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:01.466907 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac50774b2dd241de9cae8096578d183e" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:01.467061 23784 raft_consensus.cc:3037] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.469475 23784 raft_consensus.cc:2454] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:01.419276 23750 tablet_service.cc:1467] Processing CreateTablet for tablet 81a8101212df4f3e88bd322c23c17fd1 (DEFAULT_TABLE table=default.Table_1f74b893a9be40b6919d054cc22dbf98 [id=2aa334c9a83c4254893864c0f9f0e3be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.469854 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81a8101212df4f3e88bd322c23c17fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.419876 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 34c1043bf2684641a2ed13bec9e89de2 (DEFAULT_TABLE table=default.Table_f92f900ce0114ea6ac0e01a049d206e1 [id=47431ccdbb074dbf8f641eb673f9a837]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.473801 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34c1043bf2684641a2ed13bec9e89de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.474182 23858 leader_election.cc:302] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:01.462011 24175 raft_consensus.cc:729] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.474574 25494 raft_consensus.cc:2781] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:01.474701 25494 raft_consensus.cc:686] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:17:01.474601 24175 consensus_queue.cc:260] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.474928 25494 consensus_queue.cc:237] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.475296 24175 ts_tablet_manager.cc:1398] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 08:17:01.475598 24175 tablet_bootstrap.cc:492] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:01.420483 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 0cff5d6a37ac45a5a9d3cea7a493e601 (DEFAULT_TABLE table=default.Table_8f820382d8484e16834f76af57d0ab45 [id=30a680241cb24e15ab8bad3b6d6b93ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.477866 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cff5d6a37ac45a5a9d3cea7a493e601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.478289 24175 tablet_bootstrap.cc:654] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.421670 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 1d46ced6ac2e439b99fd6fa29e1a5a52 (DEFAULT_TABLE table=default.Table_5d3aa312157c4ed7baafaa4864bcc205 [id=00ecc7cedcb44c7d931bb2d5b9b6fdc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.426163 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 1be6de4fae9a4c77a529a76da0b2bddf (DEFAULT_TABLE table=default.Table_5c50ed61204d4468990c37540b069f08 [id=30ee06ad3e344bdb87c7dfcc84479054]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:01.481875 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d46ced6ac2e439b99fd6fa29e1a5a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.482107 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be6de4fae9a4c77a529a76da0b2bddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.489035 24175 tablet_bootstrap.cc:492] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:01.489238 24175 ts_tablet_manager.cc:1367] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.002s
I20240502 08:17:01.489950 24175 raft_consensus.cc:348] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.490214 24175 raft_consensus.cc:374] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.490339 24175 raft_consensus.cc:729] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.490613 24175 consensus_queue.cc:260] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.491420 24175 ts_tablet_manager.cc:1398] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:01.491737 24175 tablet_bootstrap.cc:492] T 18625a1d971c43549b9f80e6046efc9c P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:01.493256 24175 tablet_bootstrap.cc:654] T 18625a1d971c43549b9f80e6046efc9c P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.495344 24181 ts_tablet_manager.cc:1888] T 9467974d67c64c7eb5e488ac51f1240d P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:01.495484 24181 log.cc:1196] T 9467974d67c64c7eb5e488ac51f1240d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/9467974d67c64c7eb5e488ac51f1240d
I20240502 08:17:01.495819 24181 ts_tablet_manager.cc:1909] T 9467974d67c64c7eb5e488ac51f1240d P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:01.498966 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 9467974d67c64c7eb5e488ac51f1240d (table default.Table_c1886293aee349b2b9e7177c1cfd6d26 [id=0853ffc4d48549bc93e251a5dbc55901]) successfully deleted
I20240502 08:17:01.501665 24180 raft_consensus.cc:2258] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:01.502274 24180 ts_tablet_manager.cc:1875] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:01.505242 25832 raft_consensus.cc:482] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:01.505403 25832 raft_consensus.cc:504] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.506101 25832 leader_election.cc:288] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:17:01.506733 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4bea89eccaec4d049c0ec3900ed600c2 consensus state has no leader
W20240502 08:17:01.507154 23861 leader_election.cc:341] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: a4d151770ee44130adfd822518b9b017
W20240502 08:17:01.508040 23858 leader_election.cc:341] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: a4d151770ee44130adfd822518b9b017
I20240502 08:17:01.508199 23858 leader_election.cc:302] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:01.508426 25832 raft_consensus.cc:2726] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:01.513423 25743 raft_consensus.cc:482] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:01.513533 25743 raft_consensus.cc:504] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.514106 25743 leader_election.cc:288] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:01.514954 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b175e1b55bf4102804666b6fffccead" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:01.515286 25832 raft_consensus.cc:482] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:01.515430 25832 raft_consensus.cc:504] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:17:01.515861 25779 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 0.271s user 0.000s sys 0.000s
I20240502 08:17:01.515983 25832 leader_election.cc:288] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:01.516436 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b175e1b55bf4102804666b6fffccead" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:01.517261 23861 leader_election.cc:341] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: a5a5a410b6314818b10927e35032f95d
W20240502 08:17:01.517833 23723 leader_election.cc:341] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:17:01.520179 23723 leader_election.cc:341] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:01.520268 23723 leader_election.cc:302] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:01.520634 25743 raft_consensus.cc:2726] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:17:01.521430 23858 leader_election.cc:341] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: a5a5a410b6314818b10927e35032f95d
I20240502 08:17:01.521548 23858 leader_election.cc:302] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:01.521911 25832 raft_consensus.cc:2726] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:01.534144 25814 tablet.cc:1722] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:01.536697 25405 raft_consensus.cc:2726] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:01.538913 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_517b4d2b22924609b825632eaf88a0bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:01.539397 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_517b4d2b22924609b825632eaf88a0bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:01.540726 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_762d8c4aee05415b89c25e8b7b3e85b7" } modify_external_catalogs: true
I20240502 08:17:01.540906 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_762d8c4aee05415b89c25e8b7b3e85b7" } modify_external_catalogs: true
I20240502 08:17:01.542093 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_76ef356c8f6543ec93536b3079814cff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:01.542470 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76ef356c8f6543ec93536b3079814cff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:01.559345 25831 tablet.cc:1722] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:01.592196 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 2b9f0b0afc3c47759e0e94ca44cf9711
I20240502 08:17:01.593323 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b9f0b0afc3c47759e0e94ca44cf9711 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:17:01.594084 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 2b9f0b0afc3c47759e0e94ca44cf9711 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b9f0b0afc3c47759e0e94ca44cf9711 already in progress: creating tablet
I20240502 08:17:01.594614 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 336f0eb5085546dba763e0d5c5dd5324
I20240502 08:17:01.595726 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 336f0eb5085546dba763e0d5c5dd5324 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:17:01.596333 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 336f0eb5085546dba763e0d5c5dd5324 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 336f0eb5085546dba763e0d5c5dd5324 already in progress: creating tablet
I20240502 08:17:01.598470 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet 336f0eb5085546dba763e0d5c5dd5324 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:01.598711 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b9f0b0afc3c47759e0e94ca44cf9711 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:01.599802 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 4b52e5c136d84cf1bfc8fadea62491f6
I20240502 08:17:01.600721 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b52e5c136d84cf1bfc8fadea62491f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:17:01.601249 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 4b52e5c136d84cf1bfc8fadea62491f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b52e5c136d84cf1bfc8fadea62491f6 already in progress: creating tablet
I20240502 08:17:01.605248 23606 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 41f2e98ef2884b05994d02114e992e80 (table default.Table_50c546a17f484d3cb98a5ca4b3ed63e8 [id=b6dee89ecfb443d99267917d4df78e4d]). Expected version 3 got 0
I20240502 08:17:01.609201 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fe583cbf4af493e82d3ec0508ea0f0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:01.609424 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet edd7244b9f5741a8aa1c6835fe544676 with delete_type TABLET_DATA_DELETED (replaced by tablet 9eaf8d65f0b4472eb8ed0ef6fc722577) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:01.609613 24181 tablet_replica.cc:330] T 7fe583cbf4af493e82d3ec0508ea0f0b P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:01.609853 24181 raft_consensus.cc:2229] T 7fe583cbf4af493e82d3ec0508ea0f0b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:01.610013 24181 raft_consensus.cc:2258] T 7fe583cbf4af493e82d3ec0508ea0f0b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:01.610808 24181 ts_tablet_manager.cc:1875] T 7fe583cbf4af493e82d3ec0508ea0f0b P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:01.617311 24181 ts_tablet_manager.cc:1888] T 7fe583cbf4af493e82d3ec0508ea0f0b P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:01.617448 24181 log.cc:1196] T 7fe583cbf4af493e82d3ec0508ea0f0b P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/7fe583cbf4af493e82d3ec0508ea0f0b
I20240502 08:17:01.617939 24181 ts_tablet_manager.cc:1909] T 7fe583cbf4af493e82d3ec0508ea0f0b P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:01.619112 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 7fe583cbf4af493e82d3ec0508ea0f0b (table default.Table_3c1db2ea8e3d49efa8439ddf3a30734c [id=8dc5018c97434885a14739188b199dc0]) successfully deleted
I20240502 08:17:01.620317 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b52e5c136d84cf1bfc8fadea62491f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:01.620688 24181 tablet_replica.cc:330] T edd7244b9f5741a8aa1c6835fe544676 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:01.620954 24181 raft_consensus.cc:2229] T edd7244b9f5741a8aa1c6835fe544676 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:01.621208 24181 raft_consensus.cc:2258] T edd7244b9f5741a8aa1c6835fe544676 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:01.622097 24181 ts_tablet_manager.cc:1875] T edd7244b9f5741a8aa1c6835fe544676 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:01.624039 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_463d6892a6624bcaa91f7aba10334f94" } new_table_name: "default.Table_68d187a8edbd4c0a8c876c296b48ea81" modify_external_catalogs: true
I20240502 08:17:01.626909 23608 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 07a9d13d3a2d4e6dbc04998c3dd7e1a8 (table default.Table_d9f68c6b26d442bbbd485bb89ee5564c [id=7a49ccfdd6884281ba9b882debd04871]). Expected version 1 got 0
I20240502 08:17:01.627123 23608 catalog_manager.cc:5484] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:01.628329 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_655c02f28fab4cfaacd923e876e91078" } new_table_name: "default.Table_4df37371888444f4babbecd50e9b9efc" modify_external_catalogs: true
I20240502 08:17:01.629467 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_98b47a2ad25448c6b8db287180fbff73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:01.630048 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98b47a2ad25448c6b8db287180fbff73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:01.631135 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_088f2741fbb747c1b652fed2e838cc58" } modify_external_catalogs: true
I20240502 08:17:01.631279 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_088f2741fbb747c1b652fed2e838cc58" } modify_external_catalogs: true
I20240502 08:17:01.632344 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_19eec114db694534a82e5f2f867c02b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:01.632791 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19eec114db694534a82e5f2f867c02b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:01.578477 23920 raft_consensus.cc:3037] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.592911 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b9f0b0afc3c47759e0e94ca44cf9711 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:01.595566 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet 336f0eb5085546dba763e0d5c5dd5324 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:01.600467 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b52e5c136d84cf1bfc8fadea62491f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:01.603931 23904 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f1ab141585441db46d3027833932b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:01.619529 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 92adebc0c69e4a52b975253656132495 with delete_type TABLET_DATA_DELETED (replaced by tablet ae2b20ae3a014c85af33d5ee51b704e1) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:01.637380 25830 tablet.cc:1722] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20240502 08:17:01.669279 23861 consensus_peers.cc:485] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:01.669381 23784 raft_consensus.cc:1261] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Refusing update from remote peer 5b89a1f0485a44b7ac1710e2369a809b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:01.670022 25494 consensus_queue.cc:1035] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:01.672561 25012 ts_tablet_manager.cc:1888] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.3
I20240502 08:17:01.677704 25012 log.cc:1196] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/b811b81ddf2b4dc9b5e5f80331bff294
I20240502 08:17:01.679224 25012 ts_tablet_manager.cc:1909] T b811b81ddf2b4dc9b5e5f80331bff294 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:01.690502 25779 tablet.cc:1722] T 96b8f21c64e343d88d84e9acd0bc69b4 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:17:01.700162 25649 raft_consensus.cc:482] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:01.700337 25649 raft_consensus.cc:504] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.700996 25649 leader_election.cc:288] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:17:01.701508 23858 leader_election.cc:341] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 91f1ab141585441db46d3027833932b0
W20240502 08:17:01.701846 23861 leader_election.cc:341] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Not found: Tablet not found: 91f1ab141585441db46d3027833932b0
I20240502 08:17:01.701968 23861 leader_election.cc:302] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:01.702525 25649 raft_consensus.cc:2726] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:01.705101 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1137) took 2577 ms. Trace:
I20240502 08:17:01.707772 24174 tablet_bootstrap.cc:492] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:01.707916 24174 ts_tablet_manager.cc:1367] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.469s user 0.010s sys 0.096s
I20240502 08:17:01.708712 24174 raft_consensus.cc:348] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.708918 24174 raft_consensus.cc:374] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.708971 24174 raft_consensus.cc:729] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.709209 24174 consensus_queue.cc:260] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.709426 25849 tablet.cc:1722] T 74b4c563d6bd4a44bb3bab58a3fcdc07 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:17:01.710408 24174 ts_tablet_manager.cc:1398] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:01.710842 24174 tablet_bootstrap.cc:492] T 2b98cc8728c042f4b7f6496fcc5ef913 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:01.705199 25012 rpcz_store.cc:276] 0502 08:16:59.127346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.129520 (+ 2174us) service_pool.cc:225] Handling call
0502 08:17:01.278042 (+2148522us) ts_tablet_manager.cc:1120] Deleting tablet b811b81ddf2b4dc9b5e5f80331bff294
0502 08:17:01.278053 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.455047 (+176994us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.672421 (+217374us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.690426 (+ 18005us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2138786}
I20240502 08:17:01.711908 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet b811b81ddf2b4dc9b5e5f80331bff294 (table default.Table_01a5cb4b00a54d719c7f057e20bbb6e4 [id=134ba3221728463c9509f78ea3ea5418]) successfully deleted
I20240502 08:17:01.712273 25012 tablet_replica.cc:330] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:01.712483 25012 raft_consensus.cc:2229] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:01.712785 25012 raft_consensus.cc:2258] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:01.713281 25012 ts_tablet_manager.cc:1875] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:01.716764 24174 tablet_bootstrap.cc:654] T 2b98cc8728c042f4b7f6496fcc5ef913 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.717885 24173 tablet_bootstrap.cc:492] T 0501575a84ec454780f48574bf3155fe P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:01.718016 24173 ts_tablet_manager.cc:1367] T 0501575a84ec454780f48574bf3155fe P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.414s user 0.032s sys 0.054s
I20240502 08:17:01.718739 24173 raft_consensus.cc:348] T 0501575a84ec454780f48574bf3155fe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:01.718916 24173 raft_consensus.cc:374] T 0501575a84ec454780f48574bf3155fe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.718968 24173 raft_consensus.cc:729] T 0501575a84ec454780f48574bf3155fe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.719166 24173 consensus_queue.cc:260] T 0501575a84ec454780f48574bf3155fe P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:01.721527 24173 ts_tablet_manager.cc:1398] T 0501575a84ec454780f48574bf3155fe P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:01.721918 24173 tablet_bootstrap.cc:492] T eb96ad2289754e428a1c109b2d59c53b P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:01.724390 24173 tablet_bootstrap.cc:654] T eb96ad2289754e428a1c109b2d59c53b P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.729840 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 88bf97d0a0a54e828c3b0902976e0500
I20240502 08:17:01.731506 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet 88bf97d0a0a54e828c3b0902976e0500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:01.732704 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 88bf97d0a0a54e828c3b0902976e0500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:01.735805 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8fd21899dd67434986e4e8fb3b799ce1" } new_table_name: "default.Table_8a1af9c618b848a4aff10c5471e6887f" modify_external_catalogs: true
I20240502 08:17:01.738629 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 88bf97d0a0a54e828c3b0902976e0500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:01.741080 24173 tablet_bootstrap.cc:492] T eb96ad2289754e428a1c109b2d59c53b P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:01.741205 24173 ts_tablet_manager.cc:1367] T eb96ad2289754e428a1c109b2d59c53b P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.001s
I20240502 08:17:01.742197 24174 tablet_bootstrap.cc:492] T 2b98cc8728c042f4b7f6496fcc5ef913 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:01.742316 24174 ts_tablet_manager.cc:1367] T 2b98cc8728c042f4b7f6496fcc5ef913 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.004s
I20240502 08:17:01.743127 24174 raft_consensus.cc:348] T 2b98cc8728c042f4b7f6496fcc5ef913 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.743355 24174 raft_consensus.cc:374] T 2b98cc8728c042f4b7f6496fcc5ef913 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.743418 24174 raft_consensus.cc:729] T 2b98cc8728c042f4b7f6496fcc5ef913 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.743631 24174 consensus_queue.cc:260] T 2b98cc8728c042f4b7f6496fcc5ef913 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.745062 24174 ts_tablet_manager.cc:1398] T 2b98cc8728c042f4b7f6496fcc5ef913 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:17:01.746100 24174 tablet_bootstrap.cc:492] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:01.747917 25649 raft_consensus.cc:482] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:01.748067 25649 raft_consensus.cc:504] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.748626 25649 leader_election.cc:288] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:01.749195 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45aa0dd26524e0fba27e4eaf17d704e" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:01.749368 23784 raft_consensus.cc:2454] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:01.749526 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45aa0dd26524e0fba27e4eaf17d704e" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:01.750030 23861 leader_election.cc:341] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:01.751530 23858 leader_election.cc:302] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:01.751768 25357 raft_consensus.cc:2781] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:01.751852 25357 raft_consensus.cc:482] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:01.752202 25357 raft_consensus.cc:3037] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:17:01.752964 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17027bfd8c7a415591fdc3083bec83bf consensus state has no leader
I20240502 08:17:01.754755 25357 raft_consensus.cc:504] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.755302 25357 leader_election.cc:288] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:01.709728 25852 tablet.cc:1722] T 96b8f21c64e343d88d84e9acd0bc69b4 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:17:01.755571 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45aa0dd26524e0fba27e4eaf17d704e" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:01.755709 23784 raft_consensus.cc:3037] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.756840 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45aa0dd26524e0fba27e4eaf17d704e" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:17:01.757530 23861 leader_election.cc:341] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:01.758750 23784 raft_consensus.cc:2454] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:01.758955 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 0501575a84ec454780f48574bf3155fe (table default.Table_540a424c4bfa4484a65f0b24140a4bbb [id=5cee878b3a31426c879a1bf95045bb09]). Expected version 4 got 0
I20240502 08:17:01.759245 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 07a9d13d3a2d4e6dbc04998c3dd7e1a8 (table default.Table_d9f68c6b26d442bbbd485bb89ee5564c [id=7a49ccfdd6884281ba9b882debd04871]). Expected version 1 got 0
I20240502 08:17:01.759303 23858 leader_election.cc:302] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:01.759392 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet c45aa0dd26524e0fba27e4eaf17d704e (table default.Table_cbe61670c678416995bf27144ab702d9 [id=589deb067c2b4cd086d0277ecfed5126]). Expected version 2 got 0
I20240502 08:17:01.759531 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 6088f160a13c4ff09f72bed72f69eaf8 (table default.Table_851f246623304504aa9ddf2e6c5e9082 [id=4fecac7b0d9042b3b4f674f3fc1338ba]). Expected version 1 got 0
I20240502 08:17:01.759575 25357 raft_consensus.cc:2781] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:01.759699 25357 raft_consensus.cc:686] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:17:01.759699 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 63afd6e4fee24edeaba1209105f1c3e6 (table default.Table_9f3ea2fdcdc2477b89d509e868ffe4de [id=84e36915119b4125a1b0d64b2d1647fa]). Expected version 1 got 0
I20240502 08:17:01.759948 25357 consensus_queue.cc:237] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.759903 23609 catalog_manager.cc:5484] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66). New cstate: current_term: 1 leader_uuid: "5b89a1f0485a44b7ac1710e2369a809b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:01.760238 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet ac50774b2dd241de9cae8096578d183e (table default.Table_2e0fc2a0993e4208b05240b1ddc4d2ad [id=30defee3de2445f7bc5017fb23ab2ee7]). Expected version 1 got 0
I20240502 08:17:01.760423 23609 catalog_manager.cc:5484] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66). New cstate: current_term: 1 leader_uuid: "5b89a1f0485a44b7ac1710e2369a809b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:01.760752 24174 tablet_bootstrap.cc:654] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.760737 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 41f2e98ef2884b05994d02114e992e80 (table default.Table_50c546a17f484d3cb98a5ca4b3ed63e8 [id=b6dee89ecfb443d99267917d4df78e4d]). Expected version 3 got 0
I20240502 08:17:01.761988 24173 raft_consensus.cc:348] T eb96ad2289754e428a1c109b2d59c53b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:01.762151 24173 raft_consensus.cc:374] T eb96ad2289754e428a1c109b2d59c53b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.762188 24173 raft_consensus.cc:729] T eb96ad2289754e428a1c109b2d59c53b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
W20240502 08:17:01.762221 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3eb816c486a04481afe0832895acfc90 consensus state has no leader
I20240502 08:17:01.764618 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8da903edc43e49b98cb1e4f287d45633"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:01.765079 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8da903edc43e49b98cb1e4f287d45633 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:01.766388 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9dff2385175e41de9631cd27fbd848cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:01.766798 24173 consensus_queue.cc:260] T eb96ad2289754e428a1c109b2d59c53b P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:17:01.766970 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9dff2385175e41de9631cd27fbd848cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:01.767374 24173 ts_tablet_manager.cc:1398] T eb96ad2289754e428a1c109b2d59c53b P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20240502 08:17:01.769701 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_873744f861a94cc8af0fb5c3c80c944d" } new_table_name: "default.Table_98da23f792224757a75f9527a604c037" modify_external_catalogs: true
I20240502 08:17:01.771229 24173 tablet_bootstrap.cc:492] T ac50774b2dd241de9cae8096578d183e P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:01.771736 23576 master_runner.cc:186] Error getting master registration for 127.22.225.126:43265: Network error: Client connection negotiation failed: client connection to 127.22.225.126:43265: connect: Connection refused (error 111),
I20240502 08:17:01.771837 23576 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240502 08:17:01.776346 24173 tablet_bootstrap.cc:654] T ac50774b2dd241de9cae8096578d183e P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.780648 24175 tablet_bootstrap.cc:492] T 18625a1d971c43549b9f80e6046efc9c P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:01.780778 24175 ts_tablet_manager.cc:1367] T 18625a1d971c43549b9f80e6046efc9c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.289s user 0.036s sys 0.080s
I20240502 08:17:01.781028 24180 ts_tablet_manager.cc:1888] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:01.781196 24180 log.cc:1196] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/7fe583cbf4af493e82d3ec0508ea0f0b
I20240502 08:17:01.781545 24180 ts_tablet_manager.cc:1909] T 7fe583cbf4af493e82d3ec0508ea0f0b P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:01.781474 24175 raft_consensus.cc:348] T 18625a1d971c43549b9f80e6046efc9c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.781700 24175 raft_consensus.cc:374] T 18625a1d971c43549b9f80e6046efc9c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.781751 24175 raft_consensus.cc:729] T 18625a1d971c43549b9f80e6046efc9c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.781944 24175 consensus_queue.cc:260] T 18625a1d971c43549b9f80e6046efc9c P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.782702 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1222) took 2259 ms. Trace:
I20240502 08:17:01.782821 24180 rpcz_store.cc:276] 0502 08:16:59.522938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.530335 (+ 7397us) service_pool.cc:225] Handling call
0502 08:17:01.301291 (+1770956us) ts_tablet_manager.cc:1120] Deleting tablet 7fe583cbf4af493e82d3ec0508ea0f0b
0502 08:17:01.301298 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.671333 (+370035us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.780912 (+109579us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.782683 (+ 1771us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1770779}
I20240502 08:17:01.783316 24175 ts_tablet_manager.cc:1398] T 18625a1d971c43549b9f80e6046efc9c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:01.783676 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 7fe583cbf4af493e82d3ec0508ea0f0b (table default.Table_3c1db2ea8e3d49efa8439ddf3a30734c [id=8dc5018c97434885a14739188b199dc0]) successfully deleted
I20240502 08:17:01.784749 24175 tablet_bootstrap.cc:492] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:01.786099 24175 tablet_bootstrap.cc:654] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.786300 25846 raft_consensus.cc:482] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:01.786429 25846 raft_consensus.cc:504] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.787082 25846 leader_election.cc:288] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:01.787806 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a9d13d3a2d4e6dbc04998c3dd7e1a8" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:01.788363 23723 leader_election.cc:341] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:01.788612 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a9d13d3a2d4e6dbc04998c3dd7e1a8" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:01.788885 23920 raft_consensus.cc:2454] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:01.789238 23723 leader_election.cc:302] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:01.789582 25846 raft_consensus.cc:2781] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:01.789662 25846 raft_consensus.cc:482] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:01.789727 25846 raft_consensus.cc:3037] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.791822 25846 raft_consensus.cc:504] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.792330 25846 leader_election.cc:288] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:01.792879 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a9d13d3a2d4e6dbc04998c3dd7e1a8" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:17:01.793450 23723 leader_election.cc:341] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:01.793895 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a9d13d3a2d4e6dbc04998c3dd7e1a8" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:01.794539 23920 raft_consensus.cc:3037] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.797384 23920 raft_consensus.cc:2454] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:17:01.797820 23723 leader_election.cc:302] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:01.798209 25846 raft_consensus.cc:2781] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:01.798439 25846 raft_consensus.cc:686] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:17:01.798640 25846 consensus_queue.cc:237] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.802152 25846 raft_consensus.cc:482] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:01.802260 25846 raft_consensus.cc:504] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.802871 25846 leader_election.cc:288] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:01.802963 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1305dbdac3994224a047df139cf8d518" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:01.803174 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1305dbdac3994224a047df139cf8d518" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:17:01.803463 23723 leader_election.cc:341] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:01.803511 23920 raft_consensus.cc:2454] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:01.803967 23723 leader_election.cc:302] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:01.804425 25846 raft_consensus.cc:2781] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:01.804528 25846 raft_consensus.cc:482] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:01.804630 25846 raft_consensus.cc:3037] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.805013 24175 tablet_bootstrap.cc:492] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:01.805123 24175 ts_tablet_manager.cc:1367] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20240502 08:17:01.805716 24175 raft_consensus.cc:348] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.805867 24175 raft_consensus.cc:374] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.805918 24175 raft_consensus.cc:729] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.806144 24175 consensus_queue.cc:260] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.807744 25846 raft_consensus.cc:504] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.808138 24175 ts_tablet_manager.cc:1398] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:01.808633 25846 leader_election.cc:288] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:01.808634 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1305dbdac3994224a047df139cf8d518" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:01.809666 25846 raft_consensus.cc:482] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:17:01.809841 23723 leader_election.cc:341] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:01.809777 25846 raft_consensus.cc:504] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.810127 24175 tablet_bootstrap.cc:492] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:01.810487 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "665bbee2d15b4112b651844431535d0e" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:01.810892 23723 leader_election.cc:341] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:01.811242 25846 leader_election.cc:288] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:01.811465 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "665bbee2d15b4112b651844431535d0e" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:01.811688 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1305dbdac3994224a047df139cf8d518" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:01.811872 23920 raft_consensus.cc:3037] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:17:01.811890 23723 leader_election.cc:341] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:01.811975 23723 leader_election.cc:302] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:01.812307 25846 raft_consensus.cc:2726] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:01.812734 24175 tablet_bootstrap.cc:654] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.814386 23920 raft_consensus.cc:2454] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:17:01.814744 23723 leader_election.cc:302] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:01.815124 25846 raft_consensus.cc:2781] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:01.815259 25846 raft_consensus.cc:686] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:17:01.815488 25846 consensus_queue.cc:237] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.817176 24173 tablet_bootstrap.cc:492] T ac50774b2dd241de9cae8096578d183e P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:01.817324 24173 ts_tablet_manager.cc:1367] T ac50774b2dd241de9cae8096578d183e P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.000s
I20240502 08:17:01.818032 24173 raft_consensus.cc:348] T ac50774b2dd241de9cae8096578d183e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.818189 24173 raft_consensus.cc:374] T ac50774b2dd241de9cae8096578d183e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.818235 24173 raft_consensus.cc:729] T ac50774b2dd241de9cae8096578d183e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.826278 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1225) took 2257 ms. Trace:
I20240502 08:17:01.826442 24180 rpcz_store.cc:276] 0502 08:16:59.569300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.569505 (+ 205us) service_pool.cc:225] Handling call
0502 08:17:01.825913 (+2256408us) ts_tablet_manager.cc:1120] Deleting tablet 2a3e66ef4a914f009cca6d418f825d4e
0502 08:17:01.825927 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.826253 (+ 326us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2256138}
W20240502 08:17:01.827068 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 2a3e66ef4a914f009cca6d418f825d4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a3e66ef4a914f009cca6d418f825d4e already in progress: creating tablet
I20240502 08:17:01.827541 24180 tablet_replica.cc:330] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:01.827873 24180 raft_consensus.cc:2229] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:01.828119 24180 raft_consensus.cc:2258] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:01.828816 24180 ts_tablet_manager.cc:1875] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:01.829732 24173 consensus_queue.cc:260] T ac50774b2dd241de9cae8096578d183e P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.832085 24181 ts_tablet_manager.cc:1888] T edd7244b9f5741a8aa1c6835fe544676 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:01.832259 24181 log.cc:1196] T edd7244b9f5741a8aa1c6835fe544676 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/edd7244b9f5741a8aa1c6835fe544676
I20240502 08:17:01.832645 24181 ts_tablet_manager.cc:1909] T edd7244b9f5741a8aa1c6835fe544676 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:01.832065 24173 ts_tablet_manager.cc:1398] T ac50774b2dd241de9cae8096578d183e P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 08:17:01.833783 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet edd7244b9f5741a8aa1c6835fe544676 (table default.Table_4c6989f7e6bb4e99b447a3c1188f5b92 [id=9775ed3c642a4d7e81d5abc48acca163]) successfully deleted
W20240502 08:17:01.834127 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 88bf97d0a0a54e828c3b0902976e0500 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88bf97d0a0a54e828c3b0902976e0500 already in progress: creating tablet
I20240502 08:17:01.834614 24173 tablet_bootstrap.cc:492] T 63e645bb6e2948e091e08433944a52f4 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:01.836073 24173 tablet_bootstrap.cc:654] T 63e645bb6e2948e091e08433944a52f4 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.838152 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet c61243a38bfa4964b225eeae0e8cc28e
I20240502 08:17:01.838884 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet c61243a38bfa4964b225eeae0e8cc28e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:01.839202 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet c61243a38bfa4964b225eeae0e8cc28e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:17:01.839306 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet c61243a38bfa4964b225eeae0e8cc28e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c61243a38bfa4964b225eeae0e8cc28e already in progress: creating tablet
I20240502 08:17:01.840023 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7410c0888be547e69500e037718e3861"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:01.840617 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7410c0888be547e69500e037718e3861 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:01.840763 24173 tablet_bootstrap.cc:492] T 63e645bb6e2948e091e08433944a52f4 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:01.840865 24173 ts_tablet_manager.cc:1367] T 63e645bb6e2948e091e08433944a52f4 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:17:01.841432 24173 raft_consensus.cc:348] T 63e645bb6e2948e091e08433944a52f4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.841609 24173 raft_consensus.cc:374] T 63e645bb6e2948e091e08433944a52f4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.841720 24173 raft_consensus.cc:729] T 63e645bb6e2948e091e08433944a52f4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.841933 24173 consensus_queue.cc:260] T 63e645bb6e2948e091e08433944a52f4 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.842718 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet c61243a38bfa4964b225eeae0e8cc28e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:17:01.842973 23594 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 17027bfd8c7a415591fdc3083bec83bf (table default.Table_68d187a8edbd4c0a8c876c296b48ea81 [id=26e3cb7e495645b79b8d9b29118633b4], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 17027bfd8c7a415591fdc3083bec83bf consensus state has no leader
I20240502 08:17:01.843204 24173 ts_tablet_manager.cc:1398] T 63e645bb6e2948e091e08433944a52f4 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:01.844070 24173 tablet_bootstrap.cc:492] T 96b8f21c64e343d88d84e9acd0bc69b4 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:01.845798 24173 tablet_bootstrap.cc:654] T 96b8f21c64e343d88d84e9acd0bc69b4 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.850222 25843 raft_consensus.cc:482] T ac50774b2dd241de9cae8096578d183e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:01.850324 25843 raft_consensus.cc:504] T ac50774b2dd241de9cae8096578d183e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.850843 25843 leader_election.cc:288] T ac50774b2dd241de9cae8096578d183e P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:17:01.851383 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 86990bef52294256a3c4604415551fb8 consensus state has no leader
I20240502 08:17:01.851565 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac50774b2dd241de9cae8096578d183e" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:01.852178 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac50774b2dd241de9cae8096578d183e" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:01.852408 23784 raft_consensus.cc:2379] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 149ce744feab49b5aa5bcaa14bd6aba5 in current term 1: Already voted for candidate 5b89a1f0485a44b7ac1710e2369a809b in this term.
I20240502 08:17:01.852783 23993 leader_election.cc:302] T ac50774b2dd241de9cae8096578d183e P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:01.853339 25843 raft_consensus.cc:3037] T ac50774b2dd241de9cae8096578d183e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.854295 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_df7e0a371a964acc812387e7d12dd45a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:01.854813 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df7e0a371a964acc812387e7d12dd45a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:01.856120 25843 raft_consensus.cc:2726] T ac50774b2dd241de9cae8096578d183e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:01.856110 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6758c439d0264980a3c6493fe1607b36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:01.856603 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6758c439d0264980a3c6493fe1607b36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:01.856629 23429 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2vVFYb/build/asan/bin/kudu
/tmp/dist-test-task2vVFYb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.126:43265
--webserver_interface=127.22.225.126
--webserver_port=34035
--builtin_ntp_servers=127.22.225.84:40725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.225.126:43265,127.22.225.125:36189,127.22.225.124:41745
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240502 08:17:01.857494 25494 raft_consensus.cc:482] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:01.857681 25494 raft_consensus.cc:504] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.858249 25494 leader_election.cc:288] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:01.858753 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e11d405a2e4e518a10bf383e9ae6c4" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:01.859135 23861 leader_election.cc:341] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:01.859725 23608 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_9f3ea2fdcdc2477b89d509e868ffe4de [id=84e36915119b4125a1b0d64b2d1647fa] alter complete (version 1)
I20240502 08:17:01.861191 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6fc62448e8db451ebfaa3212bef8c660"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:01.861626 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6fc62448e8db451ebfaa3212bef8c660 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:01.862569 24173 tablet_bootstrap.cc:492] T 96b8f21c64e343d88d84e9acd0bc69b4 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:01.865460 24173 ts_tablet_manager.cc:1367] T 96b8f21c64e343d88d84e9acd0bc69b4 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20240502 08:17:01.866065 24173 raft_consensus.cc:348] T 96b8f21c64e343d88d84e9acd0bc69b4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:01.866235 24173 raft_consensus.cc:374] T 96b8f21c64e343d88d84e9acd0bc69b4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.866279 24173 raft_consensus.cc:729] T 96b8f21c64e343d88d84e9acd0bc69b4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.866393 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e11d405a2e4e518a10bf383e9ae6c4" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:01.866487 24173 consensus_queue.cc:260] T 96b8f21c64e343d88d84e9acd0bc69b4 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:01.866618 23784 raft_consensus.cc:2454] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:01.867306 23858 leader_election.cc:302] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:01.867590 25357 raft_consensus.cc:2781] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:01.867666 25357 raft_consensus.cc:482] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:01.867728 25357 raft_consensus.cc:3037] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.868994 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 42e11d405a2e4e518a10bf383e9ae6c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:01.869395 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 1305dbdac3994224a047df139cf8d518 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:01.870240 23904 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b98cc8728c042f4b7f6496fcc5ef913 with delete_type TABLET_DATA_DELETED (replaced by tablet 25c8a6aaabe2413d8ce88f54ce45a4c4) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:01.870579 24173 ts_tablet_manager.cc:1398] T 96b8f21c64e343d88d84e9acd0bc69b4 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 08:17:01.872249 23609 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 0501575a84ec454780f48574bf3155fe (table default.Table_540a424c4bfa4484a65f0b24140a4bbb [id=5cee878b3a31426c879a1bf95045bb09]). Expected version 4 got 0
I20240502 08:17:01.872483 23609 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet ac50774b2dd241de9cae8096578d183e (table default.Table_2e0fc2a0993e4208b05240b1ddc4d2ad [id=30defee3de2445f7bc5017fb23ab2ee7]). Expected version 1 got 0
I20240502 08:17:01.874900 24173 tablet_bootstrap.cc:492] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:01.876709 24173 tablet_bootstrap.cc:654] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.885165 25843 raft_consensus.cc:482] T 41f2e98ef2884b05994d02114e992e80 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:01.885267 25843 raft_consensus.cc:504] T 41f2e98ef2884b05994d02114e992e80 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.885847 25843 leader_election.cc:288] T 41f2e98ef2884b05994d02114e992e80 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:01.886263 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f2e98ef2884b05994d02114e992e80" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:01.886502 23784 raft_consensus.cc:2379] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 149ce744feab49b5aa5bcaa14bd6aba5 in current term 1: Already voted for candidate 5b89a1f0485a44b7ac1710e2369a809b in this term.
I20240502 08:17:01.886456 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f2e98ef2884b05994d02114e992e80" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:01.887053 23993 leader_election.cc:302] T 41f2e98ef2884b05994d02114e992e80 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:01.887351 25843 raft_consensus.cc:3037] T 41f2e98ef2884b05994d02114e992e80 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.870173 25357 raft_consensus.cc:504] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.888048 25357 leader_election.cc:288] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:01.888361 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e11d405a2e4e518a10bf383e9ae6c4" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:01.888518 23784 raft_consensus.cc:3037] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.890514 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e11d405a2e4e518a10bf383e9ae6c4" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:01.890825 23784 raft_consensus.cc:2454] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:01.891386 23858 leader_election.cc:302] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
W20240502 08:17:01.891549 23861 leader_election.cc:341] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:01.893095 25843 raft_consensus.cc:2726] T 41f2e98ef2884b05994d02114e992e80 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:01.894461 24173 tablet_bootstrap.cc:492] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:01.894469 25357 raft_consensus.cc:2781] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:01.894598 25357 raft_consensus.cc:686] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:17:01.894811 25357 consensus_queue.cc:237] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.895872 24173 ts_tablet_manager.cc:1367] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.003s
I20240502 08:17:01.896777 24173 raft_consensus.cc:348] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.897080 24173 raft_consensus.cc:374] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.897207 24173 raft_consensus.cc:729] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.897476 25649 raft_consensus.cc:482] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:01.897584 25649 raft_consensus.cc:504] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.898046 25649 leader_election.cc:288] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:01.898378 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b175e1b55bf4102804666b6fffccead" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:01.898470 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b175e1b55bf4102804666b6fffccead" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:01.898532 23784 raft_consensus.cc:2454] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:01.898861 23858 leader_election.cc:302] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
W20240502 08:17:01.899034 23861 leader_election.cc:341] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:01.899237 25649 raft_consensus.cc:2781] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:01.899293 25649 raft_consensus.cc:482] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:01.899338 25649 raft_consensus.cc:3037] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.901557 25649 raft_consensus.cc:504] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.902556 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b175e1b55bf4102804666b6fffccead" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:01.902567 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b175e1b55bf4102804666b6fffccead" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:01.902704 23784 raft_consensus.cc:3037] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:17:01.903005 23861 leader_election.cc:341] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:01.904702 25649 leader_election.cc:288] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:01.904943 23784 raft_consensus.cc:2454] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:01.904976 24173 consensus_queue.cc:260] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.905339 23858 leader_election.cc:302] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:01.906152 25649 raft_consensus.cc:2781] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:01.907155 25649 raft_consensus.cc:686] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:17:01.907413 25649 consensus_queue.cc:237] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.908669 24173 ts_tablet_manager.cc:1398] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240502 08:17:01.909873 24173 tablet_bootstrap.cc:492] T af6efeebaebc495aba017a922e243c39 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:01.911885 24173 tablet_bootstrap.cc:654] T af6efeebaebc495aba017a922e243c39 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.919085 23604 catalog_manager.cc:5484] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66). New cstate: current_term: 1 leader_uuid: "5b89a1f0485a44b7ac1710e2369a809b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:01.919519 23604 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet c45aa0dd26524e0fba27e4eaf17d704e (table default.Table_cbe61670c678416995bf27144ab702d9 [id=589deb067c2b4cd086d0277ecfed5126]). Expected version 2 got 0
I20240502 08:17:01.920619 24173 tablet_bootstrap.cc:492] T af6efeebaebc495aba017a922e243c39 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:01.920738 24173 ts_tablet_manager.cc:1367] T af6efeebaebc495aba017a922e243c39 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240502 08:17:01.921540 24173 raft_consensus.cc:348] T af6efeebaebc495aba017a922e243c39 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.921820 24173 raft_consensus.cc:374] T af6efeebaebc495aba017a922e243c39 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.921883 24173 raft_consensus.cc:729] T af6efeebaebc495aba017a922e243c39 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.922087 24173 consensus_queue.cc:260] T af6efeebaebc495aba017a922e243c39 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.922917 24174 tablet_bootstrap.cc:492] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:01.923046 24174 ts_tablet_manager.cc:1367] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.177s user 0.011s sys 0.019s
I20240502 08:17:01.923660 24174 raft_consensus.cc:348] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.923866 24174 raft_consensus.cc:374] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.923918 24174 raft_consensus.cc:729] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.924095 24174 consensus_queue.cc:260] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.924618 24174 ts_tablet_manager.cc:1398] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:01.924870 24174 tablet_bootstrap.cc:492] T fba8f6e017e848d59239ae98432b1837 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:01.924914 24173 ts_tablet_manager.cc:1398] T af6efeebaebc495aba017a922e243c39 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:17:01.925134 24173 tablet_bootstrap.cc:492] T 92adebc0c69e4a52b975253656132495 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:01.927063 24173 tablet_bootstrap.cc:654] T 92adebc0c69e4a52b975253656132495 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.927994 24174 tablet_bootstrap.cc:654] T fba8f6e017e848d59239ae98432b1837 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.931573 25012 ts_tablet_manager.cc:1888] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:01.935356 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bf29da2c486e479c8e6706cddeea1731" } modify_external_catalogs: true
I20240502 08:17:01.935508 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bf29da2c486e479c8e6706cddeea1731" } modify_external_catalogs: true
I20240502 08:17:01.936955 25012 log.cc:1196] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/ec10ff7251a5442a8ecc1881b34db74d
I20240502 08:17:01.937495 25843 raft_consensus.cc:482] T 96b8f21c64e343d88d84e9acd0bc69b4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:01.937629 25843 raft_consensus.cc:504] T 96b8f21c64e343d88d84e9acd0bc69b4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:01.938163 25843 leader_election.cc:288] T 96b8f21c64e343d88d84e9acd0bc69b4 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:01.939018 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b8f21c64e343d88d84e9acd0bc69b4" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:17:01.940382 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ef622696fe7435abe07808f5e8e918d consensus state has no leader
I20240502 08:17:01.939488 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b8f21c64e343d88d84e9acd0bc69b4" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:01.940989 23996 leader_election.cc:302] T 96b8f21c64e343d88d84e9acd0bc69b4 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:01.941358 25843 raft_consensus.cc:2726] T 96b8f21c64e343d88d84e9acd0bc69b4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:01.937508 25012 ts_tablet_manager.cc:1909] T ec10ff7251a5442a8ecc1881b34db74d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:01.945240 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f3099e422df24ee593db8bca89aba612" } modify_external_catalogs: true
I20240502 08:17:01.945380 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f3099e422df24ee593db8bca89aba612" } modify_external_catalogs: true
I20240502 08:17:01.950088 24174 tablet_bootstrap.cc:492] T fba8f6e017e848d59239ae98432b1837 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:01.950196 24174 ts_tablet_manager.cc:1367] T fba8f6e017e848d59239ae98432b1837 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.000s
I20240502 08:17:01.950806 24174 raft_consensus.cc:348] T fba8f6e017e848d59239ae98432b1837 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.950961 24174 raft_consensus.cc:374] T fba8f6e017e848d59239ae98432b1837 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.951009 24174 raft_consensus.cc:729] T fba8f6e017e848d59239ae98432b1837 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.951182 24174 consensus_queue.cc:260] T fba8f6e017e848d59239ae98432b1837 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.951689 24174 ts_tablet_manager.cc:1398] T fba8f6e017e848d59239ae98432b1837 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 08:17:01.951723 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet adce127b0ca7458cbb836b8a5a159a8a consensus state has no leader
I20240502 08:17:01.952248 24174 tablet_bootstrap.cc:492] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:01.953444 24174 tablet_bootstrap.cc:654] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.953404 24180 ts_tablet_manager.cc:1888] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:01.958272 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d731f4904f8649bdbc1369d357489e48" } modify_external_catalogs: true
I20240502 08:17:01.958424 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d731f4904f8649bdbc1369d357489e48" } modify_external_catalogs: true
I20240502 08:17:01.960132 25357 consensus_queue.cc:1035] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:01.953878 24180 log.cc:1196] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/18625a1d971c43549b9f80e6046efc9c
I20240502 08:17:01.962263 24173 tablet_bootstrap.cc:492] T 92adebc0c69e4a52b975253656132495 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:01.962415 24173 ts_tablet_manager.cc:1367] T 92adebc0c69e4a52b975253656132495 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.009s
I20240502 08:17:01.962639 24180 ts_tablet_manager.cc:1909] T 18625a1d971c43549b9f80e6046efc9c P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:01.963070 23603 master_service.cc:946] ReplaceTablet: received request to replace tablet 91167335ab4441a1a2fc8d70cebdd2eb from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:01.963061 24173 raft_consensus.cc:348] T 92adebc0c69e4a52b975253656132495 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.963240 24173 raft_consensus.cc:374] T 92adebc0c69e4a52b975253656132495 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:01.963313 24173 raft_consensus.cc:729] T 92adebc0c69e4a52b975253656132495 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.963507 24173 consensus_queue.cc:260] T 92adebc0c69e4a52b975253656132495 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:01.967235 24173 ts_tablet_manager.cc:1398] T 92adebc0c69e4a52b975253656132495 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:17:01.967499 24055 raft_consensus.cc:3037] T 0501575a84ec454780f48574bf3155fe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:01.967797 24173 tablet_bootstrap.cc:492] T f1510777d3cb4ae899542796b5a493ad P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:01.971904 24173 tablet_bootstrap.cc:654] T f1510777d3cb4ae899542796b5a493ad P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.972815 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1138) took 2633 ms. Trace:
I20240502 08:17:01.972939 25012 rpcz_store.cc:276] 0502 08:16:59.339521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.339575 (+ 54us) service_pool.cc:225] Handling call
0502 08:17:01.712205 (+2372630us) ts_tablet_manager.cc:1120] Deleting tablet ec10ff7251a5442a8ecc1881b34db74d
0502 08:17:01.712214 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.799133 (+ 86919us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.931463 (+132330us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.972791 (+ 41328us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2372441}
I20240502 08:17:01.973361 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1139) took 2549 ms. Trace:
I20240502 08:17:01.973423 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet ec10ff7251a5442a8ecc1881b34db74d (table default.Table_236150d3903d437d95f67b0edec36091 [id=1d7a43116fe74c35b505eccd165f99cd]) successfully deleted
I20240502 08:17:01.963594 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1227) took 2369 ms. Trace:
I20240502 08:17:01.973943 24180 rpcz_store.cc:276] 0502 08:16:59.594009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.594053 (+ 44us) service_pool.cc:225] Handling call
0502 08:17:01.827478 (+2233425us) ts_tablet_manager.cc:1120] Deleting tablet 18625a1d971c43549b9f80e6046efc9c
0502 08:17:01.827492 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.938810 (+111318us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.951471 (+ 12661us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:01.963578 (+ 12107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2012878}
I20240502 08:17:01.974689 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 18625a1d971c43549b9f80e6046efc9c (table default.Table_e02044df8b8c4ca9a9bdccaaf14a2eb4 [id=62ad2ebd7db541958a77f09c07ab946e]) successfully deleted
I20240502 08:17:01.975684 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1248) took 2289 ms. Trace:
I20240502 08:17:01.976068 24180 rpcz_store.cc:276] 0502 08:16:59.686287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.844376 (+158089us) service_pool.cc:225] Handling call
0502 08:17:01.975542 (+2131166us) ts_tablet_manager.cc:1120] Deleting tablet c44ce35c463f416794edccc7149aa0cf
0502 08:17:01.975550 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.975669 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2131031}
W20240502 08:17:01.976738 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet c44ce35c463f416794edccc7149aa0cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c44ce35c463f416794edccc7149aa0cf already in progress: creating tablet
I20240502 08:17:01.977716 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1234) took 2331 ms. Trace:
I20240502 08:17:01.978009 24180 rpcz_store.cc:276] 0502 08:16:59.646681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.646737 (+ 56us) service_pool.cc:225] Handling call
0502 08:17:01.977461 (+2330724us) ts_tablet_manager.cc:1120] Deleting tablet d9b67c8cd7f34cbba6401da59dec526c
0502 08:17:01.977613 (+ 152us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.977700 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2131943}
W20240502 08:17:01.978717 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet d9b67c8cd7f34cbba6401da59dec526c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9b67c8cd7f34cbba6401da59dec526c already in progress: creating tablet
I20240502 08:17:01.979043 24180 tablet_replica.cc:330] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:01.973453 25012 rpcz_store.cc:276] 0502 08:16:59.424117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.424439 (+ 322us) service_pool.cc:225] Handling call
0502 08:17:01.973221 (+2548782us) ts_tablet_manager.cc:1120] Deleting tablet 0dae6a96776c477fb0ada8226d68ea25
0502 08:17:01.973231 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.973344 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2548551}
I20240502 08:17:01.985424 23606 master_service.cc:946] ReplaceTablet: received request to replace tablet 3922e528a97a4287a0122a5b51035dcb from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:01.997218 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1140) took 2570 ms. Trace:
I20240502 08:17:01.997354 25012 rpcz_store.cc:276] 0502 08:16:59.427078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.427669 (+ 591us) service_pool.cc:225] Handling call
0502 08:17:01.997102 (+2569433us) ts_tablet_manager.cc:1120] Deleting tablet 22bc17792cf040748ab672cfc6451e3c
0502 08:17:01.997110 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.997204 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2569219}
W20240502 08:17:01.997707 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 0dae6a96776c477fb0ada8226d68ea25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0dae6a96776c477fb0ada8226d68ea25 already in progress: creating tablet
I20240502 08:17:01.997920 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1141) took 2541 ms. Trace:
W20240502 08:17:01.998004 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 22bc17792cf040748ab672cfc6451e3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22bc17792cf040748ab672cfc6451e3c already in progress: creating tablet
I20240502 08:17:01.998013 25012 rpcz_store.cc:276] 0502 08:16:59.456462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.456745 (+ 283us) service_pool.cc:225] Handling call
0502 08:17:01.997755 (+2541010us) ts_tablet_manager.cc:1120] Deleting tablet 4e32319fa9244ed2a323b2b3f5940748
0502 08:17:01.997763 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.997905 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2528876}
W20240502 08:17:01.998402 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 4e32319fa9244ed2a323b2b3f5940748 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e32319fa9244ed2a323b2b3f5940748 already in progress: creating tablet
I20240502 08:17:01.998731 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1142) took 2538 ms. Trace:
I20240502 08:17:01.998827 25012 rpcz_store.cc:276] 0502 08:16:59.460593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.460643 (+ 50us) service_pool.cc:225] Handling call
0502 08:17:01.998649 (+2538006us) ts_tablet_manager.cc:1120] Deleting tablet 7ae9c817c7be42da8c689267c7eb5a06
0502 08:17:01.998656 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.998718 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2529726}
W20240502 08:17:01.999301 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 7ae9c817c7be42da8c689267c7eb5a06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ae9c817c7be42da8c689267c7eb5a06 already in progress: creating tablet
I20240502 08:17:01.999639 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1143) took 2531 ms. Trace:
I20240502 08:17:01.999739 25012 rpcz_store.cc:276] 0502 08:16:59.468511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.468562 (+ 51us) service_pool.cc:225] Handling call
0502 08:17:01.999548 (+2530986us) ts_tablet_manager.cc:1120] Deleting tablet b1a2bcc6937c4a95b71103dbf8738702
0502 08:17:01.999556 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:01.999624 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2529482}
I20240502 08:17:02.000155 24173 tablet_bootstrap.cc:492] T f1510777d3cb4ae899542796b5a493ad P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
W20240502 08:17:02.000149 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet b1a2bcc6937c4a95b71103dbf8738702 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1a2bcc6937c4a95b71103dbf8738702 already in progress: creating tablet
I20240502 08:17:02.000314 24173 ts_tablet_manager.cc:1367] T f1510777d3cb4ae899542796b5a493ad P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.005s
I20240502 08:17:02.000450 25012 tablet_replica.cc:330] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:02.001171 24173 raft_consensus.cc:348] T f1510777d3cb4ae899542796b5a493ad P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.001391 24173 raft_consensus.cc:374] T f1510777d3cb4ae899542796b5a493ad P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.001447 24173 raft_consensus.cc:729] T f1510777d3cb4ae899542796b5a493ad P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.001578 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_42afc33a0fee436b9658e8a3e51eedfd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:02.001685 24173 consensus_queue.cc:260] T f1510777d3cb4ae899542796b5a493ad P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:17:02.002058 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42afc33a0fee436b9658e8a3e51eedfd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:02.002825 24173 ts_tablet_manager.cc:1398] T f1510777d3cb4ae899542796b5a493ad P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 08:17:02.003371 24173 tablet_bootstrap.cc:492] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:02.005334 24173 tablet_bootstrap.cc:654] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:01.982280 24175 tablet_bootstrap.cc:492] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:01.996183 25494 consensus_queue.cc:1035] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:01.996443 25832 raft_consensus.cc:482] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.005903 25832 raft_consensus.cc:504] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.006525 25832 leader_election.cc:288] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:02.007870 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088f160a13c4ff09f72bed72f69eaf8" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:02.007921 23608 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_540a424c4bfa4484a65f0b24140a4bbb [id=5cee878b3a31426c879a1bf95045bb09] alter complete (version 4)
W20240502 08:17:02.009127 23861 leader_election.cc:341] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.010475 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet eb96ad2289754e428a1c109b2d59c53b with delete_type TABLET_DATA_DELETED (replaced by tablet 6ee764398b364d029ae7f0f575a30d64) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:02.010494 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e645bb6e2948e091e08433944a52f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:02.010682 24181 tablet_replica.cc:330] T eb96ad2289754e428a1c109b2d59c53b P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:02.010823 24181 raft_consensus.cc:2229] T eb96ad2289754e428a1c109b2d59c53b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:02.010941 24181 raft_consensus.cc:2258] T eb96ad2289754e428a1c109b2d59c53b P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.011566 24181 ts_tablet_manager.cc:1875] T eb96ad2289754e428a1c109b2d59c53b P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.011919 24175 ts_tablet_manager.cc:1367] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.202s user 0.012s sys 0.010s
I20240502 08:17:02.012544 24175 raft_consensus.cc:348] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.012710 24175 raft_consensus.cc:374] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.012748 24175 raft_consensus.cc:729] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.012948 24175 consensus_queue.cc:260] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.013415 24175 ts_tablet_manager.cc:1398] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 08:17:02.013654 24175 tablet_bootstrap.cc:492] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:02.015926 24173 tablet_bootstrap.cc:492] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:02.016064 24173 ts_tablet_manager.cc:1367] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.004s
I20240502 08:17:02.016235 24175 tablet_bootstrap.cc:654] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.016736 24173 raft_consensus.cc:348] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.016903 24173 raft_consensus.cc:374] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.016952 24173 raft_consensus.cc:729] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.017123 24173 consensus_queue.cc:260] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.017591 24173 ts_tablet_manager.cc:1398] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:17:02.017803 24173 tablet_bootstrap.cc:492] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:02.018671 23608 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7aa8726847d34e7facc024d5a2287fcf" } modify_external_catalogs: true
I20240502 08:17:02.018813 23608 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7aa8726847d34e7facc024d5a2287fcf" } modify_external_catalogs: true
I20240502 08:17:02.019022 24173 tablet_bootstrap.cc:654] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.019806 25012 raft_consensus.cc:2229] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:02.019965 25012 raft_consensus.cc:2258] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.020413 25012 ts_tablet_manager.cc:1875] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.023082 24181 ts_tablet_manager.cc:1888] T eb96ad2289754e428a1c109b2d59c53b P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:02.023226 24181 log.cc:1196] T eb96ad2289754e428a1c109b2d59c53b P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/eb96ad2289754e428a1c109b2d59c53b
I20240502 08:17:02.023522 24181 ts_tablet_manager.cc:1909] T eb96ad2289754e428a1c109b2d59c53b P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:02.007057 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088f160a13c4ff09f72bed72f69eaf8" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:02.024370 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet eb96ad2289754e428a1c109b2d59c53b (table default.Table_bc1e9d346aa846f8abca7504185f7cdb [id=8d27878c6c8f4234908a2c78cb8bbd29]) successfully deleted
I20240502 08:17:02.024621 24181 tablet_replica.cc:330] T 63e645bb6e2948e091e08433944a52f4 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
W20240502 08:17:02.024621 23858 leader_election.cc:341] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.024708 23858 leader_election.cc:302] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:02.024770 24181 raft_consensus.cc:2229] T 63e645bb6e2948e091e08433944a52f4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:02.024917 24181 raft_consensus.cc:2258] T 63e645bb6e2948e091e08433944a52f4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.025060 25357 raft_consensus.cc:2726] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:02.025511 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_f7cad6a15dbc4e7ab7e60c73a2102d0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:02.025987 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7cad6a15dbc4e7ab7e60c73a2102d0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:02.029217 24181 ts_tablet_manager.cc:1875] T 63e645bb6e2948e091e08433944a52f4 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.033442 25357 raft_consensus.cc:482] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.033617 25357 raft_consensus.cc:504] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:17:02.034832 23858 leader_election.cc:341] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 92adebc0c69e4a52b975253656132495
I20240502 08:17:02.035243 25357 leader_election.cc:288] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:02.035830 24180 raft_consensus.cc:2229] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:02.036078 24180 raft_consensus.cc:2258] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.036712 24180 ts_tablet_manager.cc:1875] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.038050 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92adebc0c69e4a52b975253656132495" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:02.038209 24052 raft_consensus.cc:2454] T 92adebc0c69e4a52b975253656132495 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:02.038580 23861 leader_election.cc:302] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:02.038781 25357 raft_consensus.cc:2781] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:02.038836 25357 raft_consensus.cc:482] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:02.038885 25357 raft_consensus.cc:3037] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.040999 25814 tablet.cc:1722] T 0501575a84ec454780f48574bf3155fe P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240502 08:17:02.041545 25357 raft_consensus.cc:504] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.042851 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92adebc0c69e4a52b975253656132495" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:02.042994 24052 raft_consensus.cc:3037] T 92adebc0c69e4a52b975253656132495 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.044822 24181 ts_tablet_manager.cc:1888] T 63e645bb6e2948e091e08433944a52f4 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:02.044914 24181 log.cc:1196] T 63e645bb6e2948e091e08433944a52f4 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/63e645bb6e2948e091e08433944a52f4
I20240502 08:17:02.045148 24052 raft_consensus.cc:2454] T 92adebc0c69e4a52b975253656132495 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:02.045434 25832 raft_consensus.cc:482] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.045166 24181 ts_tablet_manager.cc:1909] T 63e645bb6e2948e091e08433944a52f4 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:02.045548 25832 raft_consensus.cc:504] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.046180 25832 leader_election.cc:288] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:02.046527 23861 leader_election.cc:302] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:17:02.047380 25832 raft_consensus.cc:2781] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:02.047497 25832 raft_consensus.cc:686] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:17:02.047760 25832 consensus_queue.cc:237] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.052096 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 91167335ab4441a1a2fc8d70cebdd2eb
I20240502 08:17:02.052506 23603 catalog_manager.cc:6554] ReplaceTablet: tablet 91167335ab4441a1a2fc8d70cebdd2eb deleted and replaced by tablet 1e57882a0a0a4ec5819a4108635062f0
I20240502 08:17:02.054190 25357 leader_election.cc:288] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:02.054611 23904 tablet_service.cc:1514] Processing DeleteTablet for tablet 91167335ab4441a1a2fc8d70cebdd2eb with delete_type TABLET_DATA_DELETED (replaced by tablet 1e57882a0a0a4ec5819a4108635062f0) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:02.054805 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 91167335ab4441a1a2fc8d70cebdd2eb with delete_type TABLET_DATA_DELETED (replaced by tablet 1e57882a0a0a4ec5819a4108635062f0) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:02.060204 24175 tablet_bootstrap.cc:492] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:02.060340 24175 ts_tablet_manager.cc:1367] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20240502 08:17:02.061093 24175 raft_consensus.cc:348] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.061251 24175 raft_consensus.cc:374] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.061317 24175 raft_consensus.cc:729] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.061511 24175 consensus_queue.cc:260] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.071095 25843 raft_consensus.cc:482] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.071301 25843 raft_consensus.cc:504] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.072099 25843 leader_election.cc:288] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:02.072713 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5f5597402374527bf0e5e50178df713" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:17:02.073257 23996 leader_election.cc:341] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: Tablet not RUNNING: STOPPED
I20240502 08:17:02.073954 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1510777d3cb4ae899542796b5a493ad" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:02.074172 24052 raft_consensus.cc:2454] T f1510777d3cb4ae899542796b5a493ad P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:02.074648 23861 leader_election.cc:302] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:17:02.074873 25357 raft_consensus.cc:2781] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:02.074949 25832 raft_consensus.cc:482] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.074960 25357 raft_consensus.cc:482] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:02.075062 25357 raft_consensus.cc:3037] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.075057 25832 raft_consensus.cc:504] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.075592 25832 leader_election.cc:288] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:02.077893 25357 raft_consensus.cc:504] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.078186 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b98cc8728c042f4b7f6496fcc5ef913" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:02.079581 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 63e645bb6e2948e091e08433944a52f4 (table default.Table_92a3c7cbfc16461ba5b08f262bc016d3 [id=47b70f46b4d345618054c69267d33c95]) successfully deleted
W20240502 08:17:02.078485 23861 consensus_peers.cc:485] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:02.079957 23861 leader_election.cc:341] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.079244 25012 ts_tablet_manager.cc:1888] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:02.080252 25012 log.cc:1196] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/e5f5597402374527bf0e5e50178df713
I20240502 08:17:02.080287 24173 tablet_bootstrap.cc:492] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:02.080431 24173 ts_tablet_manager.cc:1367] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.006s
I20240502 08:17:02.080816 25012 ts_tablet_manager.cc:1909] T e5f5597402374527bf0e5e50178df713 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:02.081200 24173 raft_consensus.cc:348] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.081411 24173 raft_consensus.cc:374] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.081494 24173 raft_consensus.cc:729] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
W20240502 08:17:02.082512 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 91167335ab4441a1a2fc8d70cebdd2eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91167335ab4441a1a2fc8d70cebdd2eb already in progress: creating tablet
I20240502 08:17:02.082831 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1510777d3cb4ae899542796b5a493ad" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:02.083184 24055 raft_consensus.cc:3037] T f1510777d3cb4ae899542796b5a493ad P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.085510 24055 raft_consensus.cc:2454] T f1510777d3cb4ae899542796b5a493ad P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:02.085721 24173 consensus_queue.cc:260] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.079682 24175 ts_tablet_manager.cc:1398] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240502 08:17:02.085903 23861 leader_election.cc:302] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:17:02.086519 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1152) took 2560 ms. Trace:
I20240502 08:17:02.086619 25832 raft_consensus.cc:2781] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:02.086738 25832 raft_consensus.cc:686] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:17:02.086984 25832 consensus_queue.cc:237] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.086738 25357 leader_election.cc:288] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:02.086591 25012 rpcz_store.cc:276] 0502 08:16:59.526223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.534041 (+ 7818us) service_pool.cc:225] Handling call
0502 08:17:02.000414 (+2466373us) ts_tablet_manager.cc:1120] Deleting tablet e5f5597402374527bf0e5e50178df713
0502 08:17:02.000422 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.037524 (+ 37102us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.079151 (+ 41627us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.086499 (+ 7348us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2466201}
I20240502 08:17:02.089089 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1159) took 2534 ms. Trace:
I20240502 08:17:02.089287 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet e5f5597402374527bf0e5e50178df713 (table default.Table_c44e58448899483fb6f36f8ee844e61d [id=ca51b24a648a4f2ea766c671f9058a1c]) successfully deleted
I20240502 08:17:02.089165 25012 rpcz_store.cc:276] 0502 08:16:59.554266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.554310 (+ 44us) service_pool.cc:225] Handling call
0502 08:17:02.088985 (+2534675us) ts_tablet_manager.cc:1120] Deleting tablet fd32de329009465c89a4917e2716d3e9
0502 08:17:02.088996 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.089075 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2471074}
W20240502 08:17:02.089888 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet fd32de329009465c89a4917e2716d3e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd32de329009465c89a4917e2716d3e9 already in progress: creating tablet
I20240502 08:17:02.090360 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1167) took 2381 ms. Trace:
I20240502 08:17:02.091025 25012 rpcz_store.cc:276] 0502 08:16:59.708882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.712107 (+ 3225us) service_pool.cc:225] Handling call
0502 08:17:02.090260 (+2378153us) ts_tablet_manager.cc:1120] Deleting tablet d9b67c8cd7f34cbba6401da59dec526c
0502 08:17:02.090268 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.090345 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2377932}
W20240502 08:17:02.092427 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet d9b67c8cd7f34cbba6401da59dec526c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9b67c8cd7f34cbba6401da59dec526c already in progress: creating tablet
I20240502 08:17:02.092943 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1160) took 2384 ms. Trace:
I20240502 08:17:02.094295 25012 rpcz_store.cc:276] 0502 08:16:59.708257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.725612 (+ 17355us) service_pool.cc:225] Handling call
0502 08:17:02.092725 (+2367113us) ts_tablet_manager.cc:1120] Deleting tablet 2a3e66ef4a914f009cca6d418f825d4e
0502 08:17:02.092856 (+ 131us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.092928 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2366793}
W20240502 08:17:02.095072 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 2a3e66ef4a914f009cca6d418f825d4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a3e66ef4a914f009cca6d418f825d4e already in progress: creating tablet
I20240502 08:17:02.095311 24175 tablet_bootstrap.cc:492] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:02.095386 25012 tablet_replica.cc:330] T a2da8b9d3f374d8a812336057c0ce7d2 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:02.095551 25012 raft_consensus.cc:2229] T a2da8b9d3f374d8a812336057c0ce7d2 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:02.095715 25012 raft_consensus.cc:2258] T a2da8b9d3f374d8a812336057c0ce7d2 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.096221 25012 ts_tablet_manager.cc:1875] T a2da8b9d3f374d8a812336057c0ce7d2 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.096268 24173 ts_tablet_manager.cc:1398] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 08:17:02.104502 24173 tablet_bootstrap.cc:492] T c45aa0dd26524e0fba27e4eaf17d704e P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:02.109236 24175 tablet_bootstrap.cc:654] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.110738 24173 tablet_bootstrap.cc:654] T c45aa0dd26524e0fba27e4eaf17d704e P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.113379 25012 ts_tablet_manager.cc:1888] T a2da8b9d3f374d8a812336057c0ce7d2 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:17:02.113480 25012 log.cc:1196] T a2da8b9d3f374d8a812336057c0ce7d2 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/a2da8b9d3f374d8a812336057c0ce7d2
W20240502 08:17:02.112757 23993 leader_election.cc:341] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: e5f5597402374527bf0e5e50178df713
I20240502 08:17:02.113664 23993 leader_election.cc:302] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:02.114087 25843 raft_consensus.cc:2726] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:02.115576 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 91167335ab4441a1a2fc8d70cebdd2eb with delete_type TABLET_DATA_DELETED (replaced by tablet 1e57882a0a0a4ec5819a4108635062f0) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:02.116725 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b98cc8728c042f4b7f6496fcc5ef913" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:02.116961 23787 raft_consensus.cc:2454] T 2b98cc8728c042f4b7f6496fcc5ef913 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:02.117134 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1510777d3cb4ae899542796b5a493ad" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:02.117316 23786 raft_consensus.cc:3037] T f1510777d3cb4ae899542796b5a493ad P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.118162 24175 tablet_bootstrap.cc:492] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:02.118376 24175 ts_tablet_manager.cc:1367] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.002s
I20240502 08:17:02.118548 25012 ts_tablet_manager.cc:1909] T a2da8b9d3f374d8a812336057c0ce7d2 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
W20240502 08:17:02.118685 23858 leader_election.cc:341] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 92adebc0c69e4a52b975253656132495
I20240502 08:17:02.119117 23858 leader_election.cc:302] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:02.119372 24175 raft_consensus.cc:348] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.119561 25832 raft_consensus.cc:2781] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:02.119689 25832 raft_consensus.cc:482] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:02.119776 25832 raft_consensus.cc:3037] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.122133 23786 raft_consensus.cc:2454] T f1510777d3cb4ae899542796b5a493ad P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:02.122828 25832 raft_consensus.cc:504] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.122658 23784 raft_consensus.cc:1261] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Refusing update from remote peer 5b89a1f0485a44b7ac1710e2369a809b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:02.122792 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1510777d3cb4ae899542796b5a493ad" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:02.123365 23783 raft_consensus.cc:2362] T f1510777d3cb4ae899542796b5a493ad P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1. Re-sending same reply.
I20240502 08:17:02.123445 25357 consensus_queue.cc:1035] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:02.123831 25832 leader_election.cc:288] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:02.124037 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b98cc8728c042f4b7f6496fcc5ef913" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:02.124461 24175 raft_consensus.cc:374] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 08:17:02.124511 23861 leader_election.cc:341] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.124538 24175 raft_consensus.cc:729] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.124760 24175 consensus_queue.cc:260] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.129179 24175 ts_tablet_manager.cc:1398] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 08:17:02.130781 24175 tablet_bootstrap.cc:492] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:02.131179 24055 raft_consensus.cc:3037] T 96b8f21c64e343d88d84e9acd0bc69b4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.132031 24175 tablet_bootstrap.cc:654] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.136607 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b98cc8728c042f4b7f6496fcc5ef913" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:02.136862 23786 raft_consensus.cc:3037] T 2b98cc8728c042f4b7f6496fcc5ef913 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.138489 24173 tablet_bootstrap.cc:492] T c45aa0dd26524e0fba27e4eaf17d704e P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:02.138711 24173 ts_tablet_manager.cc:1367] T c45aa0dd26524e0fba27e4eaf17d704e P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.001s
I20240502 08:17:02.139554 24173 raft_consensus.cc:348] T c45aa0dd26524e0fba27e4eaf17d704e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.139827 24173 raft_consensus.cc:374] T c45aa0dd26524e0fba27e4eaf17d704e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.140017 24173 raft_consensus.cc:729] T c45aa0dd26524e0fba27e4eaf17d704e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.140295 24173 consensus_queue.cc:260] T c45aa0dd26524e0fba27e4eaf17d704e P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.141060 24173 ts_tablet_manager.cc:1398] T c45aa0dd26524e0fba27e4eaf17d704e P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:02.141379 24173 tablet_bootstrap.cc:492] T 42e11d405a2e4e518a10bf383e9ae6c4 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:17:02.143430 23434 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 85079.1us
I20240502 08:17:02.145681 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7e555222aaac43b0ae44935aa7662dbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:02.146278 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e555222aaac43b0ae44935aa7662dbf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:02.147405 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1168) took 2438 ms. Trace:
I20240502 08:17:02.147485 25012 rpcz_store.cc:276] 0502 08:16:59.708953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.752724 (+ 43771us) service_pool.cc:225] Handling call
0502 08:17:02.095346 (+2342622us) ts_tablet_manager.cc:1120] Deleting tablet a2da8b9d3f374d8a812336057c0ce7d2
0502 08:17:02.095354 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.107928 (+ 12574us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.113287 (+ 5359us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.147381 (+ 34094us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26511,"tablet-delete.queue_time_us":2342426}
I20240502 08:17:02.147950 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c26660ee28674f03916ea5aecabf0a21" } new_table_name: "default.Table_222e38676f28492ba7f45fc6de942695" modify_external_catalogs: true
I20240502 08:17:02.149247 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4c6989f7e6bb4e99b447a3c1188f5b92" } new_table_name: "default.Table_06c3c496870e4b708537181d1d851737" modify_external_catalogs: true
I20240502 08:17:02.149780 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1180) took 2359 ms. Trace:
I20240502 08:17:02.149845 25012 rpcz_store.cc:276] 0502 08:16:59.790024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.790065 (+ 41us) service_pool.cc:225] Handling call
0502 08:17:02.149659 (+2359594us) ts_tablet_manager.cc:1120] Deleting tablet c44ce35c463f416794edccc7149aa0cf
0502 08:17:02.149669 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.149762 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2359255}
I20240502 08:17:02.150216 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1179) took 2360 ms. Trace:
I20240502 08:17:02.150282 25012 rpcz_store.cc:276] 0502 08:16:59.789944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.790821 (+ 877us) service_pool.cc:225] Handling call
0502 08:17:02.150132 (+2359311us) ts_tablet_manager.cc:1120] Deleting tablet 8c93c89e1341474eafb6030b95f1c68c
0502 08:17:02.150140 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.150203 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2358439}
I20240502 08:17:02.150599 25012 tablet_replica.cc:330] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:02.150751 25012 raft_consensus.cc:2229] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:02.150867 25012 raft_consensus.cc:2258] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.151309 25012 ts_tablet_manager.cc:1875] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.153203 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet a2da8b9d3f374d8a812336057c0ce7d2 (table default.Table_3edd27da6927478281e76361718425a3 [id=76b58ce209a24b9a95b405c1c50de3ca]) successfully deleted
W20240502 08:17:02.153406 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet c44ce35c463f416794edccc7149aa0cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c44ce35c463f416794edccc7149aa0cf already in progress: creating tablet
W20240502 08:17:02.153554 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 8c93c89e1341474eafb6030b95f1c68c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c93c89e1341474eafb6030b95f1c68c already in progress: creating tablet
I20240502 08:17:02.161458 25843 raft_consensus.cc:482] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.161602 25843 raft_consensus.cc:504] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.162168 25843 leader_election.cc:288] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:02.162698 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1305dbdac3994224a047df139cf8d518" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:02.162773 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1305dbdac3994224a047df139cf8d518" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:02.162882 23920 raft_consensus.cc:2379] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 149ce744feab49b5aa5bcaa14bd6aba5 in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:17:02.163012 23786 raft_consensus.cc:2454] T 2b98cc8728c042f4b7f6496fcc5ef913 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:02.163520 23993 leader_election.cc:302] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:02.164072 23858 leader_election.cc:302] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:02.164569 25832 raft_consensus.cc:2781] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:02.164719 25832 raft_consensus.cc:686] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:17:02.165004 25832 consensus_queue.cc:237] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.166532 25843 raft_consensus.cc:3037] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.169282 25843 raft_consensus.cc:2726] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:02.173995 25012 ts_tablet_manager.cc:1888] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:02.174100 25012 log.cc:1196] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/63e645bb6e2948e091e08433944a52f4
I20240502 08:17:02.174383 25012 ts_tablet_manager.cc:1909] T 63e645bb6e2948e091e08433944a52f4 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:02.177070 24174 tablet_bootstrap.cc:492] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:02.177294 24174 ts_tablet_manager.cc:1367] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.225s user 0.068s sys 0.095s
I20240502 08:17:02.178284 24174 raft_consensus.cc:348] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.178551 24174 raft_consensus.cc:374] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.178654 24174 raft_consensus.cc:729] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.178941 24174 consensus_queue.cc:260] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.182605 24024 tablet_service.cc:1467] Processing CreateTablet for tablet ea1ceb374483425194f962b5b5b87438 (DEFAULT_TABLE table=default.Table_0e95f8c694d7437c8aa8cb258bfcbe79 [id=08ed1c298e8744fc9f307ecc3834f49b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.183125 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea1ceb374483425194f962b5b5b87438. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.184298 24180 ts_tablet_manager.cc:1888] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:17:02.184401 24180 log.cc:1196] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/a2da8b9d3f374d8a812336057c0ce7d2
I20240502 08:17:02.184736 24180 ts_tablet_manager.cc:1909] T a2da8b9d3f374d8a812336057c0ce7d2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:02.185402 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1235) took 2537 ms. Trace:
I20240502 08:17:02.186261 23610 catalog_manager.cc:5484] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:02.186683 23610 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 07a9d13d3a2d4e6dbc04998c3dd7e1a8 (table default.Table_d9f68c6b26d442bbbd485bb89ee5564c [id=7a49ccfdd6884281ba9b882debd04871]). Expected version 1 got 0
I20240502 08:17:02.187055 23610 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 156c7356ece342aea0d8db667ebad89c (table default.Table_d8c6955b89724769bbfd81db2b5be455 [id=15b52f0688c24df2b6f3fda3732ceeb4]). Expected version 1 got 0
I20240502 08:17:02.187288 23610 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet fba8f6e017e848d59239ae98432b1837 (table default.Table_c7de500cdfa341798dbbd1cc9ab5e4fb [id=3deaa06a1e88491f9dda6abda2cbdd9f]). Expected version 1 got 0
I20240502 08:17:02.187491 23610 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 41f2e98ef2884b05994d02114e992e80 (table default.Table_50c546a17f484d3cb98a5ca4b3ed63e8 [id=b6dee89ecfb443d99267917d4df78e4d]). Expected version 3 got 0
I20240502 08:17:02.188303 23604 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_1b91eb0181b74d5ea0d24f2b9731c155 [id=9ff8ea75a9bc4de194a70fe2088a94bc] alter complete (version 2)
I20240502 08:17:02.207580 25357 raft_consensus.cc:482] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.210355 25357 raft_consensus.cc:504] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.210443 24023 tablet_service.cc:1467] Processing CreateTablet for tablet c8763198838c410e86be38794dfc1e39 (DEFAULT_TABLE table=default.Table_1395c9cd077b435d927da88d7bf720fd [id=f0cc4140193f4ecbb2b4fd87b05a7aef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.210809 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8763198838c410e86be38794dfc1e39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.210903 25357 leader_election.cc:288] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:02.208024 23904 tablet_service.cc:1514] Processing DeleteTablet for tablet 18625a1d971c43549b9f80e6046efc9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:02.208308 23901 tablet_service.cc:1467] Processing CreateTablet for tablet ea1ceb374483425194f962b5b5b87438 (DEFAULT_TABLE table=default.Table_0e95f8c694d7437c8aa8cb258bfcbe79 [id=08ed1c298e8744fc9f307ecc3834f49b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.209163 25852 tablet.cc:1722] T c45aa0dd26524e0fba27e4eaf17d704e P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:17:02.211683 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea1ceb374483425194f962b5b5b87438. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.214601 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d4e2f23c9649a4b1dfb6a179ca7be5" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:02.214980 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 5bbaad9e5f284ee195794e6ffec2c02b (DEFAULT_TABLE table=default.Table_0a2eb6adda7b49aeb8d231cd2306f093 [id=8e16217ebbbc474dbce0cd1e2cc69512]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.215286 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bbaad9e5f284ee195794e6ffec2c02b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:02.216866 23861 leader_election.cc:341] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.193795 24174 ts_tablet_manager.cc:1398] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 08:17:02.209192 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 5bbaad9e5f284ee195794e6ffec2c02b (DEFAULT_TABLE table=default.Table_0a2eb6adda7b49aeb8d231cd2306f093 [id=8e16217ebbbc474dbce0cd1e2cc69512]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.218530 24174 tablet_bootstrap.cc:492] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:02.218607 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bbaad9e5f284ee195794e6ffec2c02b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.220211 24174 tablet_bootstrap.cc:654] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.214869 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d4e2f23c9649a4b1dfb6a179ca7be5" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:02.215209 25752 mvcc.cc:205] Tried to move back new op lower bound from 7023156519154315264 to 7023156518277931008. Current Snapshot: MvccSnapshot[applied={T|T < 7023156519154315264}]
I20240502 08:17:02.198058 23767 tablet_service.cc:1467] Processing CreateTablet for tablet ea1ceb374483425194f962b5b5b87438 (DEFAULT_TABLE table=default.Table_0e95f8c694d7437c8aa8cb258bfcbe79 [id=08ed1c298e8744fc9f307ecc3834f49b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.222306 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea1ceb374483425194f962b5b5b87438. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.185482 24180 rpcz_store.cc:276] 0502 08:16:59.648279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.648470 (+ 191us) service_pool.cc:225] Handling call
0502 08:17:01.979006 (+2330536us) ts_tablet_manager.cc:1120] Deleting tablet a2da8b9d3f374d8a812336057c0ce7d2
0502 08:17:01.979014 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.135809 (+156795us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.184203 (+ 48394us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.185359 (+ 1156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2133434}
I20240502 08:17:02.224851 24180 tablet_replica.cc:330] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:02.225080 24180 raft_consensus.cc:2229] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:02.225234 24180 raft_consensus.cc:2258] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.225831 24180 ts_tablet_manager.cc:1875] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.226655 24174 tablet_bootstrap.cc:492] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:02.226776 24174 ts_tablet_manager.cc:1367] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 08:17:02.227066 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet a2da8b9d3f374d8a812336057c0ce7d2 (table default.Table_3edd27da6927478281e76361718425a3 [id=76b58ce209a24b9a95b405c1c50de3ca]) successfully deleted
W20240502 08:17:02.227409 23858 leader_election.cc:341] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.227525 23858 leader_election.cc:302] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:02.227551 24174 raft_consensus.cc:348] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.227737 24174 raft_consensus.cc:374] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.227800 24174 raft_consensus.cc:729] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.227881 25649 raft_consensus.cc:2726] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:02.227993 24174 consensus_queue.cc:260] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.229311 24174 ts_tablet_manager.cc:1398] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:02.229557 24174 tablet_bootstrap.cc:492] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:02.230939 24174 tablet_bootstrap.cc:654] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.231061 24052 raft_consensus.cc:1261] T ac50774b2dd241de9cae8096578d183e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Refusing update from remote peer 5b89a1f0485a44b7ac1710e2369a809b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:02.231701 25357 consensus_queue.cc:1035] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:02.232112 24180 ts_tablet_manager.cc:1888] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:02.241559 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 5bbaad9e5f284ee195794e6ffec2c02b (DEFAULT_TABLE table=default.Table_0a2eb6adda7b49aeb8d231cd2306f093 [id=8e16217ebbbc474dbce0cd1e2cc69512]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.241528 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1184) took 2450 ms. Trace:
I20240502 08:17:02.242154 25012 rpcz_store.cc:276] 0502 08:16:59.790787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.797813 (+ 7026us) service_pool.cc:225] Handling call
0502 08:17:02.150558 (+2352745us) ts_tablet_manager.cc:1120] Deleting tablet 63e645bb6e2948e091e08433944a52f4
0502 08:17:02.150566 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.160257 (+ 9691us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.173886 (+ 13629us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.241509 (+ 67623us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2352566}
I20240502 08:17:02.242660 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 63e645bb6e2948e091e08433944a52f4 (table default.Table_92a3c7cbfc16461ba5b08f262bc016d3 [id=47b70f46b4d345618054c69267d33c95]) successfully deleted
I20240502 08:17:02.243269 25843 raft_consensus.cc:482] T c45aa0dd26524e0fba27e4eaf17d704e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.243677 25843 raft_consensus.cc:504] T c45aa0dd26524e0fba27e4eaf17d704e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.244148 25012 tablet_replica.cc:330] T 9467974d67c64c7eb5e488ac51f1240d P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:02.244302 25843 leader_election.cc:288] T c45aa0dd26524e0fba27e4eaf17d704e P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:02.244324 25012 raft_consensus.cc:2229] T 9467974d67c64c7eb5e488ac51f1240d P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:02.244482 25012 raft_consensus.cc:2258] T 9467974d67c64c7eb5e488ac51f1240d P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.244931 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45aa0dd26524e0fba27e4eaf17d704e" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:02.246321 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bbaad9e5f284ee195794e6ffec2c02b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.247072 23901 tablet_service.cc:1467] Processing CreateTablet for tablet c8763198838c410e86be38794dfc1e39 (DEFAULT_TABLE table=default.Table_1395c9cd077b435d927da88d7bf720fd [id=f0cc4140193f4ecbb2b4fd87b05a7aef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.247511 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8763198838c410e86be38794dfc1e39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.247764 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 82a83530b77c4f8badd9c36c024823be (DEFAULT_TABLE table=default.Table_ce67caa9ca8a4bd092ecf98b26fd3afb [id=9ae4038cc6784eeb8c7200bd87e05ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.248106 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82a83530b77c4f8badd9c36c024823be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.255198 25012 ts_tablet_manager.cc:1875] T 9467974d67c64c7eb5e488ac51f1240d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.255919 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 82a83530b77c4f8badd9c36c024823be (DEFAULT_TABLE table=default.Table_ce67caa9ca8a4bd092ecf98b26fd3afb [id=9ae4038cc6784eeb8c7200bd87e05ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.257097 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82a83530b77c4f8badd9c36c024823be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.262526 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 5615e8bdcbfe478c8c9fb029f95ec8ad (DEFAULT_TABLE table=default.Table_963bbb5ed3184e6a95549deb63ffb35a [id=d330f8cbe8ce4b089d6f8a9435ed07cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.262892 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5615e8bdcbfe478c8c9fb029f95ec8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.270100 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 5615e8bdcbfe478c8c9fb029f95ec8ad (DEFAULT_TABLE table=default.Table_963bbb5ed3184e6a95549deb63ffb35a [id=d330f8cbe8ce4b089d6f8a9435ed07cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.270440 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5615e8bdcbfe478c8c9fb029f95ec8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.272187 25852 tablet.cc:1722] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:02.277727 24180 log.cc:1196] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/42e11d405a2e4e518a10bf383e9ae6c4
I20240502 08:17:02.234181 23786 raft_consensus.cc:1261] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Refusing update from remote peer 5b89a1f0485a44b7ac1710e2369a809b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:02.237818 25891 tablet.cc:1722] T c45aa0dd26524e0fba27e4eaf17d704e P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:17:02.278093 24180 ts_tablet_manager.cc:1909] T 42e11d405a2e4e518a10bf383e9ae6c4 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:02.280984 25649 consensus_queue.cc:1035] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:02.240532 25845 consensus_queue.cc:1035] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:02.288182 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1226) took 2694 ms. Trace:
I20240502 08:17:02.288333 24180 rpcz_store.cc:276] 0502 08:16:59.593909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.599657 (+ 5748us) service_pool.cc:225] Handling call
0502 08:17:02.224796 (+2625139us) ts_tablet_manager.cc:1120] Deleting tablet 42e11d405a2e4e518a10bf383e9ae6c4
0502 08:17:02.224807 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.229240 (+ 4433us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.232011 (+ 2771us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.288163 (+ 56152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2342787}
I20240502 08:17:02.288905 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1247) took 2603 ms. Trace:
I20240502 08:17:02.289018 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 42e11d405a2e4e518a10bf383e9ae6c4 (table default.Table_d790560c82534057ab7061352d8c71a2 [id=15cd2d29f86a49e295158ba17e364a23]) successfully deleted
I20240502 08:17:02.289041 24180 rpcz_store.cc:276] 0502 08:16:59.685410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.829285 (+143875us) service_pool.cc:225] Handling call
0502 08:17:02.288780 (+2459495us) ts_tablet_manager.cc:1120] Deleting tablet 8c93c89e1341474eafb6030b95f1c68c
0502 08:17:02.288788 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.288890 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2254172}
W20240502 08:17:02.289414 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 8c93c89e1341474eafb6030b95f1c68c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c93c89e1341474eafb6030b95f1c68c already in progress: creating tablet
I20240502 08:17:02.289723 24180 tablet_replica.cc:330] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:02.289959 24180 raft_consensus.cc:2229] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:02.290431 24180 raft_consensus.cc:2258] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.291023 24180 ts_tablet_manager.cc:1875] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.296598 24180 ts_tablet_manager.cc:1888] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:17:02.296747 24180 log.cc:1196] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/a5768cb928e244d2914f10f4eda3c1cd
I20240502 08:17:02.297154 24180 ts_tablet_manager.cc:1909] T a5768cb928e244d2914f10f4eda3c1cd P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:02.297969 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1251) took 2238 ms. Trace:
I20240502 08:17:02.298084 24180 rpcz_store.cc:276] 0502 08:17:00.059123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:00.066094 (+ 6971us) service_pool.cc:225] Handling call
0502 08:17:02.289680 (+2223586us) ts_tablet_manager.cc:1120] Deleting tablet a5768cb928e244d2914f10f4eda3c1cd
0502 08:17:02.289689 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.293843 (+ 4154us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.296525 (+ 2682us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.297952 (+ 1427us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2223386}
I20240502 08:17:02.298491 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet a5768cb928e244d2914f10f4eda3c1cd (table default.Table_a394560ca3454cb88c3cbc2e87ab4fa7 [id=09f6a10417fb4639a9b97f5bb9efee6b]) successfully deleted
I20240502 08:17:02.298846 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1290) took 1751 ms. Trace:
I20240502 08:17:02.298960 24180 rpcz_store.cc:276] 0502 08:17:00.546971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:00.606753 (+ 59782us) service_pool.cc:225] Handling call
0502 08:17:02.298743 (+1691990us) ts_tablet_manager.cc:1120] Deleting tablet 9850e84117044d2f9dc82b6314dac2cc
0502 08:17:02.298751 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.298832 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1691745}
W20240502 08:17:02.299441 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 9850e84117044d2f9dc82b6314dac2cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9850e84117044d2f9dc82b6314dac2cc already in progress: creating tablet
I20240502 08:17:02.299724 24180 tablet_replica.cc:330] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:02.299973 24180 raft_consensus.cc:2229] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:02.300132 24180 raft_consensus.cc:2258] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.300577 24180 ts_tablet_manager.cc:1875] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.305398 24180 ts_tablet_manager.cc:1888] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:02.305527 24180 log.cc:1196] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/1b175e1b55bf4102804666b6fffccead
I20240502 08:17:02.306138 24180 ts_tablet_manager.cc:1909] T 1b175e1b55bf4102804666b6fffccead P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:02.306622 25814 tablet.cc:1722] T ac50774b2dd241de9cae8096578d183e P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:02.307032 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1291) took 1651 ms. Trace:
I20240502 08:17:02.307207 24180 rpcz_store.cc:276] 0502 08:17:00.655115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:00.655173 (+ 58us) service_pool.cc:225] Handling call
0502 08:17:02.299685 (+1644512us) ts_tablet_manager.cc:1120] Deleting tablet 1b175e1b55bf4102804666b6fffccead
0502 08:17:02.299692 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.303040 (+ 3348us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.305337 (+ 2297us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.307016 (+ 1679us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1644267}
I20240502 08:17:02.307597 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 1b175e1b55bf4102804666b6fffccead (table default.Table_8c96a3fc08e94e36af627e4c8dddc7ae [id=5a9755c4cebd4ad2ab00ce225b25334c]) successfully deleted
I20240502 08:17:02.308049 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1292) took 1626 ms. Trace:
I20240502 08:17:02.308156 24180 rpcz_store.cc:276] 0502 08:17:00.700142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:00.709497 (+ 9355us) service_pool.cc:225] Handling call
0502 08:17:02.307847 (+1598350us) ts_tablet_manager.cc:1120] Deleting tablet 49acc2af1a064dd2b0b60120beed69ba
0502 08:17:02.307855 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.307953 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1597660}
W20240502 08:17:02.308559 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 49acc2af1a064dd2b0b60120beed69ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49acc2af1a064dd2b0b60120beed69ba already in progress: creating tablet
I20240502 08:17:02.309012 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1293) took 1608 ms. Trace:
I20240502 08:17:02.309111 24180 rpcz_store.cc:276] 0502 08:17:00.700274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:00.710213 (+ 9939us) service_pool.cc:225] Handling call
0502 08:17:02.308859 (+1598646us) ts_tablet_manager.cc:1120] Deleting tablet 14f54d6ded85434da35c1dfcff52763a
0502 08:17:02.308866 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.308979 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1597744}
W20240502 08:17:02.309484 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 14f54d6ded85434da35c1dfcff52763a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14f54d6ded85434da35c1dfcff52763a already in progress: creating tablet
I20240502 08:17:02.309904 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1294) took 1506 ms. Trace:
I20240502 08:17:02.310035 24180 rpcz_store.cc:276] 0502 08:17:00.803516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:00.803565 (+ 49us) service_pool.cc:225] Handling call
0502 08:17:02.309800 (+1506235us) ts_tablet_manager.cc:1120] Deleting tablet d423a92750c94ee1a08688d60dd2168e
0502 08:17:02.309808 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.309890 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1506067}
W20240502 08:17:02.310442 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet d423a92750c94ee1a08688d60dd2168e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d423a92750c94ee1a08688d60dd2168e already in progress: creating tablet
I20240502 08:17:02.310844 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1301) took 1176 ms. Trace:
I20240502 08:17:02.310958 24180 rpcz_store.cc:276] 0502 08:17:01.134778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.134829 (+ 51us) service_pool.cc:225] Handling call
0502 08:17:02.310751 (+1175922us) ts_tablet_manager.cc:1120] Deleting tablet 5027fa2c80b544b8b1e2a4694bec7454
0502 08:17:02.310760 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.310830 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1147229}
W20240502 08:17:02.311342 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 5027fa2c80b544b8b1e2a4694bec7454 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5027fa2c80b544b8b1e2a4694bec7454 already in progress: creating tablet
I20240502 08:17:02.311662 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1302) took 1172 ms. Trace:
I20240502 08:17:02.311769 24180 rpcz_store.cc:276] 0502 08:17:01.139029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.153142 (+ 14113us) service_pool.cc:225] Handling call
0502 08:17:02.311576 (+1158434us) ts_tablet_manager.cc:1120] Deleting tablet df06e0edb57f45e5ac71b568c14769cd
0502 08:17:02.311583 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.311649 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1147662}
W20240502 08:17:02.312127 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet df06e0edb57f45e5ac71b568c14769cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df06e0edb57f45e5ac71b568c14769cd already in progress: creating tablet
I20240502 08:17:02.312412 24180 tablet_replica.cc:330] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:02.312598 24180 raft_consensus.cc:2229] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:02.312913 24180 raft_consensus.cc:2258] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.277040 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa25f811d154528b3bdcd4a7b024af6 (DEFAULT_TABLE table=default.Table_497c6963c9c94ff391bc3c47722ebbe4 [id=11f843f85fa6401c85c62c34798033b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.313508 24180 ts_tablet_manager.cc:1875] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.314653 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa25f811d154528b3bdcd4a7b024af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.317071 25891 tablet.cc:1722] T 41f2e98ef2884b05994d02114e992e80 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:17:02.317567 25852 tablet.cc:1722] T 41f2e98ef2884b05994d02114e992e80 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:17:02.320516 25814 tablet.cc:1722] T 41f2e98ef2884b05994d02114e992e80 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:17:02.243346 23751 tablet_service.cc:1467] Processing CreateTablet for tablet c8763198838c410e86be38794dfc1e39 (DEFAULT_TABLE table=default.Table_1395c9cd077b435d927da88d7bf720fd [id=f0cc4140193f4ecbb2b4fd87b05a7aef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.245064 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45aa0dd26524e0fba27e4eaf17d704e" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:02.324829 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8763198838c410e86be38794dfc1e39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.325135 23993 leader_election.cc:302] T c45aa0dd26524e0fba27e4eaf17d704e P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:02.325649 25843 raft_consensus.cc:2726] T c45aa0dd26524e0fba27e4eaf17d704e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:02.245123 25846 raft_consensus.cc:482] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.325906 25846 raft_consensus.cc:504] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.326493 25846 leader_election.cc:288] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:02.327214 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d4e2f23c9649a4b1dfb6a179ca7be5" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:02.327405 23920 raft_consensus.cc:2454] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:02.328181 23723 leader_election.cc:302] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:17:02.328501 25846 raft_consensus.cc:2781] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:02.328569 25846 raft_consensus.cc:482] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:02.328621 25846 raft_consensus.cc:3037] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.329675 25843 raft_consensus.cc:482] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.329780 25843 raft_consensus.cc:504] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.330276 25843 leader_election.cc:288] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:02.330837 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18625a1d971c43549b9f80e6046efc9c" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:02.331009 23920 raft_consensus.cc:2454] T 18625a1d971c43549b9f80e6046efc9c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
W20240502 08:17:02.331233 23993 leader_election.cc:341] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 18625a1d971c43549b9f80e6046efc9c
I20240502 08:17:02.331416 23996 leader_election.cc:302] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:02.331802 25843 raft_consensus.cc:2781] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:02.331869 25843 raft_consensus.cc:482] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:02.331931 25843 raft_consensus.cc:3037] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.332132 25846 raft_consensus.cc:504] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.332692 25846 leader_election.cc:288] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:02.333173 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d4e2f23c9649a4b1dfb6a179ca7be5" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:02.333302 23920 raft_consensus.cc:3037] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.334105 25843 raft_consensus.cc:504] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:17:02.335191 23993 leader_election.cc:341] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 18625a1d971c43549b9f80e6046efc9c
I20240502 08:17:02.335536 25843 leader_election.cc:288] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:02.335783 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18625a1d971c43549b9f80e6046efc9c" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:02.335949 23922 raft_consensus.cc:3037] T 18625a1d971c43549b9f80e6046efc9c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.254393 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 82a83530b77c4f8badd9c36c024823be (DEFAULT_TABLE table=default.Table_ce67caa9ca8a4bd092ecf98b26fd3afb [id=9ae4038cc6784eeb8c7200bd87e05ee7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.256907 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 5615e8bdcbfe478c8c9fb029f95ec8ad (DEFAULT_TABLE table=default.Table_963bbb5ed3184e6a95549deb63ffb35a [id=d330f8cbe8ce4b089d6f8a9435ed07cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.273231 24174 tablet_bootstrap.cc:492] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:02.336289 24174 ts_tablet_manager.cc:1367] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.107s user 0.020s sys 0.015s
I20240502 08:17:02.336436 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5615e8bdcbfe478c8c9fb029f95ec8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.337023 24174 raft_consensus.cc:348] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.337177 24174 raft_consensus.cc:374] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.337229 24174 raft_consensus.cc:729] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.337395 24174 consensus_queue.cc:260] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.337855 24174 ts_tablet_manager.cc:1398] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:17:02.338080 24174 tablet_bootstrap.cc:492] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:02.338852 23922 raft_consensus.cc:2454] T 18625a1d971c43549b9f80e6046efc9c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:17:02.339427 24174 tablet_bootstrap.cc:654] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.339538 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d4e2f23c9649a4b1dfb6a179ca7be5" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:02.340456 25845 consensus_queue.cc:1035] T 8910bc67c6234a538c8160e7c020653c P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:02.341444 23920 raft_consensus.cc:2454] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:17:02.341971 23996 leader_election.cc:302] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:02.343691 25843 raft_consensus.cc:2781] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:02.343745 23723 leader_election.cc:302] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:17:02.344015 25845 raft_consensus.cc:2781] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:02.344100 25845 raft_consensus.cc:686] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:17:02.344084 25843 raft_consensus.cc:686] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:17:02.344290 25845 consensus_queue.cc:237] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.344293 25843 consensus_queue.cc:237] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.345748 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d4e2f23c9649a4b1dfb6a179ca7be5" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:02.349071 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82a83530b77c4f8badd9c36c024823be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.356021 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa25f811d154528b3bdcd4a7b024af6 (DEFAULT_TABLE table=default.Table_497c6963c9c94ff391bc3c47722ebbe4 [id=11f843f85fa6401c85c62c34798033b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.356510 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa25f811d154528b3bdcd4a7b024af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.363031 25012 ts_tablet_manager.cc:1888] T 9467974d67c64c7eb5e488ac51f1240d P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:02.364888 25012 log.cc:1196] T 9467974d67c64c7eb5e488ac51f1240d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/9467974d67c64c7eb5e488ac51f1240d
I20240502 08:17:02.365489 25012 ts_tablet_manager.cc:1909] T 9467974d67c64c7eb5e488ac51f1240d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:02.368685 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8fd9957835484b8cbe6bc35c08b4c0 (DEFAULT_TABLE table=default.Table_43f0e7d47b19428e856ead8b62806088 [id=4c481cfd0a7d4e3d9437ab170589747e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.369001 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8fd9957835484b8cbe6bc35c08b4c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.370304 25814 tablet.cc:1722] T 96b8f21c64e343d88d84e9acd0bc69b4 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:17:02.378978 24180 ts_tablet_manager.cc:1888] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:17:02.380450 24180 log.cc:1196] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/d1be2c29d3154d9189af3266191f0b3e
I20240502 08:17:02.380959 24180 ts_tablet_manager.cc:1909] T d1be2c29d3154d9189af3266191f0b3e P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:02.382903 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1303) took 1191 ms. Trace:
I20240502 08:17:02.383312 24180 rpcz_store.cc:276] 0502 08:17:01.191647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.191740 (+ 93us) service_pool.cc:225] Handling call
0502 08:17:02.312374 (+1120634us) ts_tablet_manager.cc:1120] Deleting tablet d1be2c29d3154d9189af3266191f0b3e
0502 08:17:02.312381 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.365298 (+ 52917us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.378903 (+ 13605us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.382885 (+ 3982us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1120424}
I20240502 08:17:02.383952 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1304) took 1191 ms. Trace:
I20240502 08:17:02.384305 24180 rpcz_store.cc:276] 0502 08:17:01.192440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.192485 (+ 45us) service_pool.cc:225] Handling call
0502 08:17:02.383837 (+1191352us) ts_tablet_manager.cc:1120] Deleting tablet b6bb4a4ccd574f4cadc0b9d81ef4dd5b
0502 08:17:02.383845 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.383938 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1191180}
I20240502 08:17:02.384918 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1305) took 1191 ms. Trace:
I20240502 08:17:02.385365 24180 rpcz_store.cc:276] 0502 08:17:01.193271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.193311 (+ 40us) service_pool.cc:225] Handling call
0502 08:17:02.384788 (+1191477us) ts_tablet_manager.cc:1120] Deleting tablet f87fe1c2c57748f0a17ee14503890c2f
0502 08:17:02.384797 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.384899 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1191344}
I20240502 08:17:02.385800 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet d1be2c29d3154d9189af3266191f0b3e (table default.Table_db86cd2cf86e48ffbc6a513b5c2bd7ab [id=6103f133bcdd4f768b92b37245bf5356]) successfully deleted
W20240502 08:17:02.386045 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet b6bb4a4ccd574f4cadc0b9d81ef4dd5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6bb4a4ccd574f4cadc0b9d81ef4dd5b already in progress: creating tablet
I20240502 08:17:02.386116 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1306) took 1076 ms. Trace:
W20240502 08:17:02.386267 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet f87fe1c2c57748f0a17ee14503890c2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f87fe1c2c57748f0a17ee14503890c2f already in progress: creating tablet
I20240502 08:17:02.386209 24180 rpcz_store.cc:276] 0502 08:17:01.309166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.310517 (+ 1351us) service_pool.cc:225] Handling call
0502 08:17:02.385843 (+1075326us) ts_tablet_manager.cc:1120] Deleting tablet ff4a333c6dd44a13bc267e885bb9e487
0502 08:17:02.385850 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.386094 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":926106}
W20240502 08:17:02.386662 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet ff4a333c6dd44a13bc267e885bb9e487 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff4a333c6dd44a13bc267e885bb9e487 already in progress: creating tablet
W20240502 08:17:02.387663 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 9467974d67c64c7eb5e488ac51f1240d because the tablet was not found. No further retry: Not found: Tablet not found: 9467974d67c64c7eb5e488ac51f1240d
W20240502 08:17:02.388113 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 336f0eb5085546dba763e0d5c5dd5324 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 336f0eb5085546dba763e0d5c5dd5324 already in progress: creating tablet
W20240502 08:17:02.388796 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 2b9f0b0afc3c47759e0e94ca44cf9711 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b9f0b0afc3c47759e0e94ca44cf9711 already in progress: creating tablet
W20240502 08:17:02.389425 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 4b52e5c136d84cf1bfc8fadea62491f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b52e5c136d84cf1bfc8fadea62491f6 already in progress: creating tablet
W20240502 08:17:02.390316 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 88bf97d0a0a54e828c3b0902976e0500 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88bf97d0a0a54e828c3b0902976e0500 already in progress: creating tablet
I20240502 08:17:02.391625 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1185) took 2593 ms. Trace:
I20240502 08:17:02.391763 25012 rpcz_store.cc:276] 0502 08:16:59.798205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.798267 (+ 62us) service_pool.cc:225] Handling call
0502 08:17:02.244097 (+2445830us) ts_tablet_manager.cc:1120] Deleting tablet 9467974d67c64c7eb5e488ac51f1240d
0502 08:17:02.244108 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.270124 (+ 26016us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.362947 (+ 92823us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.391604 (+ 28657us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2445678}
W20240502 08:17:02.392119 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet c61243a38bfa4964b225eeae0e8cc28e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c61243a38bfa4964b225eeae0e8cc28e already in progress: creating tablet
I20240502 08:17:02.392213 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 9467974d67c64c7eb5e488ac51f1240d (table default.Table_c1886293aee349b2b9e7177c1cfd6d26 [id=0853ffc4d48549bc93e251a5dbc55901]) successfully deleted
I20240502 08:17:02.392566 25012 tablet_replica.cc:330] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:02.392825 25012 raft_consensus.cc:2229] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:02.367895 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa25f811d154528b3bdcd4a7b024af6 (DEFAULT_TABLE table=default.Table_497c6963c9c94ff391bc3c47722ebbe4 [id=11f843f85fa6401c85c62c34798033b7]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:17:02.392957 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 91167335ab4441a1a2fc8d70cebdd2eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91167335ab4441a1a2fc8d70cebdd2eb already in progress: creating tablet
I20240502 08:17:02.393064 25012 raft_consensus.cc:2258] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.393353 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa25f811d154528b3bdcd4a7b024af6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:02.397977 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 858c022bdb5449d980acc8b0eeec9ff0 consensus state has no leader
I20240502 08:17:02.398525 24174 tablet_bootstrap.cc:492] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
W20240502 08:17:02.398679 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9eaf8d65f0b4472eb8ed0ef6fc722577 consensus state has no leader
I20240502 08:17:02.398761 24174 ts_tablet_manager.cc:1367] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.007s
I20240502 08:17:02.399163 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 12d4afaeae25429cb469204690a4d3ec
I20240502 08:17:02.399694 24174 raft_consensus.cc:348] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.399947 24174 raft_consensus.cc:374] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.400002 24174 raft_consensus.cc:729] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.400197 24174 consensus_queue.cc:260] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.400375 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 12d4afaeae25429cb469204690a4d3ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:02.400681 24174 ts_tablet_manager.cc:1398] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:02.400911 24174 tablet_bootstrap.cc:492] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:02.401168 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 3922e528a97a4287a0122a5b51035dcb
I20240502 08:17:02.401532 23606 catalog_manager.cc:6554] ReplaceTablet: tablet 3922e528a97a4287a0122a5b51035dcb deleted and replaced by tablet dd4d2571803643c88ffbd93d7909ac44
I20240502 08:17:02.402230 24174 tablet_bootstrap.cc:654] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.402333 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 3922e528a97a4287a0122a5b51035dcb with delete_type TABLET_DATA_DELETED (replaced by tablet dd4d2571803643c88ffbd93d7909ac44) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:02.402297 25891 tablet.cc:1722] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240502 08:17:02.402598 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 12d4afaeae25429cb469204690a4d3ec because the tablet was not found. No further retry: Not found: Tablet not found: 12d4afaeae25429cb469204690a4d3ec
W20240502 08:17:02.403167 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 3922e528a97a4287a0122a5b51035dcb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3922e528a97a4287a0122a5b51035dcb already in progress: creating tablet
I20240502 08:17:02.404450 23904 tablet_service.cc:1514] Processing DeleteTablet for tablet 3922e528a97a4287a0122a5b51035dcb with delete_type TABLET_DATA_DELETED (replaced by tablet dd4d2571803643c88ffbd93d7909ac44) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:02.404969 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 12d4afaeae25429cb469204690a4d3ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:02.410332 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8fd9957835484b8cbe6bc35c08b4c0 (DEFAULT_TABLE table=default.Table_43f0e7d47b19428e856ead8b62806088 [id=4c481cfd0a7d4e3d9437ab170589747e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.410801 23767 tablet_service.cc:1467] Processing CreateTablet for tablet dba5ed2797d64947a0b1097ea3cecb0a (DEFAULT_TABLE table=default.Table_b49e692bc51f437281bdebe78679a3ff [id=3724d5ac2e8e4a9aba21ead1879a880e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.411116 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba5ed2797d64947a0b1097ea3cecb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.411193 23904 tablet_service.cc:1467] Processing CreateTablet for tablet dba5ed2797d64947a0b1097ea3cecb0a (DEFAULT_TABLE table=default.Table_b49e692bc51f437281bdebe78679a3ff [id=3724d5ac2e8e4a9aba21ead1879a880e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.411742 24031 tablet_service.cc:1467] Processing CreateTablet for tablet dba5ed2797d64947a0b1097ea3cecb0a (DEFAULT_TABLE table=default.Table_b49e692bc51f437281bdebe78679a3ff [id=3724d5ac2e8e4a9aba21ead1879a880e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.412061 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba5ed2797d64947a0b1097ea3cecb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.413043 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet 12d4afaeae25429cb469204690a4d3ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:02.413137 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 121668ea2e2943c4b61354b8c165000e (DEFAULT_TABLE table=default.Table_5b4382ee4fc14029864a8f4b11589da4 [id=ee7160db9d5f445faa9814e1f39b2315]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.414068 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 968a36ac363e4eb1aef108ff2eb28167 (DEFAULT_TABLE table=default.Table_47192b76b10e4c79a75072b2846ac5af [id=f2d3ad73d9af4378920c922efc8bceb7]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:17:02.414213 23597 catalog_manager.cc:4797] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 12d4afaeae25429cb469204690a4d3ec because the tablet was not found. No further retry: Not found: Tablet not found: 12d4afaeae25429cb469204690a4d3ec
I20240502 08:17:02.414366 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968a36ac363e4eb1aef108ff2eb28167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.415820 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 3922e528a97a4287a0122a5b51035dcb with delete_type TABLET_DATA_DELETED (replaced by tablet dd4d2571803643c88ffbd93d7909ac44) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:02.415851 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_347c7bcdb5674f45b6f1842416643c3b" } modify_external_catalogs: true
I20240502 08:17:02.416002 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_347c7bcdb5674f45b6f1842416643c3b" } modify_external_catalogs: true
I20240502 08:17:02.417454 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 121668ea2e2943c4b61354b8c165000e (DEFAULT_TABLE table=default.Table_5b4382ee4fc14029864a8f4b11589da4 [id=ee7160db9d5f445faa9814e1f39b2315]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.417738 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 121668ea2e2943c4b61354b8c165000e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.417781 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_afe99e714f924148a842b013e4daa8e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:02.418200 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afe99e714f924148a842b013e4daa8e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:02.419560 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ab7e9343fdc74b38bb98f2ead6c8eeb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:02.419914 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab7e9343fdc74b38bb98f2ead6c8eeb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:02.417513 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cf7ced0046f647eb82829af1fb7c985b" } new_table_name: "default.Table_9d3c0f6b40954b88b4210e6c99ae99cd" modify_external_catalogs: true
W20240502 08:17:02.421774 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 3922e528a97a4287a0122a5b51035dcb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3922e528a97a4287a0122a5b51035dcb already in progress: creating tablet
I20240502 08:17:02.421980 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 968a36ac363e4eb1aef108ff2eb28167 (DEFAULT_TABLE table=default.Table_47192b76b10e4c79a75072b2846ac5af [id=f2d3ad73d9af4378920c922efc8bceb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.423174 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968a36ac363e4eb1aef108ff2eb28167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.430105 24174 tablet_bootstrap.cc:492] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:02.430322 24174 ts_tablet_manager.cc:1367] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20240502 08:17:02.431249 24175 tablet_bootstrap.cc:492] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:02.431188 24174 raft_consensus.cc:348] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.431447 24175 ts_tablet_manager.cc:1367] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.301s user 0.008s sys 0.008s
I20240502 08:17:02.431455 24174 raft_consensus.cc:374] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.431511 24174 raft_consensus.cc:729] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.431751 24174 consensus_queue.cc:260] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.432271 24174 ts_tablet_manager.cc:1398] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:02.432538 24174 tablet_bootstrap.cc:492] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:02.433694 24175 raft_consensus.cc:348] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.433952 24175 raft_consensus.cc:374] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.434048 24175 raft_consensus.cc:729] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.434401 24174 tablet_bootstrap.cc:654] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.434330 24175 consensus_queue.cc:260] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.434954 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 968a36ac363e4eb1aef108ff2eb28167 (DEFAULT_TABLE table=default.Table_47192b76b10e4c79a75072b2846ac5af [id=f2d3ad73d9af4378920c922efc8bceb7]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:17:02.435648 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 89634.5us
I20240502 08:17:02.435947 24175 ts_tablet_manager.cc:1398] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
W20240502 08:17:02.435959 23723 leader_election.cc:341] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.436123 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbe0d41ef454eddb69a532358312c56 (DEFAULT_TABLE table=default.Table_9455a9a5481a40749f263245cfa038ef [id=70ca7ad9cb554854a2f54f510ee6242d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:17:02.436286 23723 leader_election.cc:341] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.436539 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbe0d41ef454eddb69a532358312c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.437193 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbe0d41ef454eddb69a532358312c56 (DEFAULT_TABLE table=default.Table_9455a9a5481a40749f263245cfa038ef [id=70ca7ad9cb554854a2f54f510ee6242d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.437813 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 12d4afaeae25429cb469204690a4d3ec (DEFAULT_TABLE table=default.Table_d731f4904f8649bdbc1369d357489e48 [id=7cefc2e9af9d4923ac5bf936a3faee02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.438735 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 12d4afaeae25429cb469204690a4d3ec (DEFAULT_TABLE table=default.Table_d731f4904f8649bdbc1369d357489e48 [id=7cefc2e9af9d4923ac5bf936a3faee02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.439047 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12d4afaeae25429cb469204690a4d3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.439240 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8fd9957835484b8cbe6bc35c08b4c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.439699 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12d4afaeae25429cb469204690a4d3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.441869 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 338fa14b1e004c36accccb3b671103c1 (DEFAULT_TABLE table=default.Table_1bdb95f894a04d6bbeafc73022a2283c [id=13e63816db324aab8601051494492a73]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.442134 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 338fa14b1e004c36accccb3b671103c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.442216 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 121668ea2e2943c4b61354b8c165000e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.443908 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba5ed2797d64947a0b1097ea3cecb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.446789 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968a36ac363e4eb1aef108ff2eb28167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.449550 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbe0d41ef454eddb69a532358312c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.450798 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 121668ea2e2943c4b61354b8c165000e (DEFAULT_TABLE table=default.Table_5b4382ee4fc14029864a8f4b11589da4 [id=ee7160db9d5f445faa9814e1f39b2315]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.452250 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 121668ea2e2943c4b61354b8c165000e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.455578 24175 tablet_bootstrap.cc:492] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:02.457763 24175 tablet_bootstrap.cc:654] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.460526 24174 tablet_bootstrap.cc:492] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:02.460700 24174 ts_tablet_manager.cc:1367] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20240502 08:17:02.461385 24174 raft_consensus.cc:348] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.461647 24174 raft_consensus.cc:374] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.462486 24174 raft_consensus.cc:729] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.463294 25012 ts_tablet_manager.cc:1875] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.467015 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 9dbe0d41ef454eddb69a532358312c56 (DEFAULT_TABLE table=default.Table_9455a9a5481a40749f263245cfa038ef [id=70ca7ad9cb554854a2f54f510ee6242d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.469638 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dbe0d41ef454eddb69a532358312c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.473748 24174 consensus_queue.cc:260] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.474319 24174 ts_tablet_manager.cc:1398] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240502 08:17:02.476182 24174 tablet_bootstrap.cc:492] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:02.477759 24174 tablet_bootstrap.cc:654] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.484215 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 5955411f06e04737bbe0f0d8efc77102
I20240502 08:17:02.485108 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 1ded3cee64f9409583040ef4b6e1f803
I20240502 08:17:02.485123 25012 ts_tablet_manager.cc:1888] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:02.485219 25012 log.cc:1196] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/eb96ad2289754e428a1c109b2d59c53b
I20240502 08:17:02.485517 25012 ts_tablet_manager.cc:1909] T eb96ad2289754e428a1c109b2d59c53b P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:02.485852 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1186) took 2687 ms. Trace:
I20240502 08:17:02.486016 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 6d7cc1360a7d4379867e75bb7084c4fe
I20240502 08:17:02.485929 25012 rpcz_store.cc:276] 0502 08:16:59.798604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.798648 (+ 44us) service_pool.cc:225] Handling call
0502 08:17:02.392521 (+2593873us) ts_tablet_manager.cc:1120] Deleting tablet eb96ad2289754e428a1c109b2d59c53b
0502 08:17:02.392530 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.476201 (+ 83671us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.485023 (+ 8822us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.485826 (+ 803us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2593702}
I20240502 08:17:02.487998 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet eb96ad2289754e428a1c109b2d59c53b (table default.Table_bc1e9d346aa846f8abca7504185f7cdb [id=8d27878c6c8f4234908a2c78cb8bbd29]) successfully deleted
I20240502 08:17:02.488978 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ded3cee64f9409583040ef4b6e1f803 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:02.489002 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 5955411f06e04737bbe0f0d8efc77102 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:02.489184 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d7cc1360a7d4379867e75bb7084c4fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:17:02.490192 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 5955411f06e04737bbe0f0d8efc77102 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5955411f06e04737bbe0f0d8efc77102 already in progress: creating tablet
I20240502 08:17:02.490319 23604 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 0501575a84ec454780f48574bf3155fe (table default.Table_540a424c4bfa4484a65f0b24140a4bbb [id=5cee878b3a31426c879a1bf95045bb09]). Expected version 4 got 0
W20240502 08:17:02.490453 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 6d7cc1360a7d4379867e75bb7084c4fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d7cc1360a7d4379867e75bb7084c4fe already in progress: creating tablet
I20240502 08:17:02.490628 23604 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 96b8f21c64e343d88d84e9acd0bc69b4 (table default.Table_1b91eb0181b74d5ea0d24f2b9731c155 [id=9ff8ea75a9bc4de194a70fe2088a94bc]). Expected version 2 got 0
W20240502 08:17:02.490665 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 1ded3cee64f9409583040ef4b6e1f803 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ded3cee64f9409583040ef4b6e1f803 already in progress: creating tablet
I20240502 08:17:02.490772 23604 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 41f2e98ef2884b05994d02114e992e80 (table default.Table_50c546a17f484d3cb98a5ca4b3ed63e8 [id=b6dee89ecfb443d99267917d4df78e4d]). Expected version 3 got 0
I20240502 08:17:02.490991 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ded3cee64f9409583040ef4b6e1f803 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:02.491112 23761 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b98cc8728c042f4b7f6496fcc5ef913 with delete_type TABLET_DATA_DELETED (replaced by tablet 25c8a6aaabe2413d8ce88f54ce45a4c4) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:02.491569 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet 5955411f06e04737bbe0f0d8efc77102 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:02.492244 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 92adebc0c69e4a52b975253656132495 with delete_type TABLET_DATA_DELETED (replaced by tablet ae2b20ae3a014c85af33d5ee51b704e1) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:02.492508 24181 tablet_replica.cc:330] T 92adebc0c69e4a52b975253656132495 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:02.492687 24181 raft_consensus.cc:2229] T 92adebc0c69e4a52b975253656132495 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:02.492828 24181 raft_consensus.cc:2258] T 92adebc0c69e4a52b975253656132495 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.492791 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d7cc1360a7d4379867e75bb7084c4fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:02.493009 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet 1305dbdac3994224a047df139cf8d518 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:02.493163 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet a5768cb928e244d2914f10f4eda3c1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:02.493275 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 90d4e2f23c9649a4b1dfb6a179ca7be5 (table default.Table_b2e1c08bfc404c7ebb2720631ad10495 [id=9480d59a76fc401e959a1e41caafc845]). Expected version 3 got 0
I20240502 08:17:02.493331 24181 ts_tablet_manager.cc:1875] T 92adebc0c69e4a52b975253656132495 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.493453 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 665bbee2d15b4112b651844431535d0e (table default.Table_64f173e9aa5947069cae45a6cb8dd18f [id=427121bc94c247eaa315b6c830770d61]). Expected version 3 got 0
I20240502 08:17:02.493640 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 41f2e98ef2884b05994d02114e992e80 (table default.Table_50c546a17f484d3cb98a5ca4b3ed63e8 [id=b6dee89ecfb443d99267917d4df78e4d]). Expected version 3 got 0
I20240502 08:17:02.493829 23609 catalog_manager.cc:5484] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66). New cstate: current_term: 1 leader_uuid: "5b89a1f0485a44b7ac1710e2369a809b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:02.496027 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet e5f5597402374527bf0e5e50178df713 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:02.496259 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet af6efeebaebc495aba017a922e243c39 with delete_type TABLET_DATA_DELETED (replaced by tablet 9e13c0b81dc24b8ba1a88b31bf510efd) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:02.499114 24180 tablet_replica.cc:330] T 2b98cc8728c042f4b7f6496fcc5ef913 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:02.499289 24180 raft_consensus.cc:2229] T 2b98cc8728c042f4b7f6496fcc5ef913 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:02.499418 24180 raft_consensus.cc:2258] T 2b98cc8728c042f4b7f6496fcc5ef913 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.499902 24180 ts_tablet_manager.cc:1875] T 2b98cc8728c042f4b7f6496fcc5ef913 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.500957 24181 ts_tablet_manager.cc:1888] T 92adebc0c69e4a52b975253656132495 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:02.501039 24181 log.cc:1196] T 92adebc0c69e4a52b975253656132495 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/92adebc0c69e4a52b975253656132495
I20240502 08:17:02.501343 24181 ts_tablet_manager.cc:1909] T 92adebc0c69e4a52b975253656132495 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:02.501955 23904 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ded3cee64f9409583040ef4b6e1f803 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:17:02.502089 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 1ded3cee64f9409583040ef4b6e1f803 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ded3cee64f9409583040ef4b6e1f803 already in progress: creating tablet
I20240502 08:17:02.502148 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d7cc1360a7d4379867e75bb7084c4fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:01 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:02.504391 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 92adebc0c69e4a52b975253656132495 (table default.Table_308171ca5d9c4bfdb1d31a3b6e92b62f [id=07b4ec205e8a47d2a5d3b46ce99e8a4d]) successfully deleted
I20240502 08:17:02.504678 24181 tablet_replica.cc:330] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:02.504871 24181 raft_consensus.cc:2229] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:02.505033 24181 raft_consensus.cc:2258] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.505481 24181 ts_tablet_manager.cc:1875] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.509110 25012 tablet_replica.cc:330] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:02.509322 25012 raft_consensus.cc:2229] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:02.509508 25012 raft_consensus.cc:2258] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.510052 25012 ts_tablet_manager.cc:1875] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.510232 24174 tablet_bootstrap.cc:492] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:02.510421 24174 ts_tablet_manager.cc:1367] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.003s
I20240502 08:17:02.511126 24174 raft_consensus.cc:348] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.511360 24174 raft_consensus.cc:374] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.511463 24174 raft_consensus.cc:729] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.511742 24174 consensus_queue.cc:260] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.512360 24174 ts_tablet_manager.cc:1398] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:02.512646 24174 tablet_bootstrap.cc:492] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:02.513264 24180 ts_tablet_manager.cc:1888] T 2b98cc8728c042f4b7f6496fcc5ef913 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:02.513361 24180 log.cc:1196] T 2b98cc8728c042f4b7f6496fcc5ef913 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/2b98cc8728c042f4b7f6496fcc5ef913
I20240502 08:17:02.513684 24180 ts_tablet_manager.cc:1909] T 2b98cc8728c042f4b7f6496fcc5ef913 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:02.514890 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 2b98cc8728c042f4b7f6496fcc5ef913 (table default.Table_5fe628d3da624233941f440e3743da88 [id=ca161ce6780b446fa5743d3974cfa27b]) successfully deleted
I20240502 08:17:02.515133 24180 tablet_replica.cc:330] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:02.515319 24180 raft_consensus.cc:2229] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
W20240502 08:17:02.515429 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 5955411f06e04737bbe0f0d8efc77102 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5955411f06e04737bbe0f0d8efc77102 already in progress: creating tablet
I20240502 08:17:02.515530 24180 pending_rounds.cc:62] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:17:02.515570 24180 pending_rounds.cc:68] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156518158901248 op_type: NO_OP noop_request { }
W20240502 08:17:02.515638 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 6d7cc1360a7d4379867e75bb7084c4fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d7cc1360a7d4379867e75bb7084c4fe already in progress: creating tablet
I20240502 08:17:02.515666 24180 raft_consensus.cc:2866] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:02.515740 24180 raft_consensus.cc:2258] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.516726 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet 5955411f06e04737bbe0f0d8efc77102 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:02.519564 24181 ts_tablet_manager.cc:1888] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:02.520105 24174 tablet_bootstrap.cc:654] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.521035 24181 log.cc:1196] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/e5f5597402374527bf0e5e50178df713
I20240502 08:17:02.522068 24181 ts_tablet_manager.cc:1909] T e5f5597402374527bf0e5e50178df713 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:02.529104 25012 ts_tablet_manager.cc:1888] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:17:02.529220 25012 log.cc:1196] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/a5768cb928e244d2914f10f4eda3c1cd
I20240502 08:17:02.529547 25012 ts_tablet_manager.cc:1909] T a5768cb928e244d2914f10f4eda3c1cd P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:02.529837 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1188) took 2544 ms. Trace:
I20240502 08:17:02.529901 25012 rpcz_store.cc:276] 0502 08:16:59.985693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:59.986040 (+ 347us) service_pool.cc:225] Handling call
0502 08:17:02.509060 (+2523020us) ts_tablet_manager.cc:1120] Deleting tablet a5768cb928e244d2914f10f4eda3c1cd
0502 08:17:02.509069 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.514336 (+ 5267us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.529016 (+ 14680us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.529819 (+ 803us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2522810}
I20240502 08:17:02.530368 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet a5768cb928e244d2914f10f4eda3c1cd (table default.Table_a394560ca3454cb88c3cbc2e87ab4fa7 [id=09f6a10417fb4639a9b97f5bb9efee6b]) successfully deleted
I20240502 08:17:02.536036 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 12d4afaeae25429cb469204690a4d3ec (DEFAULT_TABLE table=default.Table_d731f4904f8649bdbc1369d357489e48 [id=7cefc2e9af9d4923ac5bf936a3faee02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.536581 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12d4afaeae25429cb469204690a4d3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.538026 24174 tablet_bootstrap.cc:492] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:02.538146 24174 ts_tablet_manager.cc:1367] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.004s
I20240502 08:17:02.538776 24174 raft_consensus.cc:348] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.538938 24174 raft_consensus.cc:374] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.538991 24174 raft_consensus.cc:729] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.539155 24174 consensus_queue.cc:260] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.539186 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8fd9957835484b8cbe6bc35c08b4c0 (DEFAULT_TABLE table=default.Table_43f0e7d47b19428e856ead8b62806088 [id=4c481cfd0a7d4e3d9437ab170589747e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.539582 24174 ts_tablet_manager.cc:1398] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:02.539789 24174 tablet_bootstrap.cc:492] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:02.541110 24174 tablet_bootstrap.cc:654] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.542310 24180 ts_tablet_manager.cc:1875] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.552718 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet e5f5597402374527bf0e5e50178df713 (table default.Table_c44e58448899483fb6f36f8ee844e61d [id=ca51b24a648a4f2ea766c671f9058a1c]) successfully deleted
I20240502 08:17:02.553103 24181 tablet_replica.cc:330] T af6efeebaebc495aba017a922e243c39 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:02.553323 24181 raft_consensus.cc:2229] T af6efeebaebc495aba017a922e243c39 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:02.553510 24181 raft_consensus.cc:2258] T af6efeebaebc495aba017a922e243c39 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.554041 24181 ts_tablet_manager.cc:1875] T af6efeebaebc495aba017a922e243c39 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.556996 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8fd9957835484b8cbe6bc35c08b4c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.565871 25012 tablet_replica.cc:330] T a198c33a3e314b5b81a08244862b281a P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:02.573606 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 338fa14b1e004c36accccb3b671103c1 (DEFAULT_TABLE table=default.Table_1bdb95f894a04d6bbeafc73022a2283c [id=13e63816db324aab8601051494492a73]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.574127 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 338fa14b1e004c36accccb3b671103c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.576058 24174 tablet_bootstrap.cc:492] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:02.576200 24174 ts_tablet_manager.cc:1367] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.004s
I20240502 08:17:02.577068 24174 raft_consensus.cc:348] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.577322 24174 raft_consensus.cc:374] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.577399 24174 raft_consensus.cc:729] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.577636 24174 consensus_queue.cc:260] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.579003 24174 ts_tablet_manager.cc:1398] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:02.579908 24174 tablet_bootstrap.cc:492] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:02.581284 24174 tablet_bootstrap.cc:654] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.585860 24173 tablet_bootstrap.cc:654] T 42e11d405a2e4e518a10bf383e9ae6c4 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.588212 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 338fa14b1e004c36accccb3b671103c1 (DEFAULT_TABLE table=default.Table_1bdb95f894a04d6bbeafc73022a2283c [id=13e63816db324aab8601051494492a73]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:02.588778 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 338fa14b1e004c36accccb3b671103c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.591892 24175 tablet_bootstrap.cc:492] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:02.592096 24175 ts_tablet_manager.cc:1367] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.137s user 0.006s sys 0.008s
I20240502 08:17:02.592958 24175 raft_consensus.cc:348] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.593295 24175 raft_consensus.cc:374] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.593398 24175 raft_consensus.cc:729] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.593681 24175 consensus_queue.cc:260] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.595983 24175 ts_tablet_manager.cc:1398] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:17:02.595866 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_59f981f485da4a21a1fd36cd682e1145"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:02.596495 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59f981f485da4a21a1fd36cd682e1145 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:02.596623 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_cffafed96e01457cb67da1cd5033964e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:02.597056 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cffafed96e01457cb67da1cd5033964e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:02.597765 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d1a3f21401914ea6b4e2309ee2f71089" } new_table_name: "default.Table_516a8fe82e0a4559918a434c034efe18" modify_external_catalogs: true
I20240502 08:17:02.597790 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7b2a8b4bd21542489d473a8b383f3e4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:02.598141 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b2a8b4bd21542489d473a8b383f3e4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:02.598690 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e5ad52e5d61a473c941cc2a712798cf0" } new_table_name: "default.Table_33eb8c6488fa4093a60e2adf1308907f" modify_external_catalogs: true
I20240502 08:17:02.600930 24174 tablet_bootstrap.cc:492] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:02.601064 24174 ts_tablet_manager.cc:1367] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.003s
I20240502 08:17:02.602133 24174 raft_consensus.cc:348] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.602295 24174 raft_consensus.cc:374] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.602358 24174 raft_consensus.cc:729] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.602734 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 5f8a585338be4d32acdd21f7bafa8d65
I20240502 08:17:02.604187 24055 raft_consensus.cc:3037] T c45aa0dd26524e0fba27e4eaf17d704e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.606168 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f8a585338be4d32acdd21f7bafa8d65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:02.602555 24174 consensus_queue.cc:260] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.608172 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f8a585338be4d32acdd21f7bafa8d65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:02.608839 24174 ts_tablet_manager.cc:1398] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 08:17:02.609074 24174 tablet_bootstrap.cc:492] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:02.611260 24174 tablet_bootstrap.cc:654] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.616827 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f8a585338be4d32acdd21f7bafa8d65 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:02.636799 24174 tablet_bootstrap.cc:492] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:02.636955 24174 ts_tablet_manager.cc:1367] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.004s
I20240502 08:17:02.637610 24174 raft_consensus.cc:348] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.637773 24174 raft_consensus.cc:374] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.637809 24174 raft_consensus.cc:729] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.639256 25649 raft_consensus.cc:482] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.639401 25649 raft_consensus.cc:504] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.639971 25649 leader_election.cc:288] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:02.640237 24174 consensus_queue.cc:260] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.641620 24174 ts_tablet_manager.cc:1398] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:17:02.641765 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba8f6e017e848d59239ae98432b1837" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:02.642292 23861 leader_election.cc:341] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.642454 24174 tablet_bootstrap.cc:492] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:02.643873 24174 tablet_bootstrap.cc:654] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.647423 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba8f6e017e848d59239ae98432b1837" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:02.647656 23786 raft_consensus.cc:2454] T fba8f6e017e848d59239ae98432b1837 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:02.648074 23858 leader_election.cc:302] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:02.648368 25649 raft_consensus.cc:2781] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:02.648435 25649 raft_consensus.cc:482] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:02.648490 25649 raft_consensus.cc:3037] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.650688 25649 raft_consensus.cc:504] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.651262 25649 leader_election.cc:288] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:02.651542 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba8f6e017e848d59239ae98432b1837" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:02.651716 23786 raft_consensus.cc:3037] T fba8f6e017e848d59239ae98432b1837 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.652081 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba8f6e017e848d59239ae98432b1837" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:17:02.652602 23861 leader_election.cc:341] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.654127 23786 raft_consensus.cc:2454] T fba8f6e017e848d59239ae98432b1837 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:02.654528 23858 leader_election.cc:302] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:02.654767 25649 raft_consensus.cc:2781] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:02.654856 25649 raft_consensus.cc:686] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:17:02.655047 25649 consensus_queue.cc:237] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:17:02.658146 23858 consensus_peers.cc:485] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42e11d405a2e4e518a10bf383e9ae6c4. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:02.659042 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_02e1ffb03a6543299adf3b5b8c6049fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:02.659698 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02e1ffb03a6543299adf3b5b8c6049fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:02.662295 24174 tablet_bootstrap.cc:492] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:02.662413 24174 ts_tablet_manager.cc:1367] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240502 08:17:02.663075 24174 raft_consensus.cc:348] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.663266 24174 raft_consensus.cc:374] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.663312 24174 raft_consensus.cc:729] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.663519 24174 consensus_queue.cc:260] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.664811 24174 ts_tablet_manager.cc:1398] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:02.665735 24174 tablet_bootstrap.cc:492] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:02.667333 24174 tablet_bootstrap.cc:654] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:17:02.668393 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e42383235df4f219aadc2d2e42045ac consensus state has no leader
I20240502 08:17:02.672619 25903 raft_consensus.cc:482] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.672731 25903 raft_consensus.cc:504] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.673242 25903 leader_election.cc:288] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:02.673308 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1d4afdf8dc3e4bf39090955f6cca35e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:02.673823 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d4afdf8dc3e4bf39090955f6cca35e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:02.673936 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088f160a13c4ff09f72bed72f69eaf8" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:02.674355 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088f160a13c4ff09f72bed72f69eaf8" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:17:02.674414 23723 leader_election.cc:341] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.674551 23922 raft_consensus.cc:2454] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:02.674952 23723 leader_election.cc:302] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:02.675395 25903 raft_consensus.cc:2781] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:02.675469 25903 raft_consensus.cc:482] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:02.675516 25903 raft_consensus.cc:3037] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.678325 25903 raft_consensus.cc:504] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.678928 25903 leader_election.cc:288] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:02.679493 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088f160a13c4ff09f72bed72f69eaf8" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:02.679894 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088f160a13c4ff09f72bed72f69eaf8" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:02.680239 23922 raft_consensus.cc:3037] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:17:02.679981 23723 leader_election.cc:341] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.683019 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet c86680b5d8d24c55b9c57916c6d97e2d from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:02.684201 23603 catalog_manager.cc:5484] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:02.684568 23603 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 90d4e2f23c9649a4b1dfb6a179ca7be5 (table default.Table_b2e1c08bfc404c7ebb2720631ad10495 [id=9480d59a76fc401e959a1e41caafc845]). Expected version 3 got 0
I20240502 08:17:02.684967 23603 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 6088f160a13c4ff09f72bed72f69eaf8 (table default.Table_851f246623304504aa9ddf2e6c5e9082 [id=4fecac7b0d9042b3b4f674f3fc1338ba]). Expected version 1 got 0
I20240502 08:17:02.685127 23922 raft_consensus.cc:2454] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:17:02.685170 23603 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet b0cf1a6de3d649b984c50f45393ec0f1 (table default.Table_d416b40fe3d24dafa8db5ee61d13acc0 [id=fa5100e581a7430ab8f0c19b2daae994]). Expected version 1 got 0
I20240502 08:17:02.685626 23723 leader_election.cc:302] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:02.686031 25903 raft_consensus.cc:2781] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:02.686134 25903 raft_consensus.cc:686] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:17:02.686347 25903 consensus_queue.cc:237] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.690151 25814 tablet.cc:1722] T c45aa0dd26524e0fba27e4eaf17d704e P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:17:02.709936 24175 tablet_bootstrap.cc:492] T b0cf1a6de3d649b984c50f45393ec0f1 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:02.711264 24175 tablet_bootstrap.cc:654] T b0cf1a6de3d649b984c50f45393ec0f1 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.714506 25012 raft_consensus.cc:2229] T a198c33a3e314b5b81a08244862b281a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:02.714920 25012 raft_consensus.cc:2258] T a198c33a3e314b5b81a08244862b281a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.715514 25012 ts_tablet_manager.cc:1875] T a198c33a3e314b5b81a08244862b281a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.719640 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8608727bcced4974881d8876c1e1db84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:02.720305 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8608727bcced4974881d8876c1e1db84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:02.725241 25012 ts_tablet_manager.cc:1888] T a198c33a3e314b5b81a08244862b281a P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:02.730619 25012 log.cc:1196] T a198c33a3e314b5b81a08244862b281a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/a198c33a3e314b5b81a08244862b281a
I20240502 08:17:02.731536 25012 ts_tablet_manager.cc:1909] T a198c33a3e314b5b81a08244862b281a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:02.732592 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1227) took 1988 ms. Trace:
I20240502 08:17:02.733088 25012 rpcz_store.cc:276] 0502 08:17:00.744488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:00.744538 (+ 50us) service_pool.cc:225] Handling call
0502 08:17:02.565813 (+1821275us) ts_tablet_manager.cc:1120] Deleting tablet a198c33a3e314b5b81a08244862b281a
0502 08:17:02.565825 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.721509 (+155684us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.725152 (+ 3643us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.732572 (+ 7420us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13654,"tablet-delete.queue_time_us":1821122}
I20240502 08:17:02.734022 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet a198c33a3e314b5b81a08244862b281a (table default.Table_22b226147b894b50a865f33b9f41491c [id=337cf81e0af74e8ca4333e4b1bfe4065]) successfully deleted
I20240502 08:17:02.734058 24181 ts_tablet_manager.cc:1888] T af6efeebaebc495aba017a922e243c39 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:02.734153 24181 log.cc:1196] T af6efeebaebc495aba017a922e243c39 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/af6efeebaebc495aba017a922e243c39
I20240502 08:17:02.734423 24181 ts_tablet_manager.cc:1909] T af6efeebaebc495aba017a922e243c39 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:02.734786 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1228) took 1989 ms. Trace:
I20240502 08:17:02.735137 25012 rpcz_store.cc:276] 0502 08:17:00.745575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:00.745626 (+ 51us) service_pool.cc:225] Handling call
0502 08:17:02.734458 (+1988832us) ts_tablet_manager.cc:1120] Deleting tablet 9850e84117044d2f9dc82b6314dac2cc
0502 08:17:02.734651 (+ 193us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.734769 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1988647}
I20240502 08:17:02.735652 25903 raft_consensus.cc:482] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.735916 25012 tablet_replica.cc:330] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:02.735829 25903 raft_consensus.cc:504] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.736357 25012 raft_consensus.cc:2229] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:02.736955 25903 leader_election.cc:288] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:17:02.737298 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 9850e84117044d2f9dc82b6314dac2cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9850e84117044d2f9dc82b6314dac2cc already in progress: creating tablet
I20240502 08:17:02.737577 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet af6efeebaebc495aba017a922e243c39 (table default.Table_82f82cd6e702420283c288c10f905d90 [id=f293b22818344aea9eb6232141432183]) successfully deleted
I20240502 08:17:02.737772 24180 ts_tablet_manager.cc:1888] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:02.737852 25012 pending_rounds.cc:62] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b: Trying to abort 1 pending ops.
I20240502 08:17:02.737905 24180 log.cc:1196] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/1305dbdac3994224a047df139cf8d518
I20240502 08:17:02.737814 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2792c45255c42adaeefb7aef4e25715" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:02.738013 25012 pending_rounds.cc:68] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156518535565312 op_type: NO_OP noop_request { }
I20240502 08:17:02.738230 24180 ts_tablet_manager.cc:1909] T 1305dbdac3994224a047df139cf8d518 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
W20240502 08:17:02.738534 23723 leader_election.cc:341] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.738441 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2792c45255c42adaeefb7aef4e25715" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:02.738912 25012 raft_consensus.cc:2866] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20240502 08:17:02.739034 23723 leader_election.cc:341] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.739125 23723 leader_election.cc:302] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:02.739253 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 1305dbdac3994224a047df139cf8d518 (table default.Table_37ab1c2d8931468482c9d99dd6536fe5 [id=47b742f54bb14af8a878ac416a991240]) successfully deleted
I20240502 08:17:02.739513 25903 raft_consensus.cc:2726] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:17:02.739934 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet a5768cb928e244d2914f10f4eda3c1cd because the tablet was not found. No further retry: Not found: Tablet not found: a5768cb928e244d2914f10f4eda3c1cd
I20240502 08:17:02.739883 24175 tablet_bootstrap.cc:492] T b0cf1a6de3d649b984c50f45393ec0f1 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
W20240502 08:17:02.740130 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 5f8a585338be4d32acdd21f7bafa8d65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f8a585338be4d32acdd21f7bafa8d65 already in progress: creating tablet
I20240502 08:17:02.739989 25012 raft_consensus.cc:2258] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.740180 24175 ts_tablet_manager.cc:1367] T b0cf1a6de3d649b984c50f45393ec0f1 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.000s
I20240502 08:17:02.741392 24175 raft_consensus.cc:348] T b0cf1a6de3d649b984c50f45393ec0f1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.741664 24175 raft_consensus.cc:374] T b0cf1a6de3d649b984c50f45393ec0f1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.741767 24175 raft_consensus.cc:729] T b0cf1a6de3d649b984c50f45393ec0f1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.742022 24175 consensus_queue.cc:260] T b0cf1a6de3d649b984c50f45393ec0f1 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.748198 24175 ts_tablet_manager.cc:1398] T b0cf1a6de3d649b984c50f45393ec0f1 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 08:17:02.749198 24175 tablet_bootstrap.cc:492] T 0f306a288b90413b9d8ea8b76d23363a P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:02.750618 24175 tablet_bootstrap.cc:654] T 0f306a288b90413b9d8ea8b76d23363a P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.751124 25012 ts_tablet_manager.cc:1875] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.761196 24173 tablet_bootstrap.cc:492] T 42e11d405a2e4e518a10bf383e9ae6c4 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:02.761411 24173 ts_tablet_manager.cc:1367] T 42e11d405a2e4e518a10bf383e9ae6c4 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.620s user 0.018s sys 0.005s
I20240502 08:17:02.762631 24173 raft_consensus.cc:348] T 42e11d405a2e4e518a10bf383e9ae6c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.762939 24173 raft_consensus.cc:374] T 42e11d405a2e4e518a10bf383e9ae6c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.763046 24173 raft_consensus.cc:729] T 42e11d405a2e4e518a10bf383e9ae6c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.763398 24173 consensus_queue.cc:260] T 42e11d405a2e4e518a10bf383e9ae6c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.764272 24173 ts_tablet_manager.cc:1398] T 42e11d405a2e4e518a10bf383e9ae6c4 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240502 08:17:02.779206 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9bd9e392bd0b4e678f441bba813b4549 consensus state has no leader
W20240502 08:17:02.780367 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7cbf1c0a37454beb966aa61b4f8caf72 consensus state has no leader
I20240502 08:17:02.780596 23605 master_service.cc:946] ReplaceTablet: received request to replace tablet b7339f1207924812b4ef2ac7ccffc894 from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:02.786549 24173 tablet_bootstrap.cc:492] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:02.788275 24175 tablet_bootstrap.cc:492] T 0f306a288b90413b9d8ea8b76d23363a P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:02.788419 24175 ts_tablet_manager.cc:1367] T 0f306a288b90413b9d8ea8b76d23363a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20240502 08:17:02.789067 24175 raft_consensus.cc:348] T 0f306a288b90413b9d8ea8b76d23363a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.789248 24175 raft_consensus.cc:374] T 0f306a288b90413b9d8ea8b76d23363a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.789302 24175 raft_consensus.cc:729] T 0f306a288b90413b9d8ea8b76d23363a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.789552 25903 raft_consensus.cc:482] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.789521 24175 consensus_queue.cc:260] T 0f306a288b90413b9d8ea8b76d23363a P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.789683 25903 raft_consensus.cc:504] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.790287 25903 leader_election.cc:288] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:02.790813 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0cf1a6de3d649b984c50f45393ec0f1" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:02.790980 23922 raft_consensus.cc:2454] T b0cf1a6de3d649b984c50f45393ec0f1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:02.791349 23723 leader_election.cc:302] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:17:02.791627 24175 ts_tablet_manager.cc:1398] T 0f306a288b90413b9d8ea8b76d23363a P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:02.791617 25903 raft_consensus.cc:2781] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:02.791680 25903 raft_consensus.cc:482] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:02.791733 25903 raft_consensus.cc:3037] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.792546 24175 tablet_bootstrap.cc:492] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:02.793941 24175 tablet_bootstrap.cc:654] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.794124 25903 raft_consensus.cc:504] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.795239 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0cf1a6de3d649b984c50f45393ec0f1" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:02.795405 23922 raft_consensus.cc:3037] T b0cf1a6de3d649b984c50f45393ec0f1 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.795719 25903 leader_election.cc:288] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:02.797863 23922 raft_consensus.cc:2454] T b0cf1a6de3d649b984c50f45393ec0f1 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:17:02.798228 23723 leader_election.cc:302] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:17:02.798523 25903 raft_consensus.cc:2781] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:02.798661 25903 raft_consensus.cc:686] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:17:02.798921 25903 consensus_queue.cc:237] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.791265 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0cf1a6de3d649b984c50f45393ec0f1" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:02.798242 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0cf1a6de3d649b984c50f45393ec0f1" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:17:02.801498 23723 leader_election.cc:341] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:17:02.801795 23723 leader_election.cc:341] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.803221 25012 ts_tablet_manager.cc:1888] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:02.803364 25012 log.cc:1196] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/1b175e1b55bf4102804666b6fffccead
I20240502 08:17:02.803709 25012 ts_tablet_manager.cc:1909] T 1b175e1b55bf4102804666b6fffccead P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:02.804808 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1229) took 2057 ms. Trace:
I20240502 08:17:02.804919 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_67fda5063740421fb45a174acee7c96a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:02.805437 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67fda5063740421fb45a174acee7c96a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:02.807602 25012 rpcz_store.cc:276] 0502 08:17:00.747589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:00.747631 (+ 42us) service_pool.cc:225] Handling call
0502 08:17:02.735868 (+1988237us) ts_tablet_manager.cc:1120] Deleting tablet 1b175e1b55bf4102804666b6fffccead
0502 08:17:02.735877 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.758861 (+ 22984us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.803119 (+ 44258us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:02.804791 (+ 1672us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1988096}
I20240502 08:17:02.808043 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 1b175e1b55bf4102804666b6fffccead (table default.Table_8c96a3fc08e94e36af627e4c8dddc7ae [id=5a9755c4cebd4ad2ab00ce225b25334c]) successfully deleted
I20240502 08:17:02.809043 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1230) took 2061 ms. Trace:
I20240502 08:17:02.809106 25012 rpcz_store.cc:276] 0502 08:17:00.748031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:00.748068 (+ 37us) service_pool.cc:225] Handling call
0502 08:17:02.808927 (+2060859us) ts_tablet_manager.cc:1120] Deleting tablet 49acc2af1a064dd2b0b60120beed69ba
0502 08:17:02.808935 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.809027 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2060729}
W20240502 08:17:02.809505 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 49acc2af1a064dd2b0b60120beed69ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49acc2af1a064dd2b0b60120beed69ba already in progress: creating tablet
I20240502 08:17:02.810129 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1231) took 2061 ms. Trace:
I20240502 08:17:02.810191 25012 rpcz_store.cc:276] 0502 08:17:00.749108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:00.749144 (+ 36us) service_pool.cc:225] Handling call
0502 08:17:02.810027 (+2060883us) ts_tablet_manager.cc:1120] Deleting tablet 14f54d6ded85434da35c1dfcff52763a
0502 08:17:02.810036 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.810112 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2060764}
W20240502 08:17:02.810539 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 14f54d6ded85434da35c1dfcff52763a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14f54d6ded85434da35c1dfcff52763a already in progress: creating tablet
I20240502 08:17:02.810902 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1232) took 2008 ms. Trace:
I20240502 08:17:02.810961 25012 rpcz_store.cc:276] 0502 08:17:00.802839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:00.802896 (+ 57us) service_pool.cc:225] Handling call
0502 08:17:02.810809 (+2007913us) ts_tablet_manager.cc:1120] Deleting tablet d423a92750c94ee1a08688d60dd2168e
0502 08:17:02.810818 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.810887 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2007747}
W20240502 08:17:02.811273 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet d423a92750c94ee1a08688d60dd2168e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d423a92750c94ee1a08688d60dd2168e already in progress: creating tablet
I20240502 08:17:02.811573 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1239) took 1693 ms. Trace:
I20240502 08:17:02.811631 25012 rpcz_store.cc:276] 0502 08:17:01.118312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.118538 (+ 226us) service_pool.cc:225] Handling call
0502 08:17:02.811482 (+1692944us) ts_tablet_manager.cc:1120] Deleting tablet 5027fa2c80b544b8b1e2a4694bec7454
0502 08:17:02.811490 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.811558 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1692738}
W20240502 08:17:02.811933 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 5027fa2c80b544b8b1e2a4694bec7454 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5027fa2c80b544b8b1e2a4694bec7454 already in progress: creating tablet
I20240502 08:17:02.812265 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1240) took 1693 ms. Trace:
I20240502 08:17:02.812325 25012 rpcz_store.cc:276] 0502 08:17:01.118483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.118853 (+ 370us) service_pool.cc:225] Handling call
0502 08:17:02.812179 (+1693326us) ts_tablet_manager.cc:1120] Deleting tablet df06e0edb57f45e5ac71b568c14769cd
0502 08:17:02.812185 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.812252 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1693189}
W20240502 08:17:02.812901 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet df06e0edb57f45e5ac71b568c14769cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df06e0edb57f45e5ac71b568c14769cd already in progress: creating tablet
I20240502 08:17:02.813288 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1243) took 1682 ms. Trace:
I20240502 08:17:02.813386 25012 rpcz_store.cc:276] 0502 08:17:01.130635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.130672 (+ 37us) service_pool.cc:225] Handling call
0502 08:17:02.813202 (+1682530us) ts_tablet_manager.cc:1120] Deleting tablet 7fe583cbf4af493e82d3ec0508ea0f0b
0502 08:17:02.813210 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.813257 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1682376}
I20240502 08:17:02.813690 24175 tablet_bootstrap.cc:492] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
W20240502 08:17:02.813732 23597 catalog_manager.cc:4797] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 7fe583cbf4af493e82d3ec0508ea0f0b because the tablet was not found. No further retry: Not found: Tablet not found: 7fe583cbf4af493e82d3ec0508ea0f0b
I20240502 08:17:02.813804 24175 ts_tablet_manager.cc:1367] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240502 08:17:02.814018 25012 tablet_replica.cc:330] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:02.814463 24175 raft_consensus.cc:348] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.814661 24175 raft_consensus.cc:374] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.814698 24175 raft_consensus.cc:729] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.814801 25494 raft_consensus.cc:482] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.814901 24175 consensus_queue.cc:260] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.814915 25494 raft_consensus.cc:504] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.815289 24175 ts_tablet_manager.cc:1398] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:17:02.815492 25494 leader_election.cc:288] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:02.815649 24175 tablet_bootstrap.cc:492] T fddac683eda14ec1a0140926380f2c07 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:02.816985 24175 tablet_bootstrap.cc:654] T fddac683eda14ec1a0140926380f2c07 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.819072 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d96cc17d9b4bcf8ce7c4414a63f9cb" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:02.819267 23786 raft_consensus.cc:2454] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:02.819831 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d96cc17d9b4bcf8ce7c4414a63f9cb" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:02.820426 23858 leader_election.cc:302] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:17:02.820650 25832 raft_consensus.cc:2781] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:02.820711 25832 raft_consensus.cc:482] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:02.820766 25832 raft_consensus.cc:3037] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.823040 25832 raft_consensus.cc:504] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.823154 25902 raft_consensus.cc:482] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.823266 25902 raft_consensus.cc:504] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.823518 25832 leader_election.cc:288] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:02.823791 25902 leader_election.cc:288] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:17:02.824092 23861 leader_election.cc:341] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.825002 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0281c8123eab4c48bddc377f5695c6e0" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:02.825690 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d96cc17d9b4bcf8ce7c4414a63f9cb" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:02.826370 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0281c8123eab4c48bddc377f5695c6e0" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:02.826547 23922 raft_consensus.cc:2454] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:02.827008 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d96cc17d9b4bcf8ce7c4414a63f9cb" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:02.827208 23786 raft_consensus.cc:3037] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.829429 24175 tablet_bootstrap.cc:492] T fddac683eda14ec1a0140926380f2c07 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:02.829566 24175 ts_tablet_manager.cc:1367] T fddac683eda14ec1a0140926380f2c07 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
W20240502 08:17:02.829860 23861 leader_election.cc:341] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:17:02.830169 23723 leader_election.cc:341] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.830334 24175 raft_consensus.cc:348] T fddac683eda14ec1a0140926380f2c07 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.830487 24175 raft_consensus.cc:374] T fddac683eda14ec1a0140926380f2c07 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.830528 24175 raft_consensus.cc:729] T fddac683eda14ec1a0140926380f2c07 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.830544 23723 leader_election.cc:302] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:02.830966 24175 consensus_queue.cc:260] T fddac683eda14ec1a0140926380f2c07 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.831727 24175 ts_tablet_manager.cc:1398] T fddac683eda14ec1a0140926380f2c07 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:02.831960 24175 tablet_bootstrap.cc:492] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:02.834379 23610 master_service.cc:946] ReplaceTablet: received request to replace tablet 01143d14aca64adb846c8141ae41e6ac from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:02.835094 23786 raft_consensus.cc:2454] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:02.835458 23858 leader_election.cc:302] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:02.835798 25832 raft_consensus.cc:2781] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:02.835901 25832 raft_consensus.cc:686] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:17:02.836145 24175 tablet_bootstrap.cc:654] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.836092 25832 consensus_queue.cc:237] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.837278 25903 raft_consensus.cc:2781] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:02.837414 25903 raft_consensus.cc:482] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:02.837531 25903 raft_consensus.cc:3037] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.840732 25903 raft_consensus.cc:504] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.842185 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0281c8123eab4c48bddc377f5695c6e0" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:02.842929 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0281c8123eab4c48bddc377f5695c6e0" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:02.843086 23922 raft_consensus.cc:3037] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.844234 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_3859b687d5484bfab843cd42a2f296c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:02.844771 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3859b687d5484bfab843cd42a2f296c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:17:02.846132 23723 leader_election.cc:341] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.846313 25902 raft_consensus.cc:482] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.846436 25902 raft_consensus.cc:504] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.847024 25902 leader_election.cc:288] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:17:02.847517 23597 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 665bbee2d15b4112b651844431535d0e (table default.Table_64f173e9aa5947069cae45a6cb8dd18f [id=427121bc94c247eaa315b6c830770d61], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 665bbee2d15b4112b651844431535d0e consensus state has no leader
I20240502 08:17:02.848469 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b4a4d6a364de8af3b54d99a0ae138" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:02.849382 23723 leader_election.cc:341] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.849268 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b4a4d6a364de8af3b54d99a0ae138" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:02.845568 23922 raft_consensus.cc:2454] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
W20240502 08:17:02.850306 23723 leader_election.cc:341] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.850395 23723 leader_election.cc:302] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:02.850697 23723 leader_election.cc:302] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:02.851058 25846 raft_consensus.cc:2726] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:02.851215 25846 raft_consensus.cc:2781] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:02.851311 25846 raft_consensus.cc:686] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:17:02.851527 25846 consensus_queue.cc:237] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.854548 25494 consensus_queue.cc:1035] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:02.855134 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 5f8a585338be4d32acdd21f7bafa8d65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f8a585338be4d32acdd21f7bafa8d65 already in progress: creating tablet
I20240502 08:17:02.856320 24174 tablet_bootstrap.cc:492] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:02.856452 24174 ts_tablet_manager.cc:1367] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.191s user 0.026s sys 0.045s
I20240502 08:17:02.857120 24174 raft_consensus.cc:348] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.857314 24174 raft_consensus.cc:374] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.857358 24174 raft_consensus.cc:729] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.857545 24174 consensus_queue.cc:260] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.858057 24055 raft_consensus.cc:3037] T 42e11d405a2e4e518a10bf383e9ae6c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.858230 24174 ts_tablet_manager.cc:1398] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:02.858480 24174 tablet_bootstrap.cc:492] T 5e56af7b42cb481a8363d60f49ee3f61 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:02.858925 24173 tablet_bootstrap.cc:654] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.859712 24174 tablet_bootstrap.cc:654] T 5e56af7b42cb481a8363d60f49ee3f61 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.860126 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d40724ccadc148cd91ea36278515b65c" } new_table_name: "default.Table_2ac7535bc8ee4c479fcb7efe8edbb845" modify_external_catalogs: true
I20240502 08:17:02.861042 25494 consensus_queue.cc:1035] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:02.868788 25902 raft_consensus.cc:482] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.868959 25902 raft_consensus.cc:504] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.869655 25902 leader_election.cc:288] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:02.870324 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e0f767c6534a608023e6e6d33ca7a8" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:02.870680 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e0f767c6534a608023e6e6d33ca7a8" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:17:02.871387 23723 leader_election.cc:341] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:17:02.871563 23723 leader_election.cc:341] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.871618 23723 leader_election.cc:302] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:02.871814 25902 raft_consensus.cc:2726] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:02.841555 25903 leader_election.cc:288] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:02.879041 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_891c4bb8dd8d4f1c8de1dc4eacb711a1" } new_table_name: "default.Table_e329f94303224962a1501ef5b2d22d55" modify_external_catalogs: true
I20240502 08:17:02.890040 23602 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet c45aa0dd26524e0fba27e4eaf17d704e (table default.Table_cbe61670c678416995bf27144ab702d9 [id=589deb067c2b4cd086d0277ecfed5126]). Expected version 2 got 0
I20240502 08:17:02.891920 25012 raft_consensus.cc:2229] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:02.891710 23604 catalog_manager.cc:5484] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66). New cstate: current_term: 1 leader_uuid: "5b89a1f0485a44b7ac1710e2369a809b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:02.892088 25012 raft_consensus.cc:2258] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:02.892127 23604 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet fba8f6e017e848d59239ae98432b1837 (table default.Table_c7de500cdfa341798dbbd1cc9ab5e4fb [id=3deaa06a1e88491f9dda6abda2cbdd9f]). Expected version 1 got 0
I20240502 08:17:02.892614 25012 ts_tablet_manager.cc:1875] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:02.908824 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet c86680b5d8d24c55b9c57916c6d97e2d
I20240502 08:17:02.909366 23609 catalog_manager.cc:6554] ReplaceTablet: tablet c86680b5d8d24c55b9c57916c6d97e2d deleted and replaced by tablet 2a68d4921605419d81a581a68adbe5b4
W20240502 08:17:02.910742 23861 consensus_peers.cc:485] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92adebc0c69e4a52b975253656132495. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:02.911583 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet c86680b5d8d24c55b9c57916c6d97e2d with delete_type TABLET_DATA_DELETED (replaced by tablet 2a68d4921605419d81a581a68adbe5b4) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:02.911896 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet c86680b5d8d24c55b9c57916c6d97e2d with delete_type TABLET_DATA_DELETED (replaced by tablet 2a68d4921605419d81a581a68adbe5b4) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:02.912207 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet c86680b5d8d24c55b9c57916c6d97e2d with delete_type TABLET_DATA_DELETED (replaced by tablet 2a68d4921605419d81a581a68adbe5b4) from {username='slave'} at 127.0.0.1:33620
W20240502 08:17:02.912802 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet c86680b5d8d24c55b9c57916c6d97e2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c86680b5d8d24c55b9c57916c6d97e2d already in progress: creating tablet
W20240502 08:17:02.913600 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet c86680b5d8d24c55b9c57916c6d97e2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c86680b5d8d24c55b9c57916c6d97e2d already in progress: creating tablet
I20240502 08:17:02.915001 25846 consensus_queue.cc:1035] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:02.916435 23609 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_540a424c4bfa4484a65f0b24140a4bbb" } modify_external_catalogs: true
I20240502 08:17:02.916607 23609 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_540a424c4bfa4484a65f0b24140a4bbb" } modify_external_catalogs: true
I20240502 08:17:02.927201 23608 master_service.cc:946] ReplaceTablet: received request to replace tablet 30de4d9cb40e4037b99b71bff5667e8c from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:02.928791 24173 tablet_bootstrap.cc:492] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:02.928999 24173 ts_tablet_manager.cc:1367] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.143s user 0.003s sys 0.000s
I20240502 08:17:02.929749 25649 consensus_queue.cc:1035] T f1510777d3cb4ae899542796b5a493ad P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:02.930289 24173 raft_consensus.cc:348] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.930457 24173 raft_consensus.cc:374] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.930518 24173 raft_consensus.cc:729] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.930716 24173 consensus_queue.cc:260] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.931092 25902 raft_consensus.cc:482] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.931174 24173 ts_tablet_manager.cc:1398] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:17:02.931226 25902 raft_consensus.cc:504] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.931404 24173 tablet_bootstrap.cc:492] T 665bbee2d15b4112b651844431535d0e P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:02.931769 25902 leader_election.cc:288] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:02.933418 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a74a6eb6d214d858903b3307e09e9e6" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:02.933722 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a74a6eb6d214d858903b3307e09e9e6" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:17:02.934442 23723 leader_election.cc:341] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:17:02.934669 23723 leader_election.cc:341] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.934782 23723 leader_election.cc:302] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:02.935050 25902 raft_consensus.cc:2726] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:02.942684 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet b7339f1207924812b4ef2ac7ccffc894
I20240502 08:17:02.943096 23605 catalog_manager.cc:6554] ReplaceTablet: tablet b7339f1207924812b4ef2ac7ccffc894 deleted and replaced by tablet c4a343c44863403faea8acc65262cb34
I20240502 08:17:02.944527 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet b7339f1207924812b4ef2ac7ccffc894 with delete_type TABLET_DATA_DELETED (replaced by tablet c4a343c44863403faea8acc65262cb34) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:02.944926 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet b7339f1207924812b4ef2ac7ccffc894 with delete_type TABLET_DATA_DELETED (replaced by tablet c4a343c44863403faea8acc65262cb34) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:02.945470 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet b7339f1207924812b4ef2ac7ccffc894 with delete_type TABLET_DATA_DELETED (replaced by tablet c4a343c44863403faea8acc65262cb34) from {username='slave'} at 127.0.0.1:33620
W20240502 08:17:02.946116 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet b7339f1207924812b4ef2ac7ccffc894 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7339f1207924812b4ef2ac7ccffc894 already in progress: creating tablet
W20240502 08:17:02.949834 23858 consensus_peers.cc:485] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b98cc8728c042f4b7f6496fcc5ef913. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:02.952023 24173 tablet_bootstrap.cc:654] T 665bbee2d15b4112b651844431535d0e P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.953081 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_581ed0c699e94b5bb670d72c63f40197" } modify_external_catalogs: true
I20240502 08:17:02.953260 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_581ed0c699e94b5bb670d72c63f40197" } modify_external_catalogs: true
I20240502 08:17:02.966830 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0a2eb6adda7b49aeb8d231cd2306f093" } modify_external_catalogs: true
I20240502 08:17:02.967022 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0a2eb6adda7b49aeb8d231cd2306f093" } modify_external_catalogs: true
W20240502 08:17:02.968220 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet b7339f1207924812b4ef2ac7ccffc894 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7339f1207924812b4ef2ac7ccffc894 already in progress: creating tablet
I20240502 08:17:02.976523 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_40d78b21141944aea5fe3b77b41a7534" } modify_external_catalogs: true
I20240502 08:17:02.976711 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_40d78b21141944aea5fe3b77b41a7534" } modify_external_catalogs: true
W20240502 08:17:02.984340 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6cdf43567be47c1b9d876017b31d157 consensus state has no leader
W20240502 08:17:02.986675 23723 consensus_peers.cc:485] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:02.988209 24174 tablet_bootstrap.cc:492] T 5e56af7b42cb481a8363d60f49ee3f61 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:02.988279 25832 raft_consensus.cc:482] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:02.988384 24174 ts_tablet_manager.cc:1367] T 5e56af7b42cb481a8363d60f49ee3f61 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.130s user 0.020s sys 0.042s
I20240502 08:17:02.988422 25832 raft_consensus.cc:504] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:02.989193 24174 raft_consensus.cc:348] T 5e56af7b42cb481a8363d60f49ee3f61 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.989413 24174 raft_consensus.cc:374] T 5e56af7b42cb481a8363d60f49ee3f61 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.989490 24174 raft_consensus.cc:729] T 5e56af7b42cb481a8363d60f49ee3f61 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.990103 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1395c9cd077b435d927da88d7bf720fd" } modify_external_catalogs: true
I20240502 08:17:02.990255 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1395c9cd077b435d927da88d7bf720fd" } modify_external_catalogs: true
I20240502 08:17:02.990693 24174 consensus_queue.cc:260] T 5e56af7b42cb481a8363d60f49ee3f61 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:02.991612 24174 ts_tablet_manager.cc:1398] T 5e56af7b42cb481a8363d60f49ee3f61 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:02.991853 24174 tablet_bootstrap.cc:492] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:02.993278 25832 leader_election.cc:288] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:02.993532 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "665bbee2d15b4112b651844431535d0e" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:02.994063 23861 leader_election.cc:341] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:02.995398 24174 tablet_bootstrap.cc:654] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.998626 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "665bbee2d15b4112b651844431535d0e" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:17:03.002959 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0000f1984ab64164868de1ada6638007 consensus state has no leader
I20240502 08:17:03.004524 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8b41e4c4860d4f0882c1a60a9404a218"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.005029 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b41e4c4860d4f0882c1a60a9404a218 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.005335 23786 raft_consensus.cc:2454] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:03.005705 23858 leader_election.cc:302] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:03.006052 25934 raft_consensus.cc:2781] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:03.006158 25934 raft_consensus.cc:482] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:03.006285 25934 raft_consensus.cc:3037] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:17:03.006624 23858 consensus_peers.cc:485] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92adebc0c69e4a52b975253656132495. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:03.011581 24175 tablet_bootstrap.cc:492] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:03.011703 24175 ts_tablet_manager.cc:1367] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.180s user 0.030s sys 0.051s
I20240502 08:17:03.012292 24175 raft_consensus.cc:348] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.012459 24175 raft_consensus.cc:374] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.012501 24175 raft_consensus.cc:729] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.012678 24175 consensus_queue.cc:260] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.013077 24175 ts_tablet_manager.cc:1398] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:03.013267 24175 tablet_bootstrap.cc:492] T d16110e1b1664802a98182c73f1e41d3 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:03.014681 24175 tablet_bootstrap.cc:654] T d16110e1b1664802a98182c73f1e41d3 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.015651 23603 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_50c546a17f484d3cb98a5ca4b3ed63e8 [id=b6dee89ecfb443d99267917d4df78e4d] alter complete (version 3)
I20240502 08:17:03.016788 25012 ts_tablet_manager.cc:1888] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:03.016896 25012 log.cc:1196] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/a4d151770ee44130adfd822518b9b017
I20240502 08:17:03.017268 25012 ts_tablet_manager.cc:1909] T a4d151770ee44130adfd822518b9b017 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:03.017973 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1242) took 1887 ms. Trace:
I20240502 08:17:03.018047 25012 rpcz_store.cc:276] 0502 08:17:01.130551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.130826 (+ 275us) service_pool.cc:225] Handling call
0502 08:17:02.813972 (+1683146us) ts_tablet_manager.cc:1120] Deleting tablet a4d151770ee44130adfd822518b9b017
0502 08:17:02.813983 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:02.908931 (+ 94948us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:03.016683 (+107752us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:03.017952 (+ 1269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1683047}
I20240502 08:17:03.018429 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet a4d151770ee44130adfd822518b9b017 (table default.Table_fb39fa7949f3412c83974132184b5d9f [id=8b62b02f324d4b6cbc7095de109955b5]) successfully deleted
I20240502 08:17:03.019529 25934 raft_consensus.cc:504] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.020555 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "665bbee2d15b4112b651844431535d0e" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:03.020839 23786 raft_consensus.cc:3037] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.022403 24174 tablet_bootstrap.cc:492] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:03.022559 24174 ts_tablet_manager.cc:1367] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.006s
I20240502 08:17:03.023551 23786 raft_consensus.cc:2454] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:03.024588 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "665bbee2d15b4112b651844431535d0e" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:17:03.025086 23861 leader_election.cc:341] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.025149 24173 tablet_bootstrap.cc:492] T 665bbee2d15b4112b651844431535d0e P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:03.023381 24174 raft_consensus.cc:348] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.025302 24173 ts_tablet_manager.cc:1367] T 665bbee2d15b4112b651844431535d0e P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.094s user 0.002s sys 0.001s
I20240502 08:17:03.025384 24174 raft_consensus.cc:374] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.025476 24174 raft_consensus.cc:729] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.025740 23858 leader_election.cc:302] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:03.025702 24174 consensus_queue.cc:260] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.026021 24173 raft_consensus.cc:348] T 665bbee2d15b4112b651844431535d0e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.026208 24173 raft_consensus.cc:374] T 665bbee2d15b4112b651844431535d0e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.026260 24173 raft_consensus.cc:729] T 665bbee2d15b4112b651844431535d0e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.026325 24174 ts_tablet_manager.cc:1398] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:17:03.026638 24174 tablet_bootstrap.cc:492] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:03.026548 24173 consensus_queue.cc:260] T 665bbee2d15b4112b651844431535d0e P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.028290 25649 raft_consensus.cc:2781] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:03.028394 24174 tablet_bootstrap.cc:654] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.029953 25649 raft_consensus.cc:686] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:17:03.030464 24175 tablet_bootstrap.cc:492] T d16110e1b1664802a98182c73f1e41d3 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:03.030578 24175 ts_tablet_manager.cc:1367] T d16110e1b1664802a98182c73f1e41d3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20240502 08:17:03.030730 25649 consensus_queue.cc:237] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.031210 24175 raft_consensus.cc:348] T d16110e1b1664802a98182c73f1e41d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.031369 24175 raft_consensus.cc:374] T d16110e1b1664802a98182c73f1e41d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.031421 24175 raft_consensus.cc:729] T d16110e1b1664802a98182c73f1e41d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.031582 24175 consensus_queue.cc:260] T d16110e1b1664802a98182c73f1e41d3 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.032022 24173 ts_tablet_manager.cc:1398] T 665bbee2d15b4112b651844431535d0e P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 08:17:03.032037 24175 ts_tablet_manager.cc:1398] T d16110e1b1664802a98182c73f1e41d3 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:03.032245 24175 tablet_bootstrap.cc:492] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:03.033607 24175 tablet_bootstrap.cc:654] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.034822 25012 tablet_replica.cc:330] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:03.036759 25012 raft_consensus.cc:2229] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:03.036985 25012 raft_consensus.cc:2258] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.037591 25012 ts_tablet_manager.cc:1875] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.034981 25934 leader_election.cc:288] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:03.039330 24174 tablet_bootstrap.cc:492] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:03.039492 24174 ts_tablet_manager.cc:1367] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240502 08:17:03.040320 24174 raft_consensus.cc:348] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.040544 24174 raft_consensus.cc:374] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.040606 24174 raft_consensus.cc:729] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.040853 24174 consensus_queue.cc:260] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.041376 24174 ts_tablet_manager.cc:1398] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:03.041684 24174 tablet_bootstrap.cc:492] T e9171ebf74424c9aae5e1c8a5118f768 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:03.042261 24173 tablet_bootstrap.cc:492] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:03.043357 24174 tablet_bootstrap.cc:654] T e9171ebf74424c9aae5e1c8a5118f768 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.057207 25903 raft_consensus.cc:482] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:03.057364 25903 raft_consensus.cc:504] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.057932 25903 leader_election.cc:288] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:03.058794 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f306a288b90413b9d8ea8b76d23363a" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:03.058979 23922 raft_consensus.cc:2454] T 0f306a288b90413b9d8ea8b76d23363a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:03.059444 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f306a288b90413b9d8ea8b76d23363a" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:03.061564 23723 leader_election.cc:341] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.061851 23723 leader_election.cc:302] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:03.062323 25903 raft_consensus.cc:2781] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:03.062409 25903 raft_consensus.cc:482] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:03.062465 25903 raft_consensus.cc:3037] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.064884 25903 raft_consensus.cc:504] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.065788 25903 leader_election.cc:288] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:03.066098 24173 tablet_bootstrap.cc:654] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.067498 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f306a288b90413b9d8ea8b76d23363a" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:03.067687 23922 raft_consensus.cc:3037] T 0f306a288b90413b9d8ea8b76d23363a P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.070544 24174 tablet_bootstrap.cc:492] T e9171ebf74424c9aae5e1c8a5118f768 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:03.070684 24174 ts_tablet_manager.cc:1367] T e9171ebf74424c9aae5e1c8a5118f768 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.012s
I20240502 08:17:03.071084 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f306a288b90413b9d8ea8b76d23363a" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:03.071465 24174 raft_consensus.cc:348] T e9171ebf74424c9aae5e1c8a5118f768 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.071593 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 0501575a84ec454780f48574bf3155fe
I20240502 08:17:03.071664 24174 raft_consensus.cc:374] T e9171ebf74424c9aae5e1c8a5118f768 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.071717 24174 raft_consensus.cc:729] T e9171ebf74424c9aae5e1c8a5118f768 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
W20240502 08:17:03.071978 23723 leader_election.cc:341] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.073320 23922 raft_consensus.cc:2454] T 0f306a288b90413b9d8ea8b76d23363a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:17:03.073796 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 0501575a84ec454780f48574bf3155fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.074075 24180 tablet_replica.cc:330] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:03.074271 24180 raft_consensus.cc:2229] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:03.074442 24180 raft_consensus.cc:2258] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.074997 24180 ts_tablet_manager.cc:1875] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.075758 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0501575a84ec454780f48574bf3155fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.076635 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 0501575a84ec454780f48574bf3155fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:03.076886 24181 tablet_replica.cc:330] T 0501575a84ec454780f48574bf3155fe P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:03.077069 24181 raft_consensus.cc:2229] T 0501575a84ec454780f48574bf3155fe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:03.077244 24181 raft_consensus.cc:2258] T 0501575a84ec454780f48574bf3155fe P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.078157 24181 ts_tablet_manager.cc:1875] T 0501575a84ec454780f48574bf3155fe P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.081916 23723 leader_election.cc:302] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:03.081969 24180 ts_tablet_manager.cc:1888] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:17:03.082543 25012 ts_tablet_manager.cc:1888] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:03.082646 25012 log.cc:1196] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/a5a5a410b6314818b10927e35032f95d
I20240502 08:17:03.082206 25902 raft_consensus.cc:2781] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:03.082957 25012 ts_tablet_manager.cc:1909] T a5a5a410b6314818b10927e35032f95d P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:03.083047 25902 raft_consensus.cc:686] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:17:03.083227 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1244) took 1952 ms. Trace:
I20240502 08:17:03.083289 25012 rpcz_store.cc:276] 0502 08:17:01.131192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.131226 (+ 34us) service_pool.cc:225] Handling call
0502 08:17:03.034780 (+1903554us) ts_tablet_manager.cc:1120] Deleting tablet a5a5a410b6314818b10927e35032f95d
0502 08:17:03.034788 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:03.051335 (+ 16547us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:03.082432 (+ 31097us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:03.083209 (+ 777us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1903422}
I20240502 08:17:03.083288 25902 consensus_queue.cc:237] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.083681 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet a5a5a410b6314818b10927e35032f95d (table default.Table_79bb10e705c047138e1c4363673a65e6 [id=e5b9ab35a16943af94e5108dc0d29426]) successfully deleted
I20240502 08:17:03.085773 24181 ts_tablet_manager.cc:1888] T 0501575a84ec454780f48574bf3155fe P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:17:03.085914 24181 log.cc:1196] T 0501575a84ec454780f48574bf3155fe P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/0501575a84ec454780f48574bf3155fe
I20240502 08:17:03.086515 24181 ts_tablet_manager.cc:1909] T 0501575a84ec454780f48574bf3155fe P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:03.086687 25845 raft_consensus.cc:482] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:03.086810 25845 raft_consensus.cc:504] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.087469 25845 leader_election.cc:288] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:03.087661 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 0501575a84ec454780f48574bf3155fe (table default.Table_540a424c4bfa4484a65f0b24140a4bbb [id=5cee878b3a31426c879a1bf95045bb09]) successfully deleted
I20240502 08:17:03.088513 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16110e1b1664802a98182c73f1e41d3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:03.088533 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16110e1b1664802a98182c73f1e41d3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:03.088711 23922 raft_consensus.cc:2454] T d16110e1b1664802a98182c73f1e41d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:03.088725 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ac7aaf3a56b1446482a72525630c9947"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:03.089134 23723 leader_election.cc:302] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
W20240502 08:17:03.089282 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac7aaf3a56b1446482a72525630c9947 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:17:03.089406 23723 leader_election.cc:341] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.089711 25845 raft_consensus.cc:2781] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:03.089790 25845 raft_consensus.cc:482] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:03.089866 25845 raft_consensus.cc:3037] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.092342 25845 raft_consensus.cc:504] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.093024 25845 leader_election.cc:288] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:03.093850 25012 tablet_replica.cc:330] T d1be2c29d3154d9189af3266191f0b3e P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:03.093896 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16110e1b1664802a98182c73f1e41d3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:03.094089 25012 raft_consensus.cc:2229] T d1be2c29d3154d9189af3266191f0b3e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:03.094259 25012 raft_consensus.cc:2258] T d1be2c29d3154d9189af3266191f0b3e P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.094784 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d16110e1b1664802a98182c73f1e41d3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:03.094950 23922 raft_consensus.cc:3037] T d16110e1b1664802a98182c73f1e41d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:17:03.094856 23723 leader_election.cc:341] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.097028 24180 log.cc:1196] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/0501575a84ec454780f48574bf3155fe
I20240502 08:17:03.097522 23922 raft_consensus.cc:2454] T d16110e1b1664802a98182c73f1e41d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:17:03.097935 23723 leader_election.cc:302] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:03.098809 25845 raft_consensus.cc:2781] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:03.098944 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 5f9ba58808ec45c4bf053076c8c1e1a1
I20240502 08:17:03.098981 25845 raft_consensus.cc:686] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:17:03.099212 25845 consensus_queue.cc:237] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.100778 25012 ts_tablet_manager.cc:1875] T d1be2c29d3154d9189af3266191f0b3e P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.102591 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f9ba58808ec45c4bf053076c8c1e1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.103381 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f9ba58808ec45c4bf053076c8c1e1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:03.103886 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f9ba58808ec45c4bf053076c8c1e1a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.097616 24180 ts_tablet_manager.cc:1909] T 0501575a84ec454780f48574bf3155fe P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
W20240502 08:17:03.104503 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 5f9ba58808ec45c4bf053076c8c1e1a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f9ba58808ec45c4bf053076c8c1e1a1 already in progress: creating tablet
I20240502 08:17:03.105374 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_78dbedcd5bd64c8ba2b168eb29d9ca36" } new_table_name: "default.Table_f58d9956fcb946a8821dda27ca3bbe20" modify_external_catalogs: true
I20240502 08:17:03.105549 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 0501575a84ec454780f48574bf3155fe (table default.Table_540a424c4bfa4484a65f0b24140a4bbb [id=5cee878b3a31426c879a1bf95045bb09]) successfully deleted
I20240502 08:17:03.106231 25012 ts_tablet_manager.cc:1888] T d1be2c29d3154d9189af3266191f0b3e P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:17:03.106320 25012 log.cc:1196] T d1be2c29d3154d9189af3266191f0b3e P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/d1be2c29d3154d9189af3266191f0b3e
I20240502 08:17:03.106678 25012 ts_tablet_manager.cc:1909] T d1be2c29d3154d9189af3266191f0b3e P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:03.106968 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1245) took 1921 ms. Trace:
I20240502 08:17:03.107053 25012 rpcz_store.cc:276] 0502 08:17:01.185808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.185863 (+ 55us) service_pool.cc:225] Handling call
0502 08:17:03.093800 (+1907937us) ts_tablet_manager.cc:1120] Deleting tablet d1be2c29d3154d9189af3266191f0b3e
0502 08:17:03.093809 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:03.103652 (+ 9843us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:03.106151 (+ 2499us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:03.106951 (+ 800us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1907721}
I20240502 08:17:03.107502 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet d1be2c29d3154d9189af3266191f0b3e (table default.Table_db86cd2cf86e48ffbc6a513b5c2bd7ab [id=6103f133bcdd4f768b92b37245bf5356]) successfully deleted
I20240502 08:17:03.071919 24174 consensus_queue.cc:260] T e9171ebf74424c9aae5e1c8a5118f768 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.108175 24174 ts_tablet_manager.cc:1398] T e9171ebf74424c9aae5e1c8a5118f768 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.037s user 0.001s sys 0.001s
I20240502 08:17:03.108448 24174 tablet_bootstrap.cc:492] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
W20240502 08:17:03.109941 23723 consensus_peers.cc:485] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:03.110494 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 5f9ba58808ec45c4bf053076c8c1e1a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f9ba58808ec45c4bf053076c8c1e1a1 already in progress: creating tablet
I20240502 08:17:03.111034 24175 tablet_bootstrap.cc:492] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:03.111192 24175 ts_tablet_manager.cc:1367] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.079s user 0.005s sys 0.009s
I20240502 08:17:03.111224 24174 tablet_bootstrap.cc:654] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.111837 24175 raft_consensus.cc:348] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.112475 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 5bbaad9e5f284ee195794e6ffec2c02b
I20240502 08:17:03.114729 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bbaad9e5f284ee195794e6ffec2c02b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.114318 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bbaad9e5f284ee195794e6ffec2c02b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.115475 24175 raft_consensus.cc:374] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.115638 24175 raft_consensus.cc:729] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.115980 24175 consensus_queue.cc:260] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:17:03.116324 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 5bbaad9e5f284ee195794e6ffec2c02b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bbaad9e5f284ee195794e6ffec2c02b already in progress: creating tablet
I20240502 08:17:03.116735 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bbaad9e5f284ee195794e6ffec2c02b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:03.116838 24175 ts_tablet_manager.cc:1398] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:17:03.117086 24175 tablet_bootstrap.cc:492] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:03.118614 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1246) took 1930 ms. Trace:
I20240502 08:17:03.118710 25012 rpcz_store.cc:276] 0502 08:17:01.188632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.188855 (+ 223us) service_pool.cc:225] Handling call
0502 08:17:03.118490 (+1929635us) ts_tablet_manager.cc:1120] Deleting tablet b6bb4a4ccd574f4cadc0b9d81ef4dd5b
0502 08:17:03.118501 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:03.118596 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1929452}
I20240502 08:17:03.119907 24175 tablet_bootstrap.cc:654] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.119940 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1247) took 1929 ms. Trace:
I20240502 08:17:03.121129 25012 rpcz_store.cc:276] 0502 08:17:01.190403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.190536 (+ 133us) service_pool.cc:225] Handling call
0502 08:17:03.119815 (+1929279us) ts_tablet_manager.cc:1120] Deleting tablet f87fe1c2c57748f0a17ee14503890c2f
0502 08:17:03.119827 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:03.119919 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1929147}
W20240502 08:17:03.121534 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet f87fe1c2c57748f0a17ee14503890c2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f87fe1c2c57748f0a17ee14503890c2f already in progress: creating tablet
I20240502 08:17:03.121881 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1250) took 1825 ms. Trace:
I20240502 08:17:03.122013 25012 rpcz_store.cc:276] 0502 08:17:01.296559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.297175 (+ 616us) service_pool.cc:225] Handling call
0502 08:17:03.121758 (+1824583us) ts_tablet_manager.cc:1120] Deleting tablet ff4a333c6dd44a13bc267e885bb9e487
0502 08:17:03.121767 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:03.121861 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1808032}
W20240502 08:17:03.122404 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet ff4a333c6dd44a13bc267e885bb9e487 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff4a333c6dd44a13bc267e885bb9e487 already in progress: creating tablet
I20240502 08:17:03.123306 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1259) took 1530 ms. Trace:
W20240502 08:17:03.124006 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 5bbaad9e5f284ee195794e6ffec2c02b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bbaad9e5f284ee195794e6ffec2c02b already in progress: creating tablet
I20240502 08:17:03.124864 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8e43140476184c94b144c88c14afe815"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.125349 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e43140476184c94b144c88c14afe815 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.126714 25012 rpcz_store.cc:276] 0502 08:17:01.592767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.592817 (+ 50us) service_pool.cc:225] Handling call
0502 08:17:03.123194 (+1530377us) ts_tablet_manager.cc:1120] Deleting tablet 2b9f0b0afc3c47759e0e94ca44cf9711
0502 08:17:03.123205 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:03.123288 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1478966}
W20240502 08:17:03.127067 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 2b9f0b0afc3c47759e0e94ca44cf9711 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b9f0b0afc3c47759e0e94ca44cf9711 already in progress: creating tablet
I20240502 08:17:03.127399 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1260) took 1532 ms. Trace:
I20240502 08:17:03.127478 25012 rpcz_store.cc:276] 0502 08:17:01.595243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.595489 (+ 246us) service_pool.cc:225] Handling call
0502 08:17:03.127276 (+1531787us) ts_tablet_manager.cc:1120] Deleting tablet 336f0eb5085546dba763e0d5c5dd5324
0502 08:17:03.127288 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:03.127383 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1482998}
W20240502 08:17:03.127795 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 336f0eb5085546dba763e0d5c5dd5324 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 336f0eb5085546dba763e0d5c5dd5324 already in progress: creating tablet
I20240502 08:17:03.128348 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1261) took 1528 ms. Trace:
I20240502 08:17:03.128422 25012 rpcz_store.cc:276] 0502 08:17:01.600342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.600411 (+ 69us) service_pool.cc:225] Handling call
0502 08:17:03.128235 (+1527824us) ts_tablet_manager.cc:1120] Deleting tablet 4b52e5c136d84cf1bfc8fadea62491f6
0502 08:17:03.128243 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:03.128323 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1483923}
W20240502 08:17:03.128724 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 4b52e5c136d84cf1bfc8fadea62491f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b52e5c136d84cf1bfc8fadea62491f6 already in progress: creating tablet
I20240502 08:17:03.129159 25012 tablet_replica.cc:330] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:03.129354 25012 raft_consensus.cc:2229] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:03.129492 25012 raft_consensus.cc:2258] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.129973 25012 ts_tablet_manager.cc:1875] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.132795 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet c8763198838c410e86be38794dfc1e39
I20240502 08:17:03.133268 23606 master_service.cc:946] ReplaceTablet: received request to replace tablet 397003e47c78418182e0167c3b876d4e from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:03.134477 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet c8763198838c410e86be38794dfc1e39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.134989 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet c8763198838c410e86be38794dfc1e39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.136471 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet c8763198838c410e86be38794dfc1e39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:17:03.136880 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet c8763198838c410e86be38794dfc1e39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8763198838c410e86be38794dfc1e39 already in progress: creating tablet
W20240502 08:17:03.137616 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet c8763198838c410e86be38794dfc1e39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8763198838c410e86be38794dfc1e39 already in progress: creating tablet
I20240502 08:17:03.142375 25405 raft_consensus.cc:482] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:03.142503 25405 raft_consensus.cc:504] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.143111 25405 leader_election.cc:288] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:03.144165 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b98cc8728c042f4b7f6496fcc5ef913" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:17:03.144941 23993 leader_election.cc:341] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 2b98cc8728c042f4b7f6496fcc5ef913
I20240502 08:17:03.145038 23993 leader_election.cc:302] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:03.145426 25405 raft_consensus.cc:2726] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:03.149374 25012 ts_tablet_manager.cc:1888] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:03.149834 24175 tablet_bootstrap.cc:492] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:03.157055 25012 log.cc:1196] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/91f1ab141585441db46d3027833932b0
I20240502 08:17:03.157028 24175 ts_tablet_manager.cc:1367] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.008s
I20240502 08:17:03.157796 25012 ts_tablet_manager.cc:1909] T 91f1ab141585441db46d3027833932b0 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:03.158377 25902 raft_consensus.cc:482] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:03.158509 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1262) took 1554 ms. Trace:
I20240502 08:17:03.158515 25902 raft_consensus.cc:504] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.158573 25012 rpcz_store.cc:276] 0502 08:17:01.603823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.603876 (+ 53us) service_pool.cc:225] Handling call
0502 08:17:03.129123 (+1525247us) ts_tablet_manager.cc:1120] Deleting tablet 91f1ab141585441db46d3027833932b0
0502 08:17:03.129130 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:03.146688 (+ 17558us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:03.149295 (+ 2607us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:03.158491 (+ 9196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1484776}
I20240502 08:17:03.158927 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 91f1ab141585441db46d3027833932b0 (table default.Table_b141c2b67ffe43ff97d32c9fff1d1fe0 [id=b2163faeeed146d9936a6e1a5cfbc46e]) successfully deleted
I20240502 08:17:03.159094 25902 leader_election.cc:288] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:03.158037 24175 raft_consensus.cc:348] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.159812 24175 raft_consensus.cc:374] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.159866 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c36b622cd94f23b6e7ebdd2ab5b6b3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:03.159874 24175 raft_consensus.cc:729] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.160028 23922 raft_consensus.cc:2454] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:03.160132 24175 consensus_queue.cc:260] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.159729 25012 tablet_replica.cc:330] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:03.160630 24175 ts_tablet_manager.cc:1398] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:17:03.160677 25012 raft_consensus.cc:2229] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:03.160893 24175 tablet_bootstrap.cc:492] T 72ee52bcc61b4816acda5b8140b63504 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:03.161191 25012 pending_rounds.cc:62] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b: Trying to abort 1 pending ops.
I20240502 08:17:03.161237 25012 pending_rounds.cc:68] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156519110012928 op_type: NO_OP noop_request { }
I20240502 08:17:03.161900 25012 raft_consensus.cc:2866] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:03.162124 25012 raft_consensus.cc:2258] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.162607 25012 ts_tablet_manager.cc:1875] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.164311 23723 leader_election.cc:302] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:17:03.164928 25902 raft_consensus.cc:2781] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:03.165000 25902 raft_consensus.cc:482] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:03.165050 25902 raft_consensus.cc:3037] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.167610 25902 raft_consensus.cc:504] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.170214 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c36b622cd94f23b6e7ebdd2ab5b6b3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:03.170397 23922 raft_consensus.cc:3037] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.171602 24175 tablet_bootstrap.cc:654] T 72ee52bcc61b4816acda5b8140b63504 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.172608 23922 raft_consensus.cc:2454] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:17:03.172739 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c36b622cd94f23b6e7ebdd2ab5b6b3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:03.173329 23723 leader_election.cc:302] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:17:03.173604 25845 raft_consensus.cc:2781] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:03.173710 25845 raft_consensus.cc:686] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:17:03.173971 25845 consensus_queue.cc:237] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.175529 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c36b622cd94f23b6e7ebdd2ab5b6b3" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:17:03.176012 23723 leader_election.cc:341] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.176005 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_447f4a9a45334a60a8f4fbb914b1e546" } new_table_name: "default.Table_bc442a3c3c0a4856ae7bf4bcf7960092" modify_external_catalogs: true
W20240502 08:17:03.176447 23723 leader_election.cc:341] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.176894 25902 leader_election.cc:288] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:03.178722 25012 ts_tablet_manager.cc:1888] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:03.178822 25012 log.cc:1196] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/92adebc0c69e4a52b975253656132495
W20240502 08:17:03.180723 23858 consensus_peers.cc:485] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0501575a84ec454780f48574bf3155fe. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:03.180912 25012 ts_tablet_manager.cc:1909] T 92adebc0c69e4a52b975253656132495 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:03.181166 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1263) took 1577 ms. Trace:
I20240502 08:17:03.181221 25012 rpcz_store.cc:276] 0502 08:17:01.604137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.619391 (+ 15254us) service_pool.cc:225] Handling call
0502 08:17:03.159682 (+1540291us) ts_tablet_manager.cc:1120] Deleting tablet 92adebc0c69e4a52b975253656132495
0502 08:17:03.159691 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:03.167473 (+ 7782us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:03.178628 (+ 11155us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:03.181153 (+ 2525us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":159,"tablet-delete.queue_time_us":1512213}
I20240502 08:17:03.182088 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 92adebc0c69e4a52b975253656132495 (table default.Table_308171ca5d9c4bfdb1d31a3b6e92b62f [id=07b4ec205e8a47d2a5d3b46ce99e8a4d]) successfully deleted
I20240502 08:17:03.182618 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1265) took 1450 ms. Trace:
I20240502 08:17:03.182682 25012 rpcz_store.cc:276] 0502 08:17:01.732257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.732626 (+ 369us) service_pool.cc:225] Handling call
0502 08:17:03.182516 (+1449890us) ts_tablet_manager.cc:1120] Deleting tablet 88bf97d0a0a54e828c3b0902976e0500
0502 08:17:03.182524 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:03.182604 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1449701}
W20240502 08:17:03.182996 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 88bf97d0a0a54e828c3b0902976e0500 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88bf97d0a0a54e828c3b0902976e0500 already in progress: creating tablet
I20240502 08:17:03.183672 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1266) took 1344 ms. Trace:
I20240502 08:17:03.183728 25012 rpcz_store.cc:276] 0502 08:17:01.838958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.839131 (+ 173us) service_pool.cc:225] Handling call
0502 08:17:03.183572 (+1344441us) ts_tablet_manager.cc:1120] Deleting tablet c61243a38bfa4964b225eeae0e8cc28e
0502 08:17:03.183580 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:03.183647 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1344256}
W20240502 08:17:03.184012 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet c61243a38bfa4964b225eeae0e8cc28e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c61243a38bfa4964b225eeae0e8cc28e already in progress: creating tablet
I20240502 08:17:03.184569 25012 tablet_replica.cc:330] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:03.184734 25012 raft_consensus.cc:2229] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:03.184990 25012 raft_consensus.cc:2258] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.185142 24055 raft_consensus.cc:3037] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.188856 25012 ts_tablet_manager.cc:1875] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.189865 25649 consensus_queue.cc:1035] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:03.190073 24173 tablet_bootstrap.cc:492] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:03.190214 24173 ts_tablet_manager.cc:1367] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.148s user 0.003s sys 0.000s
I20240502 08:17:03.191135 24173 raft_consensus.cc:348] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.191296 24173 raft_consensus.cc:374] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.191345 24173 raft_consensus.cc:729] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.191512 24173 consensus_queue.cc:260] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.192461 24173 ts_tablet_manager.cc:1398] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:03.192799 24173 tablet_bootstrap.cc:492] T 1b175e1b55bf4102804666b6fffccead P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:03.194674 24173 tablet_bootstrap.cc:654] T 1b175e1b55bf4102804666b6fffccead P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.199435 24175 tablet_bootstrap.cc:492] T 72ee52bcc61b4816acda5b8140b63504 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:03.199584 24175 ts_tablet_manager.cc:1367] T 72ee52bcc61b4816acda5b8140b63504 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20240502 08:17:03.199982 25012 ts_tablet_manager.cc:1888] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:03.200068 25012 log.cc:1196] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/42e11d405a2e4e518a10bf383e9ae6c4
I20240502 08:17:03.200404 25012 ts_tablet_manager.cc:1909] T 42e11d405a2e4e518a10bf383e9ae6c4 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:03.200320 24175 raft_consensus.cc:348] T 72ee52bcc61b4816acda5b8140b63504 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.200513 24175 raft_consensus.cc:374] T 72ee52bcc61b4816acda5b8140b63504 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.200567 24175 raft_consensus.cc:729] T 72ee52bcc61b4816acda5b8140b63504 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.200696 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1267) took 1331 ms. Trace:
I20240502 08:17:03.200783 25012 rpcz_store.cc:276] 0502 08:17:01.868892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.868944 (+ 52us) service_pool.cc:225] Handling call
0502 08:17:03.184531 (+1315587us) ts_tablet_manager.cc:1120] Deleting tablet 42e11d405a2e4e518a10bf383e9ae6c4
0502 08:17:03.184538 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:03.196572 (+ 12034us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:03.199895 (+ 3323us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:03.200680 (+ 785us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1315421}
I20240502 08:17:03.201179 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 42e11d405a2e4e518a10bf383e9ae6c4 (table default.Table_d790560c82534057ab7061352d8c71a2 [id=15cd2d29f86a49e295158ba17e364a23]) successfully deleted
I20240502 08:17:03.204541 25012 tablet_replica.cc:330] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:03.204761 25012 raft_consensus.cc:2229] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:03.204936 25012 raft_consensus.cc:2258] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.205462 25012 ts_tablet_manager.cc:1875] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.206534 24175 consensus_queue.cc:260] T 72ee52bcc61b4816acda5b8140b63504 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.211264 24175 ts_tablet_manager.cc:1398] T 72ee52bcc61b4816acda5b8140b63504 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20240502 08:17:03.214351 24175 tablet_bootstrap.cc:492] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:03.216065 25012 ts_tablet_manager.cc:1888] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:03.216214 25012 log.cc:1196] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/1305dbdac3994224a047df139cf8d518
I20240502 08:17:03.216574 25012 ts_tablet_manager.cc:1909] T 1305dbdac3994224a047df139cf8d518 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:03.217860 25845 raft_consensus.cc:482] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:03.218031 25845 raft_consensus.cc:504] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:03.220352 25845 leader_election.cc:288] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:03.220957 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddac683eda14ec1a0140926380f2c07" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:03.222005 23723 leader_election.cc:341] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.219296 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddac683eda14ec1a0140926380f2c07" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:03.224293 23922 raft_consensus.cc:2454] T fddac683eda14ec1a0140926380f2c07 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:03.225477 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1268) took 1356 ms. Trace:
I20240502 08:17:03.225775 23723 leader_election.cc:302] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:03.225811 25012 rpcz_store.cc:276] 0502 08:17:01.869172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.869345 (+ 173us) service_pool.cc:225] Handling call
0502 08:17:03.204456 (+1335111us) ts_tablet_manager.cc:1120] Deleting tablet 1305dbdac3994224a047df139cf8d518
0502 08:17:03.204467 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:03.213183 (+ 8716us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:03.215956 (+ 2773us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:03.225434 (+ 9478us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1334955}
I20240502 08:17:03.226330 25902 raft_consensus.cc:2781] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:03.226507 25902 raft_consensus.cc:482] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:03.226540 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 1305dbdac3994224a047df139cf8d518 (table default.Table_37ab1c2d8931468482c9d99dd6536fe5 [id=47b742f54bb14af8a878ac416a991240]) successfully deleted
I20240502 08:17:03.226655 25902 raft_consensus.cc:3037] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.226980 25012 tablet_replica.cc:330] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:03.227442 25012 raft_consensus.cc:2229] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:03.227993 25012 pending_rounds.cc:62] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b: Trying to abort 1 pending ops.
I20240502 08:17:03.228370 25012 pending_rounds.cc:68] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156519607046144 op_type: NO_OP noop_request { }
I20240502 08:17:03.228731 25012 raft_consensus.cc:2866] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:03.229055 25012 raft_consensus.cc:2258] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.230082 25902 raft_consensus.cc:504] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:03.230329 25012 ts_tablet_manager.cc:1875] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.231644 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddac683eda14ec1a0140926380f2c07" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:03.231875 23922 raft_consensus.cc:3037] T fddac683eda14ec1a0140926380f2c07 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.236862 23922 raft_consensus.cc:2454] T fddac683eda14ec1a0140926380f2c07 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:17:03.237363 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddac683eda14ec1a0140926380f2c07" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:17:03.238143 23723 leader_election.cc:341] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.238358 23723 leader_election.cc:302] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:03.238670 25845 raft_consensus.cc:2781] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:03.238788 25845 raft_consensus.cc:686] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:17:03.239099 25845 consensus_queue.cc:237] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:03.244319 25902 leader_election.cc:288] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:03.247915 24175 tablet_bootstrap.cc:654] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.250604 24174 tablet_bootstrap.cc:492] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:03.250746 24174 ts_tablet_manager.cc:1367] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.142s user 0.020s sys 0.024s
I20240502 08:17:03.251462 24174 raft_consensus.cc:348] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.251806 24174 raft_consensus.cc:374] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.251873 24174 raft_consensus.cc:729] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.252255 24174 consensus_queue.cc:260] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.252740 24174 ts_tablet_manager.cc:1398] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:03.253008 24174 tablet_bootstrap.cc:492] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:03.254257 24174 tablet_bootstrap.cc:654] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.259522 23603 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_cbe61670c678416995bf27144ab702d9 [id=589deb067c2b4cd086d0277ecfed5126] alter complete (version 2)
W20240502 08:17:03.260888 23603 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.225.65:44121 (request call id 168) took 762 ms (client timeout 1000 ms). Trace:
I20240502 08:17:03.261162 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f306a288b90413b9d8ea8b76d23363a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.261365 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet fddac683eda14ec1a0140926380f2c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.261539 24180 tablet_replica.cc:330] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:03.261775 24180 raft_consensus.cc:2229] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:03.262049 24180 pending_rounds.cc:62] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:17:03.262090 24180 pending_rounds.cc:68] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156523351838720 op_type: NO_OP noop_request { }
I20240502 08:17:03.262205 24180 raft_consensus.cc:2866] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:03.262290 24180 raft_consensus.cc:2258] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.259522 23602 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_2e0fc2a0993e4208b05240b1ddc4d2ad [id=30defee3de2445f7bc5017fb23ab2ee7] alter complete (version 1)
I20240502 08:17:03.262995 24180 ts_tablet_manager.cc:1875] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:17:03.263900 23602 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.22.225.67:46497 (request call id 176) took 762 ms (client timeout 1000 ms). Trace:
W20240502 08:17:03.263967 23602 rpcz_store.cc:269] 0502 08:17:02.501240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:02.625509 (+124269us) service_pool.cc:225] Handling call
0502 08:17:02.642865 (+ 17356us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x616000021d54. stack: 00007f7e39fca756 00007f7e39fca510 00007f7e38ffd48d 0000000000955804 00000000009557cd 00007f7e48b0915d 00007f7e48b09253 00007f7e4583bd41 00007f7e4d42734d 00007f7e4d5ae668 00007f7e44399786 00007f7e443994b5 00007f7e40cd204e 00007f7e40cd0f2e 00007f7e40cd7405 00007f7e40cd7d2f
0502 08:17:03.263882 (+621017us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:17:03.206346 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:17:03.206527 (+ 181us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:17:03.206537 (+ 10us) write_op.cc:427] Acquired schema lock
0502 08:17:03.206542 (+ 5us) tablet.cc:660] Decoding operations
0502 08:17:03.206684 (+ 142us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:17:03.206696 (+ 12us) write_op.cc:633] Partition lock acquired for write op
0502 08:17:03.206700 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0502 08:17:03.206774 (+ 74us) tablet.cc:699] Row locks acquired
0502 08:17:03.206778 (+ 4us) write_op.cc:260] PREPARE: finished
0502 08:17:03.206927 (+ 149us) write_op.cc:270] Start()
0502 08:17:03.206965 (+ 38us) write_op.cc:276] Timestamp: P: 1714637823206895 usec, L: 0
0502 08:17:03.206968 (+ 3us) op_driver.cc:331] REPLICATION: starting
0502 08:17:03.207184 (+ 216us) log.cc:842] Serialized 356 byte log entry
0502 08:17:03.258249 (+ 51065us) op_driver.cc:444] REPLICATION: finished
0502 08:17:03.258978 (+ 729us) write_op.cc:301] APPLY: starting
0502 08:17:03.258994 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0502 08:17:03.259034 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0502 08:17:03.259066 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:17:03.259120 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:17:03.259157 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0502 08:17:03.259169 (+ 12us) write_op.cc:312] APPLY: finished
0502 08:17:03.259238 (+ 69us) log.cc:842] Serialized 23 byte log entry
0502 08:17:03.259292 (+ 54us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:17:03.259306 (+ 14us) write_op.cc:446] Released schema lock
0502 08:17:03.259312 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0502 08:17:03.259365 (+ 53us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:17:03.259370 (+ 5us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":9,"spinlock_wait_cycles":37815680,"child_traces":[["op",{"apply.queue_time_us":673,"apply.run_cpu_time_us":452,"apply.run_wall_time_us":451,"num_ops":1,"prepare.queue_time_us":188618,"prepare.run_cpu_time_us":854,"prepare.run_wall_time_us":1338,"raft.queue_time_us":37,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":5,"replication_time_us":51265,"threads_started":1}]]}
I20240502 08:17:03.265149 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 1305dbdac3994224a047df139cf8d518 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:03.265161 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 18625a1d971c43549b9f80e6046efc9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:03.268638 24180 ts_tablet_manager.cc:1888] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:03.268733 24180 log.cc:1196] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/0f306a288b90413b9d8ea8b76d23363a
I20240502 08:17:03.269097 24180 ts_tablet_manager.cc:1909] T 0f306a288b90413b9d8ea8b76d23363a P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:03.269728 24181 tablet_replica.cc:330] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:03.269918 24181 raft_consensus.cc:2229] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:03.270056 24181 raft_consensus.cc:2258] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:17:03.260972 23603 rpcz_store.cc:269] 0502 08:17:02.498790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:02.603178 (+104388us) service_pool.cc:225] Handling call
0502 08:17:03.260867 (+657689us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0502 08:17:02.880268 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:17:02.880361 (+ 93us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:17:02.880368 (+ 7us) write_op.cc:427] Acquired schema lock
0502 08:17:02.880372 (+ 4us) tablet.cc:660] Decoding operations
0502 08:17:02.880481 (+ 109us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:17:02.880512 (+ 31us) write_op.cc:633] Partition lock acquired for write op
0502 08:17:02.880518 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0502 08:17:02.880584 (+ 66us) tablet.cc:699] Row locks acquired
0502 08:17:02.880588 (+ 4us) write_op.cc:260] PREPARE: finished
0502 08:17:02.880621 (+ 33us) write_op.cc:270] Start()
0502 08:17:02.880654 (+ 33us) write_op.cc:276] Timestamp: P: 1714637822880577 usec, L: 0
0502 08:17:02.880659 (+ 5us) op_driver.cc:331] REPLICATION: starting
0502 08:17:02.880852 (+ 193us) log.cc:842] Serialized 533 byte log entry
0502 08:17:02.884164 (+ 3312us) op_driver.cc:444] REPLICATION: finished
0502 08:17:02.887309 (+ 3145us) write_op.cc:301] APPLY: starting
0502 08:17:02.887324 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0502 08:17:02.887359 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0502 08:17:02.887385 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:17:02.887576 (+ 191us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:17:02.887617 (+ 41us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0502 08:17:02.887629 (+ 12us) write_op.cc:312] APPLY: finished
0502 08:17:02.887707 (+ 78us) log.cc:842] Serialized 23 byte log entry
0502 08:17:02.887939 (+ 232us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:17:02.887956 (+ 17us) write_op.cc:446] Released schema lock
0502 08:17:02.887961 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0502 08:17:02.893029 (+ 5068us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:17:02.893037 (+ 8us) write_op.cc:446] Released schema lock
Related trace 'op':
0502 08:17:03.009083 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:17:03.009175 (+ 92us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:17:03.009180 (+ 5us) write_op.cc:427] Acquired schema lock
0502 08:17:03.009182 (+ 2us) tablet.cc:660] Decoding operations
0502 08:17:03.009591 (+ 409us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:17:03.009603 (+ 12us) write_op.cc:633] Partition lock acquired for write op
0502 08:17:03.009607 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0502 08:17:03.009664 (+ 57us) tablet.cc:699] Row locks acquired
0502 08:17:03.009667 (+ 3us) write_op.cc:260] PREPARE: finished
0502 08:17:03.009696 (+ 29us) write_op.cc:270] Start()
0502 08:17:03.009722 (+ 26us) write_op.cc:276] Timestamp: P: 1714637823009644 usec, L: 0
0502 08:17:03.009725 (+ 3us) op_driver.cc:331] REPLICATION: starting
0502 08:17:03.010721 (+ 996us) log.cc:842] Serialized 356 byte log entry
0502 08:17:03.015077 (+ 4356us) op_driver.cc:444] REPLICATION: finished
0502 08:17:03.015225 (+ 148us) write_op.cc:301] APPLY: starting
0502 08:17:03.015239 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0502 08:17:03.015277 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0502 08:17:03.015300 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:17:03.015349 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:17:03.015380 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0502 08:17:03.015386 (+ 6us) write_op.cc:312] APPLY: finished
0502 08:17:03.015449 (+ 63us) log.cc:842] Serialized 23 byte log entry
0502 08:17:03.015498 (+ 49us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:17:03.015515 (+ 17us) write_op.cc:446] Released schema lock
0502 08:17:03.015520 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0502 08:17:03.018788 (+ 3268us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:17:03.018793 (+ 5us) write_op.cc:446] Released schema lock
Related trace 'op':
0502 08:17:03.178098 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0502 08:17:03.178206 (+ 108us) write_op.cc:424] Acquiring schema lock in shared mode
0502 08:17:03.178212 (+ 6us) write_op.cc:427] Acquired schema lock
0502 08:17:03.178214 (+ 2us) tablet.cc:660] Decoding operations
0502 08:17:03.178308 (+ 94us) write_op.cc:612] Acquiring the partition lock for write op
0502 08:17:03.178320 (+ 12us) write_op.cc:633] Partition lock acquired for write op
0502 08:17:03.178324 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0502 08:17:03.178440 (+ 116us) tablet.cc:699] Row locks acquired
0502 08:17:03.178443 (+ 3us) write_op.cc:260] PREPARE: finished
0502 08:17:03.178478 (+ 35us) write_op.cc:270] Start()
0502 08:17:03.178513 (+ 35us) write_op.cc:276] Timestamp: P: 1714637823178456 usec, L: 0
0502 08:17:03.178518 (+ 5us) op_driver.cc:331] REPLICATION: starting
0502 08:17:03.178686 (+ 168us) log.cc:842] Serialized 356 byte log entry
0502 08:17:03.258027 (+ 79341us) op_driver.cc:444] REPLICATION: finished
0502 08:17:03.258413 (+ 386us) write_op.cc:301] APPLY: starting
0502 08:17:03.258433 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0502 08:17:03.258478 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0502 08:17:03.258507 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0502 08:17:03.258568 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0502 08:17:03.258602 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0502 08:17:03.258610 (+ 8us) write_op.cc:312] APPLY: finished
0502 08:17:03.258690 (+ 80us) log.cc:842] Serialized 23 byte log entry
0502 08:17:03.258764 (+ 74us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:17:03.258784 (+ 20us) write_op.cc:446] Released schema lock
0502 08:17:03.258792 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0502 08:17:03.258854 (+ 62us) write_op.cc:481] Releasing partition, row and schema locks
0502 08:17:03.258860 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":3089,"apply.run_cpu_time_us":499,"apply.run_wall_time_us":5783,"num_ops":1,"prepare.queue_time_us":194323,"prepare.run_cpu_time_us":752,"prepare.run_wall_time_us":20752,"raft.queue_time_us":70,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":608,"replication_time_us":3479}],["op",{"apply.queue_time_us":107,"apply.run_cpu_time_us":410,"apply.run_wall_time_us":3610,"num_ops":1,"prepare.queue_time_us":120114,"prepare.run_cpu_time_us":753,"prepare.run_wall_time_us":14514,"raft.queue_time_us":364,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":482,"replication_time_us":5331}],["op",{"apply.queue_time_us":333,"apply.run_cpu_time_us":502,"apply.run_wall_time_us":501,"num_ops":1,"prepare.queue_time_us":161262,"prepare.run_cpu_time_us":777,"prepare.run_wall_time_us":28199,"raft.queue_time_us":478,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":285,"replication_time_us":79472}]]}
I20240502 08:17:03.270531 24181 ts_tablet_manager.cc:1875] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.249447 25012 ts_tablet_manager.cc:1888] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:03.271354 25012 log.cc:1196] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/2b98cc8728c042f4b7f6496fcc5ef913
I20240502 08:17:03.271826 25012 ts_tablet_manager.cc:1909] T 2b98cc8728c042f4b7f6496fcc5ef913 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:03.276517 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1270) took 1406 ms. Trace:
I20240502 08:17:03.276624 25012 rpcz_store.cc:276] 0502 08:17:01.870137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.870176 (+ 39us) service_pool.cc:225] Handling call
0502 08:17:03.226913 (+1356737us) ts_tablet_manager.cc:1120] Deleting tablet 2b98cc8728c042f4b7f6496fcc5ef913
0502 08:17:03.226923 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:03.245407 (+ 18484us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:03.249346 (+ 3939us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:03.276463 (+ 27117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17792,"tablet-delete.queue_time_us":1356561}
I20240502 08:17:03.276934 24175 tablet_bootstrap.cc:492] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:03.277104 24175 ts_tablet_manager.cc:1367] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.002s
I20240502 08:17:03.277175 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 2b98cc8728c042f4b7f6496fcc5ef913 (table default.Table_5fe628d3da624233941f440e3743da88 [id=ca161ce6780b446fa5743d3974cfa27b]) successfully deleted
I20240502 08:17:03.277663 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1271) took 1223 ms. Trace:
I20240502 08:17:03.277805 25012 rpcz_store.cc:276] 0502 08:17:02.054135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:02.054560 (+ 425us) service_pool.cc:225] Handling call
0502 08:17:03.277483 (+1222923us) ts_tablet_manager.cc:1120] Deleting tablet 91167335ab4441a1a2fc8d70cebdd2eb
0502 08:17:03.277491 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:03.277620 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1222765}
I20240502 08:17:03.277904 24175 raft_consensus.cc:348] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.278100 24175 raft_consensus.cc:374] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.278182 24175 raft_consensus.cc:729] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
W20240502 08:17:03.278366 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 91167335ab4441a1a2fc8d70cebdd2eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91167335ab4441a1a2fc8d70cebdd2eb already in progress: creating tablet
I20240502 08:17:03.278443 24175 consensus_queue.cc:260] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.278795 25012 tablet_replica.cc:330] T 18625a1d971c43549b9f80e6046efc9c P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:03.279088 24175 ts_tablet_manager.cc:1398] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:03.279232 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 0f306a288b90413b9d8ea8b76d23363a (table default.Table_9bb2be9b511a49eca4228b582c5ac739 [id=97c877d344bd44d896a989e63768579b]) successfully deleted
I20240502 08:17:03.279361 24175 tablet_bootstrap.cc:492] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:03.279680 25012 raft_consensus.cc:2229] T 18625a1d971c43549b9f80e6046efc9c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:03.280301 25012 raft_consensus.cc:2258] T 18625a1d971c43549b9f80e6046efc9c P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.280874 24175 tablet_bootstrap.cc:654] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.280973 25012 ts_tablet_manager.cc:1875] T 18625a1d971c43549b9f80e6046efc9c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.283643 24181 ts_tablet_manager.cc:1888] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:03.283771 24181 log.cc:1196] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/1305dbdac3994224a047df139cf8d518
I20240502 08:17:03.284103 24181 ts_tablet_manager.cc:1909] T 1305dbdac3994224a047df139cf8d518 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:03.284674 24175 tablet_bootstrap.cc:492] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:03.284811 24175 ts_tablet_manager.cc:1367] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:17:03.285100 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 1305dbdac3994224a047df139cf8d518 (table default.Table_37ab1c2d8931468482c9d99dd6536fe5 [id=47b742f54bb14af8a878ac416a991240]) successfully deleted
I20240502 08:17:03.285423 24180 tablet_replica.cc:330] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:03.285454 24175 raft_consensus.cc:348] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.285650 24175 raft_consensus.cc:374] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.285648 24180 raft_consensus.cc:2229] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:03.285718 24175 raft_consensus.cc:729] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.285945 24180 pending_rounds.cc:62] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:17:03.285989 24180 pending_rounds.cc:68] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156523990171648 op_type: NO_OP noop_request { }
I20240502 08:17:03.285914 24175 consensus_queue.cc:260] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.286144 24180 raft_consensus.cc:2866] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:03.286509 24180 raft_consensus.cc:2258] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.287456 24180 ts_tablet_manager.cc:1875] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.288359 24175 ts_tablet_manager.cc:1398] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:03.288637 24175 tablet_bootstrap.cc:492] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:03.289872 24175 tablet_bootstrap.cc:654] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:17:03.292299 23993 consensus_peers.cc:485] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18625a1d971c43549b9f80e6046efc9c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:03.293390 24173 tablet_bootstrap.cc:492] T 1b175e1b55bf4102804666b6fffccead P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:03.294418 24173 ts_tablet_manager.cc:1367] T 1b175e1b55bf4102804666b6fffccead P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.102s user 0.003s sys 0.000s
W20240502 08:17:03.294649 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d7e8b340d0f40e88930324eb76c0ead has no consensus state
I20240502 08:17:03.294696 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_4e5d657dc235492dad016da586febee0" } new_table_name: "default.Table_5a9deff891ad40538ea9f2574ee0ff55" modify_external_catalogs: true
I20240502 08:17:03.296180 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5805b20a437a4a3784a19f373a194bc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.296659 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5805b20a437a4a3784a19f373a194bc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.298125 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_54324a5422aa4797a3b221716afe0cf3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.298516 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54324a5422aa4797a3b221716afe0cf3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.298558 24173 raft_consensus.cc:348] T 1b175e1b55bf4102804666b6fffccead P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.301694 24173 raft_consensus.cc:374] T 1b175e1b55bf4102804666b6fffccead P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.301817 24173 raft_consensus.cc:729] T 1b175e1b55bf4102804666b6fffccead P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.302182 24181 tablet_replica.cc:330] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:03.302107 24173 consensus_queue.cc:260] T 1b175e1b55bf4102804666b6fffccead P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.302382 24181 raft_consensus.cc:2229] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:03.302527 24175 tablet_bootstrap.cc:492] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:03.302624 24180 ts_tablet_manager.cc:1888] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:03.302673 24175 ts_tablet_manager.cc:1367] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240502 08:17:03.302817 24181 pending_rounds.cc:62] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5: Trying to abort 1 pending ops.
I20240502 08:17:03.302912 24181 pending_rounds.cc:68] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156520324694016 op_type: NO_OP noop_request { }
I20240502 08:17:03.303053 24181 raft_consensus.cc:2866] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:03.303431 24181 raft_consensus.cc:2258] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.303324 24175 raft_consensus.cc:348] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.303545 24175 raft_consensus.cc:374] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.303627 24175 raft_consensus.cc:729] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.303797 24175 consensus_queue.cc:260] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.304371 24175 ts_tablet_manager.cc:1398] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:03.304373 24181 ts_tablet_manager.cc:1875] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.304646 24175 tablet_bootstrap.cc:492] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:03.304803 24180 log.cc:1196] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/fddac683eda14ec1a0140926380f2c07
I20240502 08:17:03.306085 24175 tablet_bootstrap.cc:654] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.306532 24180 ts_tablet_manager.cc:1909] T fddac683eda14ec1a0140926380f2c07 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:03.308151 24173 ts_tablet_manager.cc:1398] T 1b175e1b55bf4102804666b6fffccead P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 08:17:03.308560 24173 tablet_bootstrap.cc:492] T 6088f160a13c4ff09f72bed72f69eaf8 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:03.309547 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet fddac683eda14ec1a0140926380f2c07 (table default.Table_4b967ef2b2fc4e2caa4219afcae841a6 [id=c32fc96889554c32bdf1d4550633e91f]) successfully deleted
W20240502 08:17:03.309929 25867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:03.310824 25867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:03.311200 25867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 08:17:03.311633 25832 raft_consensus.cc:482] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:03.311766 25832 raft_consensus.cc:504] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.312319 25832 leader_election.cc:288] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:17:03.313807 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9171ebf74424c9aae5e1c8a5118f768" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:03.314244 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9171ebf74424c9aae5e1c8a5118f768" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:03.314446 23786 raft_consensus.cc:2454] T e9171ebf74424c9aae5e1c8a5118f768 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
W20240502 08:17:03.317966 23861 leader_election.cc:341] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.318043 24181 ts_tablet_manager.cc:1888] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:03.318130 24181 log.cc:1196] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/18625a1d971c43549b9f80e6046efc9c
I20240502 08:17:03.318455 24181 ts_tablet_manager.cc:1909] T 18625a1d971c43549b9f80e6046efc9c P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:03.318794 23858 leader_election.cc:302] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:03.319671 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 18625a1d971c43549b9f80e6046efc9c (table default.Table_e02044df8b8c4ca9a9bdccaaf14a2eb4 [id=62ad2ebd7db541958a77f09c07ab946e]) successfully deleted
I20240502 08:17:03.320021 25832 raft_consensus.cc:2781] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:03.320122 25832 raft_consensus.cc:482] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:03.320204 25832 raft_consensus.cc:3037] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.320569 24175 tablet_bootstrap.cc:492] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:03.320689 24175 ts_tablet_manager.cc:1367] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240502 08:17:03.321331 24175 raft_consensus.cc:348] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.321539 24175 raft_consensus.cc:374] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.321652 24175 raft_consensus.cc:729] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.321846 24175 consensus_queue.cc:260] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.322464 24175 ts_tablet_manager.cc:1398] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:03.322728 24175 tablet_bootstrap.cc:492] T 5e56af7b42cb481a8363d60f49ee3f61 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:03.323177 25832 raft_consensus.cc:504] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.324203 24175 tablet_bootstrap.cc:654] T 5e56af7b42cb481a8363d60f49ee3f61 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.325395 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9171ebf74424c9aae5e1c8a5118f768" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:17:03.325939 23861 leader_election.cc:341] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.326864 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9171ebf74424c9aae5e1c8a5118f768" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:03.327066 23786 raft_consensus.cc:3037] T e9171ebf74424c9aae5e1c8a5118f768 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:17:03.328243 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 106c39807ac14e7fba27fcec9f129642 consensus state has no leader
W20240502 08:17:03.328716 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 79a36b64095f445ba21a6304b887a112 consensus state has no leader
I20240502 08:17:03.330145 24175 tablet_bootstrap.cc:492] T 5e56af7b42cb481a8363d60f49ee3f61 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:03.330277 24175 ts_tablet_manager.cc:1367] T 5e56af7b42cb481a8363d60f49ee3f61 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:17:03.330965 24175 raft_consensus.cc:348] T 5e56af7b42cb481a8363d60f49ee3f61 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:03.331158 24175 raft_consensus.cc:374] T 5e56af7b42cb481a8363d60f49ee3f61 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.331241 24175 raft_consensus.cc:729] T 5e56af7b42cb481a8363d60f49ee3f61 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.331467 24175 consensus_queue.cc:260] T 5e56af7b42cb481a8363d60f49ee3f61 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:03.331717 25832 leader_election.cc:288] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:17:03.332033 23786 raft_consensus.cc:2454] T e9171ebf74424c9aae5e1c8a5118f768 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:03.332129 24175 ts_tablet_manager.cc:1398] T 5e56af7b42cb481a8363d60f49ee3f61 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:03.332417 24175 tablet_bootstrap.cc:492] T b2eac3bde41c4415bdb97fc53a5b395f P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:03.332918 23858 leader_election.cc:302] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:03.333729 25832 raft_consensus.cc:2781] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:03.334404 25832 raft_consensus.cc:686] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:17:03.335017 25832 consensus_queue.cc:237] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:17:03.335577 25867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20240502 08:17:03.333969 24175 tablet_bootstrap.cc:654] T b2eac3bde41c4415bdb97fc53a5b395f P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:17:03.336535 25867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:03.336683 25867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:17:03.336822 25867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240502 08:17:03.337406 23595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.126:43265, user_credentials={real_user=slave}} blocked reactor thread for 52243.2us
W20240502 08:17:03.340466 23595 consensus_peers.cc:485] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25 -> Peer bcc8a323e6b9429eaaf4e0c1698ad077 (127.22.225.126:43265): Couldn't send request to peer bcc8a323e6b9429eaaf4e0c1698ad077. Status: Network error: Client connection negotiation failed: client connection to 127.22.225.126:43265: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:17:03.341348 24173 tablet_bootstrap.cc:654] T 6088f160a13c4ff09f72bed72f69eaf8 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.342182 24174 tablet_bootstrap.cc:492] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:03.342309 24174 ts_tablet_manager.cc:1367] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.000s
I20240502 08:17:03.343014 24174 raft_consensus.cc:348] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.343183 24174 raft_consensus.cc:374] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.343228 24174 raft_consensus.cc:729] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.343400 24174 consensus_queue.cc:260] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.343992 24174 ts_tablet_manager.cc:1398] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:03.344247 24174 tablet_bootstrap.cc:492] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:03.345669 24174 tablet_bootstrap.cc:654] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.346441 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_273425ba746d4d198b31df2a796dcc35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.347030 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_273425ba746d4d198b31df2a796dcc35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.348348 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e46ef84488044afc9e7deb0fb0375871"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.348769 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e46ef84488044afc9e7deb0fb0375871 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.349690 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_9fc71ad3aa2d407a9db02e57623dc854"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.350071 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9fc71ad3aa2d407a9db02e57623dc854 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.351655 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_773220dd4c35411ab8824a3f2368ca11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.352021 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_773220dd4c35411ab8824a3f2368ca11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.355711 24173 tablet_bootstrap.cc:492] T 6088f160a13c4ff09f72bed72f69eaf8 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:03.357115 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7e7adde66ddc4951a3af46dd52d0c3b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.357550 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e7adde66ddc4951a3af46dd52d0c3b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.360213 25867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.225.84:40725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.22.225.126:43265,127.22.225.125:36189,127.22.225.124:41745
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.225.126:43265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.225.126
--webserver_port=34035
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:03.362059 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e329f94303224962a1501ef5b2d22d55" } new_table_name: "default.Table_a3e486ef9b114960bb32eda6125949a0" modify_external_catalogs: true
I20240502 08:17:03.367058 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2c882aad1a40484297ed2b6af34de3c6" } new_table_name: "default.Table_21d2bfc37ee54973b6cad5d7093cd913" modify_external_catalogs: true
I20240502 08:17:03.368615 25867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:03.370141 25012 ts_tablet_manager.cc:1888] T 18625a1d971c43549b9f80e6046efc9c P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:03.370272 25012 log.cc:1196] T 18625a1d971c43549b9f80e6046efc9c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/18625a1d971c43549b9f80e6046efc9c
I20240502 08:17:03.371742 24173 ts_tablet_manager.cc:1367] T 6088f160a13c4ff09f72bed72f69eaf8 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.002s
I20240502 08:17:03.372172 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_32c806f7aa664335b36da222d5fd5618" } modify_external_catalogs: true
I20240502 08:17:03.372308 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_32c806f7aa664335b36da222d5fd5618" } modify_external_catalogs: true
I20240502 08:17:03.372483 24173 raft_consensus.cc:348] T 6088f160a13c4ff09f72bed72f69eaf8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.372781 25867 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:17:03.374955 25012 ts_tablet_manager.cc:1909] T 18625a1d971c43549b9f80e6046efc9c P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:03.375321 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1273) took 1167 ms. Trace:
I20240502 08:17:03.375442 25012 rpcz_store.cc:276] 0502 08:17:02.207909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:02.207968 (+ 59us) service_pool.cc:225] Handling call
0502 08:17:03.278726 (+1070758us) ts_tablet_manager.cc:1120] Deleting tablet 18625a1d971c43549b9f80e6046efc9c
0502 08:17:03.278736 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:03.309327 (+ 30591us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:03.369983 (+ 60656us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:03.375289 (+ 5306us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":495,"tablet-delete.queue_time_us":1067622}
I20240502 08:17:03.375946 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 18625a1d971c43549b9f80e6046efc9c (table default.Table_e02044df8b8c4ca9a9bdccaaf14a2eb4 [id=62ad2ebd7db541958a77f09c07ab946e]) successfully deleted
I20240502 08:17:03.376228 24173 raft_consensus.cc:374] T 6088f160a13c4ff09f72bed72f69eaf8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.376379 24173 raft_consensus.cc:729] T 6088f160a13c4ff09f72bed72f69eaf8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.376674 24173 consensus_queue.cc:260] T 6088f160a13c4ff09f72bed72f69eaf8 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.376798 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b1e5c38f8b45471698ed791873ecdb99" } modify_external_catalogs: true
I20240502 08:17:03.376955 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b1e5c38f8b45471698ed791873ecdb99" } modify_external_catalogs: true
W20240502 08:17:03.381101 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0000f1984ab64164868de1ada6638007 consensus state has no leader
I20240502 08:17:03.382212 24173 ts_tablet_manager.cc:1398] T 6088f160a13c4ff09f72bed72f69eaf8 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240502 08:17:03.382579 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0e95f8c694d7437c8aa8cb258bfcbe79" } modify_external_catalogs: true
I20240502 08:17:03.382712 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_0e95f8c694d7437c8aa8cb258bfcbe79" } modify_external_catalogs: true
W20240502 08:17:03.387367 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 3922e528a97a4287a0122a5b51035dcb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3922e528a97a4287a0122a5b51035dcb already in progress: creating tablet
W20240502 08:17:03.392302 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 69246bf9f6be42b9a0671ae68a633255 consensus state has no leader
I20240502 08:17:03.396024 24173 tablet_bootstrap.cc:492] T fba8f6e017e848d59239ae98432b1837 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:17:03.396620 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 12d4afaeae25429cb469204690a4d3ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12d4afaeae25429cb469204690a4d3ec already in progress: creating tablet
I20240502 08:17:03.398102 24173 tablet_bootstrap.cc:654] T fba8f6e017e848d59239ae98432b1837 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:17:03.401515 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 1ded3cee64f9409583040ef4b6e1f803 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ded3cee64f9409583040ef4b6e1f803 already in progress: creating tablet
W20240502 08:17:03.402040 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 6d7cc1360a7d4379867e75bb7084c4fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d7cc1360a7d4379867e75bb7084c4fe already in progress: creating tablet
W20240502 08:17:03.402679 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 5955411f06e04737bbe0f0d8efc77102 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5955411f06e04737bbe0f0d8efc77102 already in progress: creating tablet
I20240502 08:17:03.402909 24175 tablet_bootstrap.cc:492] T b2eac3bde41c4415bdb97fc53a5b395f P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:03.403065 24175 ts_tablet_manager.cc:1367] T b2eac3bde41c4415bdb97fc53a5b395f P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.071s user 0.020s sys 0.012s
W20240502 08:17:03.403229 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 5f8a585338be4d32acdd21f7bafa8d65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f8a585338be4d32acdd21f7bafa8d65 already in progress: creating tablet
W20240502 08:17:03.403762 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet c86680b5d8d24c55b9c57916c6d97e2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c86680b5d8d24c55b9c57916c6d97e2d already in progress: creating tablet
I20240502 08:17:03.403822 24175 raft_consensus.cc:348] T b2eac3bde41c4415bdb97fc53a5b395f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.404062 24175 raft_consensus.cc:374] T b2eac3bde41c4415bdb97fc53a5b395f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.404153 24175 raft_consensus.cc:729] T b2eac3bde41c4415bdb97fc53a5b395f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
W20240502 08:17:03.404342 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet b7339f1207924812b4ef2ac7ccffc894 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7339f1207924812b4ef2ac7ccffc894 already in progress: creating tablet
I20240502 08:17:03.404428 24175 consensus_queue.cc:260] T b2eac3bde41c4415bdb97fc53a5b395f P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.405647 24175 ts_tablet_manager.cc:1398] T b2eac3bde41c4415bdb97fc53a5b395f P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:03.406107 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b2b68fb4c3c14505b960a7ec8623b0e2" } modify_external_catalogs: true
I20240502 08:17:03.406298 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b2b68fb4c3c14505b960a7ec8623b0e2" } modify_external_catalogs: true
I20240502 08:17:03.407708 25012 tablet_replica.cc:330] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:03.407909 25012 raft_consensus.cc:2229] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:03.408231 25012 raft_consensus.cc:2258] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:17:03.409003 25975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:03.410640 25976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:03.415115 25902 raft_consensus.cc:482] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:03.415292 25902 raft_consensus.cc:504] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.416045 25902 leader_election.cc:288] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:03.417029 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ee52bcc61b4816acda5b8140b63504" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:03.417773 23723 leader_election.cc:341] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.414412 25832 raft_consensus.cc:482] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:03.409512 24175 tablet_bootstrap.cc:492] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:03.417667 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ee52bcc61b4816acda5b8140b63504" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:03.431949 23922 raft_consensus.cc:2454] T 72ee52bcc61b4816acda5b8140b63504 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:03.432798 23723 leader_election.cc:302] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:03.433218 24175 tablet_bootstrap.cc:654] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.433595 25845 raft_consensus.cc:2781] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:03.433694 25845 raft_consensus.cc:482] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:03.433769 25845 raft_consensus.cc:3037] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.437371 25845 raft_consensus.cc:504] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.439111 25903 consensus_queue.cc:1035] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:03.439919 25845 leader_election.cc:288] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:03.440493 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ee52bcc61b4816acda5b8140b63504" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:03.441561 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ee52bcc61b4816acda5b8140b63504" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:03.441756 23920 raft_consensus.cc:3037] T 72ee52bcc61b4816acda5b8140b63504 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:17:03.441704 23723 leader_election.cc:341] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.409267 25012 ts_tablet_manager.cc:1875] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.444865 23920 raft_consensus.cc:2454] T 72ee52bcc61b4816acda5b8140b63504 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:17:03.445351 23723 leader_election.cc:302] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:03.445937 25845 raft_consensus.cc:2781] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:03.446197 25845 raft_consensus.cc:686] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:17:03.446503 25845 consensus_queue.cc:237] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.450743 24175 tablet_bootstrap.cc:492] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:03.450873 24175 ts_tablet_manager.cc:1367] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.000s
I20240502 08:17:03.451514 24175 raft_consensus.cc:348] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:03.451694 24175 raft_consensus.cc:374] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.451750 24175 raft_consensus.cc:729] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.452297 24175 consensus_queue.cc:260] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:03.454867 24175 ts_tablet_manager.cc:1398] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:03.456826 24175 tablet_bootstrap.cc:492] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:03.458343 24175 tablet_bootstrap.cc:654] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.460564 25012 ts_tablet_manager.cc:1888] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:17:03.460687 25012 log.cc:1196] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/0501575a84ec454780f48574bf3155fe
I20240502 08:17:03.461104 25012 ts_tablet_manager.cc:1909] T 0501575a84ec454780f48574bf3155fe P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:03.461720 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 0501575a84ec454780f48574bf3155fe (table default.Table_540a424c4bfa4484a65f0b24140a4bbb [id=5cee878b3a31426c879a1bf95045bb09]) successfully deleted
I20240502 08:17:03.462996 24173 tablet_bootstrap.cc:492] T fba8f6e017e848d59239ae98432b1837 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:03.466938 24174 tablet_bootstrap.cc:492] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:03.467110 24174 ts_tablet_manager.cc:1367] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.123s user 0.027s sys 0.034s
I20240502 08:17:03.467803 24174 raft_consensus.cc:348] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.467988 24174 raft_consensus.cc:374] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.468041 24174 raft_consensus.cc:729] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.468924 24174 consensus_queue.cc:260] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.471467 24174 ts_tablet_manager.cc:1398] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:17:03.472656 24174 tablet_bootstrap.cc:492] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:03.473567 24175 tablet_bootstrap.cc:492] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:03.473727 24175 ts_tablet_manager.cc:1367] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.005s
I20240502 08:17:03.474412 24175 raft_consensus.cc:348] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.474592 24175 raft_consensus.cc:374] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.474649 24175 raft_consensus.cc:729] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.474076 24174 tablet_bootstrap.cc:654] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.480091 24175 consensus_queue.cc:260] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.481434 24173 ts_tablet_manager.cc:1367] T fba8f6e017e848d59239ae98432b1837 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.085s user 0.006s sys 0.000s
I20240502 08:17:03.482093 24175 ts_tablet_manager.cc:1398] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 08:17:03.483713 24175 tablet_bootstrap.cc:492] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:03.485126 24175 tablet_bootstrap.cc:654] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.487221 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 0082d71f92424d4d8e55f8dfd739889e
I20240502 08:17:03.488126 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1fd8c397b2b846f0b1a34f57047b988e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.488741 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fd8c397b2b846f0b1a34f57047b988e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.489430 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 14075c65ed7a4aa78166943aa32f904f
I20240502 08:17:03.483618 24173 raft_consensus.cc:348] T fba8f6e017e848d59239ae98432b1837 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.489760 24173 raft_consensus.cc:374] T fba8f6e017e848d59239ae98432b1837 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.489831 24173 raft_consensus.cc:729] T fba8f6e017e848d59239ae98432b1837 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.490042 23610 master_service.cc:946] ReplaceTablet: received request to replace tablet b9f83bcf640f465bb95d87a991cc5ff6 from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:03.490386 24173 consensus_queue.cc:260] T fba8f6e017e848d59239ae98432b1837 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.492658 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0082d71f92424d4d8e55f8dfd739889e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.493002 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 0082d71f92424d4d8e55f8dfd739889e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:03.493237 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 14075c65ed7a4aa78166943aa32f904f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:03.493472 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet 14075c65ed7a4aa78166943aa32f904f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:17:03.495211 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 0082d71f92424d4d8e55f8dfd739889e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0082d71f92424d4d8e55f8dfd739889e already in progress: creating tablet
W20240502 08:17:03.495536 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 14075c65ed7a4aa78166943aa32f904f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14075c65ed7a4aa78166943aa32f904f already in progress: creating tablet
I20240502 08:17:03.496959 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 0082d71f92424d4d8e55f8dfd739889e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.497202 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 14075c65ed7a4aa78166943aa32f904f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:17:03.498430 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 0082d71f92424d4d8e55f8dfd739889e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0082d71f92424d4d8e55f8dfd739889e already in progress: creating tablet
W20240502 08:17:03.498656 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 14075c65ed7a4aa78166943aa32f904f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14075c65ed7a4aa78166943aa32f904f already in progress: creating tablet
I20240502 08:17:03.500320 24173 ts_tablet_manager.cc:1398] T fba8f6e017e848d59239ae98432b1837 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240502 08:17:03.508133 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet b9f83bcf640f465bb95d87a991cc5ff6
I20240502 08:17:03.508636 23610 catalog_manager.cc:6554] ReplaceTablet: tablet b9f83bcf640f465bb95d87a991cc5ff6 deleted and replaced by tablet 9cb9cb0164164b43bf08c2a8532b766b
I20240502 08:17:03.509812 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f83bcf640f465bb95d87a991cc5ff6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9cb9cb0164164b43bf08c2a8532b766b) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.510304 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f83bcf640f465bb95d87a991cc5ff6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9cb9cb0164164b43bf08c2a8532b766b) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:03.510529 24181 tablet_replica.cc:330] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:03.510689 24181 raft_consensus.cc:2229] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:03.510690 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f83bcf640f465bb95d87a991cc5ff6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9cb9cb0164164b43bf08c2a8532b766b) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.510833 24181 raft_consensus.cc:2258] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.511194 24180 tablet_replica.cc:330] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:03.511346 24181 ts_tablet_manager.cc:1875] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.511376 24180 raft_consensus.cc:2229] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 2 LEADER]: Raft consensus shutting down.
I20240502 08:17:03.511526 24173 tablet_bootstrap.cc:492] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:03.511646 24180 raft_consensus.cc:2258] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.512107 24180 ts_tablet_manager.cc:1875] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.515703 24175 tablet_bootstrap.cc:492] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:03.515820 24175 ts_tablet_manager.cc:1367] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.012s
I20240502 08:17:03.516868 24175 raft_consensus.cc:348] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.517171 24175 raft_consensus.cc:374] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.517237 24175 raft_consensus.cc:729] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.517566 24175 consensus_queue.cc:260] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.518203 24180 ts_tablet_manager.cc:1888] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20240502 08:17:03.518294 24180 log.cc:1196] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/b9f83bcf640f465bb95d87a991cc5ff6
I20240502 08:17:03.520670 24180 ts_tablet_manager.cc:1909] T b9f83bcf640f465bb95d87a991cc5ff6 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:03.522138 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet b9f83bcf640f465bb95d87a991cc5ff6 (table default.Table_e00515baeaae4ce9ad93000143f2a53a [id=13293a9ab7004e8a965e004f2fb5a14a]) successfully deleted
I20240502 08:17:03.524132 24181 ts_tablet_manager.cc:1888] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20240502 08:17:03.524257 24181 log.cc:1196] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/b9f83bcf640f465bb95d87a991cc5ff6
I20240502 08:17:03.524425 25845 raft_consensus.cc:482] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:03.524598 24181 ts_tablet_manager.cc:1909] T b9f83bcf640f465bb95d87a991cc5ff6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:03.524611 25845 raft_consensus.cc:504] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:17:03.525023 23434 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 101160us
I20240502 08:17:03.526491 24175 ts_tablet_manager.cc:1398] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240502 08:17:03.527669 24175 tablet_bootstrap.cc:492] T 91a3455e5931488da3134d6b5c18aef2 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:03.527681 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet b9f83bcf640f465bb95d87a991cc5ff6 (table default.Table_e00515baeaae4ce9ad93000143f2a53a [id=13293a9ab7004e8a965e004f2fb5a14a]) successfully deleted
I20240502 08:17:03.527880 25867 server_base.cc:1034] running on GCE node
I20240502 08:17:03.529232 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2eac3bde41c4415bdb97fc53a5b395f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:03.529369 25980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:03.529852 23723 leader_election.cc:341] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.530519 25867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:03.530768 24175 tablet_bootstrap.cc:654] T 91a3455e5931488da3134d6b5c18aef2 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.534474 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 30de4d9cb40e4037b99b71bff5667e8c
I20240502 08:17:03.535015 23608 catalog_manager.cc:6554] ReplaceTablet: tablet 30de4d9cb40e4037b99b71bff5667e8c deleted and replaced by tablet a165732a666542d3a5c92ee4e303af77
I20240502 08:17:03.536015 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 30de4d9cb40e4037b99b71bff5667e8c with delete_type TABLET_DATA_DELETED (replaced by tablet a165732a666542d3a5c92ee4e303af77) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:03.536182 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 30de4d9cb40e4037b99b71bff5667e8c with delete_type TABLET_DATA_DELETED (replaced by tablet a165732a666542d3a5c92ee4e303af77) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.536413 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet 30de4d9cb40e4037b99b71bff5667e8c with delete_type TABLET_DATA_DELETED (replaced by tablet a165732a666542d3a5c92ee4e303af77) from {username='slave'} at 127.0.0.1:33620
W20240502 08:17:03.536676 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 30de4d9cb40e4037b99b71bff5667e8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30de4d9cb40e4037b99b71bff5667e8c already in progress: creating tablet
I20240502 08:17:03.538237 25845 leader_election.cc:288] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:03.538259 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2eac3bde41c4415bdb97fc53a5b395f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:03.538456 23920 raft_consensus.cc:2454] T b2eac3bde41c4415bdb97fc53a5b395f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:03.538846 23723 leader_election.cc:302] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:03.539336 25845 raft_consensus.cc:2781] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240502 08:17:03.540225 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 30de4d9cb40e4037b99b71bff5667e8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30de4d9cb40e4037b99b71bff5667e8c already in progress: creating tablet
I20240502 08:17:03.540519 25845 raft_consensus.cc:482] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:03.540709 25845 raft_consensus.cc:3037] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.541826 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2e0fc2a0993e4208b05240b1ddc4d2ad" } new_table_name: "default.Table_c36327219de24751921b49cc4beed5fc" modify_external_catalogs: true
I20240502 08:17:03.544303 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c30eb6e23bfb485d8071c183499e1ac2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.544876 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 5f9ba58808ec45c4bf053076c8c1e1a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f9ba58808ec45c4bf053076c8c1e1a1 already in progress: creating tablet
W20240502 08:17:03.544929 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c30eb6e23bfb485d8071c183499e1ac2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.545934 24173 tablet_bootstrap.cc:654] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:17:03.552301 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 5bbaad9e5f284ee195794e6ffec2c02b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bbaad9e5f284ee195794e6ffec2c02b already in progress: creating tablet
W20240502 08:17:03.552935 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet c8763198838c410e86be38794dfc1e39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8763198838c410e86be38794dfc1e39 already in progress: creating tablet
I20240502 08:17:03.554044 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 397003e47c78418182e0167c3b876d4e
I20240502 08:17:03.554486 23606 catalog_manager.cc:6554] ReplaceTablet: tablet 397003e47c78418182e0167c3b876d4e deleted and replaced by tablet b30e5c761e0040ad8c32a25a0d2906e3
I20240502 08:17:03.554766 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_34217583459a498eaa704feafc2a5939" } new_table_name: "default.Table_2dd52fe14c3a4a18926430f365b64763" modify_external_catalogs: true
I20240502 08:17:03.554944 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 397003e47c78418182e0167c3b876d4e with delete_type TABLET_DATA_DELETED (replaced by tablet b30e5c761e0040ad8c32a25a0d2906e3) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:03.555135 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet 397003e47c78418182e0167c3b876d4e with delete_type TABLET_DATA_DELETED (replaced by tablet b30e5c761e0040ad8c32a25a0d2906e3) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.556229 23605 catalog_manager.cc:5484] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66). New cstate: current_term: 1 leader_uuid: "5b89a1f0485a44b7ac1710e2369a809b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:03.556550 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 397003e47c78418182e0167c3b876d4e with delete_type TABLET_DATA_DELETED (replaced by tablet b30e5c761e0040ad8c32a25a0d2906e3) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.556655 23605 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 665bbee2d15b4112b651844431535d0e (table default.Table_64f173e9aa5947069cae45a6cb8dd18f [id=427121bc94c247eaa315b6c830770d61]). Expected version 3 got 0
W20240502 08:17:03.556797 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 0082d71f92424d4d8e55f8dfd739889e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0082d71f92424d4d8e55f8dfd739889e already in progress: creating tablet
W20240502 08:17:03.556998 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 397003e47c78418182e0167c3b876d4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 397003e47c78418182e0167c3b876d4e already in progress: creating tablet
I20240502 08:17:03.557273 23605 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 07a9d13d3a2d4e6dbc04998c3dd7e1a8 (table default.Table_d9f68c6b26d442bbbd485bb89ee5564c [id=7a49ccfdd6884281ba9b882debd04871]). Expected version 1 got 0
I20240502 08:17:03.557420 23605 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet b0cf1a6de3d649b984c50f45393ec0f1 (table default.Table_d416b40fe3d24dafa8db5ee61d13acc0 [id=fa5100e581a7430ab8f0c19b2daae994]). Expected version 1 got 0
I20240502 08:17:03.558990 23605 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet fba8f6e017e848d59239ae98432b1837 (table default.Table_c7de500cdfa341798dbbd1cc9ab5e4fb [id=3deaa06a1e88491f9dda6abda2cbdd9f]). Expected version 1 got 0
I20240502 08:17:03.559556 23605 catalog_manager.cc:5484] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66). New cstate: current_term: 1 leader_uuid: "5b89a1f0485a44b7ac1710e2369a809b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: HEALTHY } } }
W20240502 08:17:03.560421 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 397003e47c78418182e0167c3b876d4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 397003e47c78418182e0167c3b876d4e already in progress: creating tablet
I20240502 08:17:03.563197 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 6e35efdc0434429aa093fec7770782ef
I20240502 08:17:03.563324 25845 raft_consensus.cc:504] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:17:03.564072 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 14075c65ed7a4aa78166943aa32f904f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14075c65ed7a4aa78166943aa32f904f already in progress: creating tablet
I20240502 08:17:03.564342 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e35efdc0434429aa093fec7770782ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.564630 25012 tablet_replica.cc:330] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:03.564813 25012 raft_consensus.cc:2229] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:03.564777 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e35efdc0434429aa093fec7770782ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:03.564963 25012 raft_consensus.cc:2258] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.565008 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2eac3bde41c4415bdb97fc53a5b395f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:03.565488 25012 ts_tablet_manager.cc:1875] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.565510 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 5604212dd32c430ab35f8c50fb9d7023
W20240502 08:17:03.566309 23723 leader_election.cc:341] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.567013 25867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20240502 08:17:03.566078 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 6e35efdc0434429aa093fec7770782ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e35efdc0434429aa093fec7770782ef already in progress: creating tablet
I20240502 08:17:03.568500 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 5604212dd32c430ab35f8c50fb9d7023 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.568668 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet 5604212dd32c430ab35f8c50fb9d7023 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:03.568403 25845 leader_election.cc:288] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:17:03.569545 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 5604212dd32c430ab35f8c50fb9d7023 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5604212dd32c430ab35f8c50fb9d7023 already in progress: creating tablet
I20240502 08:17:03.570634 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2eac3bde41c4415bdb97fc53a5b395f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:03.570840 23920 raft_consensus.cc:3037] T b2eac3bde41c4415bdb97fc53a5b395f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.571110 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 5604212dd32c430ab35f8c50fb9d7023 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.571110 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e35efdc0434429aa093fec7770782ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:17:03.571991 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 5604212dd32c430ab35f8c50fb9d7023 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5604212dd32c430ab35f8c50fb9d7023 already in progress: creating tablet
W20240502 08:17:03.572208 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 6e35efdc0434429aa093fec7770782ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e35efdc0434429aa093fec7770782ef already in progress: creating tablet
I20240502 08:17:03.573398 23920 raft_consensus.cc:2454] T b2eac3bde41c4415bdb97fc53a5b395f P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:17:03.574124 23723 leader_election.cc:302] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:03.574487 25902 consensus_queue.cc:1035] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:03.574868 25902 raft_consensus.cc:2781] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:03.574965 25902 raft_consensus.cc:686] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:17:03.575229 25902 consensus_queue.cc:237] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.581693 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 97f6e11bcded40de8e0087b8361b9036 (DEFAULT_TABLE table=default.Table_59f981f485da4a21a1fd36cd682e1145 [id=d72e296e24c54089be122bfa03535336]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.582067 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f6e11bcded40de8e0087b8361b9036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.586236 24173 tablet_bootstrap.cc:492] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:03.586401 24173 ts_tablet_manager.cc:1367] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.075s user 0.002s sys 0.002s
I20240502 08:17:03.588662 25832 raft_consensus.cc:504] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.591549 24173 raft_consensus.cc:348] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:03.594539 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_f59e6ff978214561a7e00a415c70835f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.595081 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f59e6ff978214561a7e00a415c70835f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.596827 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_f5dcf3c3174742b0ad367dda4fb666bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.597455 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5dcf3c3174742b0ad367dda4fb666bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.597702 25867 hybrid_clock.cc:648] HybridClock initialized: now 1714637823597536 us; error 142 us; skew 500 ppm
I20240502 08:17:03.598258 25867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:03.598840 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_597cd56369f34933bb8b8d05e8bc6c0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.599252 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_597cd56369f34933bb8b8d05e8bc6c0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.599362 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 97f6e11bcded40de8e0087b8361b9036 (DEFAULT_TABLE table=default.Table_59f981f485da4a21a1fd36cd682e1145 [id=d72e296e24c54089be122bfa03535336]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.600296 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7cdeb95f30074a40b9daa0ccb4e47a04" } new_table_name: "default.Table_45691f9b0fd44328a709aa6a3e3b1f0d" modify_external_catalogs: true
I20240502 08:17:03.600421 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f6e11bcded40de8e0087b8361b9036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.601920 23608 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c646218554fc4d108a83a3db92322f1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.602484 23608 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c646218554fc4d108a83a3db92322f1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.605460 24174 tablet_bootstrap.cc:492] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:03.605628 24174 ts_tablet_manager.cc:1367] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.133s user 0.010s sys 0.009s
I20240502 08:17:03.606345 24174 raft_consensus.cc:348] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.606540 24174 raft_consensus.cc:374] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.606596 24174 raft_consensus.cc:729] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.606781 24174 consensus_queue.cc:260] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:17:03.609789 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1be95e0576c143e5a94c7aaf8d3e0625 consensus state has no leader
I20240502 08:17:03.610738 24175 tablet_bootstrap.cc:492] T 91a3455e5931488da3134d6b5c18aef2 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:03.610875 24175 ts_tablet_manager.cc:1367] T 91a3455e5931488da3134d6b5c18aef2 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.003s
I20240502 08:17:03.611507 24175 raft_consensus.cc:348] T 91a3455e5931488da3134d6b5c18aef2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.611707 24175 raft_consensus.cc:374] T 91a3455e5931488da3134d6b5c18aef2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.611757 24175 raft_consensus.cc:729] T 91a3455e5931488da3134d6b5c18aef2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.612097 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet ea1ceb374483425194f962b5b5b87438
I20240502 08:17:03.613422 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet ea1ceb374483425194f962b5b5b87438 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.616500 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ce8849dfab4243758ff61836ba4a7070"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.617125 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce8849dfab4243758ff61836ba4a7070 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.619560 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet d16110e1b1664802a98182c73f1e41d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.619629 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet bb4b4a4d6a364de8af3b54d99a0ae138 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.619967 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0501575a84ec454780f48574bf3155fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.619988 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ee52bcc61b4816acda5b8140b63504 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.620692 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f306a288b90413b9d8ea8b76d23363a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.620689 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0281c8123eab4c48bddc377f5695c6e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.621307 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_35a5601b06fc48a88a18e298ebc76964"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:03.621668 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet e9171ebf74424c9aae5e1c8a5118f768 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.621812 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet fddac683eda14ec1a0140926380f2c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
W20240502 08:17:03.621903 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35a5601b06fc48a88a18e298ebc76964 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.622182 23609 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 90d4e2f23c9649a4b1dfb6a179ca7be5 (table default.Table_b2e1c08bfc404c7ebb2720631ad10495 [id=9480d59a76fc401e959a1e41caafc845]). Expected version 3 got 0
I20240502 08:17:03.622392 23609 catalog_manager.cc:5484] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:03.622748 23609 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet b0cf1a6de3d649b984c50f45393ec0f1 (table default.Table_d416b40fe3d24dafa8db5ee61d13acc0 [id=fa5100e581a7430ab8f0c19b2daae994]). Expected version 1 got 0
I20240502 08:17:03.622987 23609 catalog_manager.cc:5484] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:03.623327 23609 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 6088f160a13c4ff09f72bed72f69eaf8 (table default.Table_851f246623304504aa9ddf2e6c5e9082 [id=4fecac7b0d9042b3b4f674f3fc1338ba]). Expected version 1 got 0
I20240502 08:17:03.623509 23609 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 156c7356ece342aea0d8db667ebad89c (table default.Table_d8c6955b89724769bbfd81db2b5be455 [id=15b52f0688c24df2b6f3fda3732ceeb4]). Expected version 1 got 0
I20240502 08:17:03.623695 23609 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 9761df0bbdee44af90135bffb8b5828f (table default.Table_8ba240c391da43978f93a27dce48e557 [id=1b5536950db244d1978b154da0be33e0]). Expected version 2 got 0
I20240502 08:17:03.623925 23609 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 07a9d13d3a2d4e6dbc04998c3dd7e1a8 (table default.Table_d9f68c6b26d442bbbd485bb89ee5564c [id=7a49ccfdd6884281ba9b882debd04871]). Expected version 1 got 0
I20240502 08:17:03.624246 23609 catalog_manager.cc:5484] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:03.628952 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ec9c41f8b19f478ba8524c3c13311bdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.629415 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec9c41f8b19f478ba8524c3c13311bdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.630031 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 97f6e11bcded40de8e0087b8361b9036 (DEFAULT_TABLE table=default.Table_59f981f485da4a21a1fd36cd682e1145 [id=d72e296e24c54089be122bfa03535336]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.630504 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f6e11bcded40de8e0087b8361b9036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.630513 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_15a09f13f7024d2cbe6d5c0fec986fd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.630870 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15a09f13f7024d2cbe6d5c0fec986fd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.636698 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b5566be23d3744348216cab2e9a0c6bf" } new_table_name: "default.Table_dd18d96e98ec4c1ca5e651b6d9e5a4dc" modify_external_catalogs: true
I20240502 08:17:03.636795 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9edd6347f7cb47229111f08382effee5" } new_table_name: "default.Table_94fc9fc0e27b43c8807dd10d6c3f94a6" modify_external_catalogs: true
I20240502 08:17:03.638352 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet ea1ceb374483425194f962b5b5b87438 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:17:03.638870 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet ea1ceb374483425194f962b5b5b87438 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea1ceb374483425194f962b5b5b87438 already in progress: creating tablet
I20240502 08:17:03.639508 24175 consensus_queue.cc:260] T 91a3455e5931488da3134d6b5c18aef2 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.640270 24175 ts_tablet_manager.cc:1398] T 91a3455e5931488da3134d6b5c18aef2 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20240502 08:17:03.640515 24175 tablet_bootstrap.cc:492] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:03.643882 23890 tablet_service.cc:1467] Processing CreateTablet for tablet c278749056864799bf01a5113659da08 (DEFAULT_TABLE table=default.Table_afe99e714f924148a842b013e4daa8e4 [id=e31cb22f2a53450f83040edf85cc34cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.644013 23898 tablet_service.cc:1467] Processing CreateTablet for tablet ec8a9e9783154486b3959fad73453e8b (DEFAULT_TABLE table=default.Table_7e555222aaac43b0ae44935aa7662dbf [id=ab7bf26979a04600924023409d203b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.644194 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c278749056864799bf01a5113659da08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.644377 23893 tablet_service.cc:1467] Processing CreateTablet for tablet cfba7f96cd874166bee24ac20f565b2e (DEFAULT_TABLE table=default.Table_6fc62448e8db451ebfaa3212bef8c660 [id=296ca780c8fd4cda8b7441d14ebdfca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.644629 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfba7f96cd874166bee24ac20f565b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.649662 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8a9e9783154486b3959fad73453e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.650087 23760 tablet_service.cc:1467] Processing CreateTablet for tablet ec8a9e9783154486b3959fad73453e8b (DEFAULT_TABLE table=default.Table_7e555222aaac43b0ae44935aa7662dbf [id=ab7bf26979a04600924023409d203b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.650408 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8a9e9783154486b3959fad73453e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.656366 23762 tablet_service.cc:1467] Processing CreateTablet for tablet c278749056864799bf01a5113659da08 (DEFAULT_TABLE table=default.Table_afe99e714f924148a842b013e4daa8e4 [id=e31cb22f2a53450f83040edf85cc34cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.656750 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c278749056864799bf01a5113659da08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.657635 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 2bee7a73991e425eb41d912704824419 (DEFAULT_TABLE table=default.Table_6758c439d0264980a3c6493fe1607b36 [id=03c9aa715a8e4db9ad86d7afb75dd02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.657943 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bee7a73991e425eb41d912704824419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.665997 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 2bee7a73991e425eb41d912704824419 (DEFAULT_TABLE table=default.Table_6758c439d0264980a3c6493fe1607b36 [id=03c9aa715a8e4db9ad86d7afb75dd02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.666437 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bee7a73991e425eb41d912704824419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.672394 26003 raft_consensus.cc:482] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:03.672559 26003 raft_consensus.cc:504] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.673238 26003 leader_election.cc:288] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:03.674391 24174 ts_tablet_manager.cc:1398] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.069s user 0.002s sys 0.000s
I20240502 08:17:03.674690 24174 tablet_bootstrap.cc:492] T 981b7a7292884b1fb48af99db6e762c0 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:03.677948 23762 tablet_service.cc:1467] Processing CreateTablet for tablet cfba7f96cd874166bee24ac20f565b2e (DEFAULT_TABLE table=default.Table_6fc62448e8db451ebfaa3212bef8c660 [id=296ca780c8fd4cda8b7441d14ebdfca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.678381 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfba7f96cd874166bee24ac20f565b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.680714 23604 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 665bbee2d15b4112b651844431535d0e (table default.Table_64f173e9aa5947069cae45a6cb8dd18f [id=427121bc94c247eaa315b6c830770d61]). Expected version 3 got 0
I20240502 08:17:03.680905 23604 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 07a9d13d3a2d4e6dbc04998c3dd7e1a8 (table default.Table_d9f68c6b26d442bbbd485bb89ee5564c [id=7a49ccfdd6884281ba9b882debd04871]). Expected version 1 got 0
I20240502 08:17:03.682461 25832 leader_election.cc:288] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:17:03.687187 23610 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 981b7a7292884b1fb48af99db6e762c0 (table default.Table_2911b4ae0383471eafe8761c4674beed [id=a681d30fb3e5469aad1bc27dfdd56ef7]). Expected version 3 got 0
I20240502 08:17:03.687670 23610 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet b0cf1a6de3d649b984c50f45393ec0f1 (table default.Table_d416b40fe3d24dafa8db5ee61d13acc0 [id=fa5100e581a7430ab8f0c19b2daae994]). Expected version 1 got 0
I20240502 08:17:03.687999 23610 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 90d4e2f23c9649a4b1dfb6a179ca7be5 (table default.Table_b2e1c08bfc404c7ebb2720631ad10495 [id=9480d59a76fc401e959a1e41caafc845]). Expected version 3 got 0
I20240502 08:17:03.593539 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b4a4d6a364de8af3b54d99a0ae138" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:03.691458 23861 leader_election.cc:341] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.614991 24024 tablet_service.cc:1514] Processing DeleteTablet for tablet ea1ceb374483425194f962b5b5b87438 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:03.641253 24031 tablet_service.cc:1467] Processing CreateTablet for tablet c278749056864799bf01a5113659da08 (DEFAULT_TABLE table=default.Table_afe99e714f924148a842b013e4daa8e4 [id=e31cb22f2a53450f83040edf85cc34cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.654305 24027 tablet_service.cc:1467] Processing CreateTablet for tablet cfba7f96cd874166bee24ac20f565b2e (DEFAULT_TABLE table=default.Table_6fc62448e8db451ebfaa3212bef8c660 [id=296ca780c8fd4cda8b7441d14ebdfca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.692822 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c278749056864799bf01a5113659da08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.693238 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfba7f96cd874166bee24ac20f565b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.655370 24051 raft_consensus.cc:3037] T fba8f6e017e848d59239ae98432b1837 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.694878 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bcfbfe93e2a14297828b5837679bfd8d" } modify_external_catalogs: true
I20240502 08:17:03.695050 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_bcfbfe93e2a14297828b5837679bfd8d" } modify_external_catalogs: true
I20240502 08:17:03.696168 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_68a38c13229040daac749e3816047500" } modify_external_catalogs: true
I20240502 08:17:03.696326 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_68a38c13229040daac749e3816047500" } modify_external_catalogs: true
I20240502 08:17:03.655400 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 2bee7a73991e425eb41d912704824419 (DEFAULT_TABLE table=default.Table_6758c439d0264980a3c6493fe1607b36 [id=03c9aa715a8e4db9ad86d7afb75dd02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.682723 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet 42e11d405a2e4e518a10bf383e9ae6c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:03.682881 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b98cc8728c042f4b7f6496fcc5ef913 with delete_type TABLET_DATA_DELETED (replaced by tablet 25c8a6aaabe2413d8ce88f54ce45a4c4) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:03.654433 24029 tablet_service.cc:1467] Processing CreateTablet for tablet ec8a9e9783154486b3959fad73453e8b (DEFAULT_TABLE table=default.Table_7e555222aaac43b0ae44935aa7662dbf [id=ab7bf26979a04600924023409d203b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.699704 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bee7a73991e425eb41d912704824419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.699826 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet b2eac3bde41c4415bdb97fc53a5b395f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.700064 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8a9e9783154486b3959fad73453e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.702267 23904 tablet_service.cc:1514] Processing DeleteTablet for tablet 76e0f767c6534a608023e6e6d33ca7a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.702457 23903 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d8410ba7cc4b03bb64a138b1f9bb14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.702592 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a74a6eb6d214d858903b3307e09e9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.702764 24181 tablet_replica.cc:330] T 42e11d405a2e4e518a10bf383e9ae6c4 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:03.702801 25649 consensus_queue.cc:1035] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:03.702955 24181 raft_consensus.cc:2229] T 42e11d405a2e4e518a10bf383e9ae6c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:03.703106 24181 raft_consensus.cc:2258] T 42e11d405a2e4e518a10bf383e9ae6c4 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 08:17:03.703568 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet ea1ceb374483425194f962b5b5b87438 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea1ceb374483425194f962b5b5b87438 already in progress: creating tablet
I20240502 08:17:03.703904 24181 ts_tablet_manager.cc:1875] T 42e11d405a2e4e518a10bf383e9ae6c4 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.711565 23601 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 90d4e2f23c9649a4b1dfb6a179ca7be5 (table default.Table_b2e1c08bfc404c7ebb2720631ad10495 [id=9480d59a76fc401e959a1e41caafc845]). Expected version 3 got 0
I20240502 08:17:03.714962 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e56af7b42cb481a8363d60f49ee3f61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.715193 24180 tablet_replica.cc:330] T 5e56af7b42cb481a8363d60f49ee3f61 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:03.715374 24180 raft_consensus.cc:2229] T 5e56af7b42cb481a8363d60f49ee3f61 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:03.715505 24180 raft_consensus.cc:2258] T 5e56af7b42cb481a8363d60f49ee3f61 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.715999 24180 ts_tablet_manager.cc:1875] T 5e56af7b42cb481a8363d60f49ee3f61 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.717739 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a3455e5931488da3134d6b5c18aef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.719292 25649 raft_consensus.cc:482] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:03.719491 25649 raft_consensus.cc:504] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:03.720275 25649 leader_election.cc:288] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:03.720790 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee764398b364d029ae7f0f575a30d64" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:03.721321 23861 leader_election.cc:341] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.699373 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3c00e5393c47df924bcb5dcf2a0f19 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:51 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.723357 24181 ts_tablet_manager.cc:1888] T 42e11d405a2e4e518a10bf383e9ae6c4 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:03.723515 24181 log.cc:1196] T 42e11d405a2e4e518a10bf383e9ae6c4 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/42e11d405a2e4e518a10bf383e9ae6c4
I20240502 08:17:03.723963 24181 ts_tablet_manager.cc:1909] T 42e11d405a2e4e518a10bf383e9ae6c4 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:03.724050 24180 ts_tablet_manager.cc:1888] T 5e56af7b42cb481a8363d60f49ee3f61 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:03.724160 24180 log.cc:1196] T 5e56af7b42cb481a8363d60f49ee3f61 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/5e56af7b42cb481a8363d60f49ee3f61
I20240502 08:17:03.724464 24180 ts_tablet_manager.cc:1909] T 5e56af7b42cb481a8363d60f49ee3f61 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:03.725143 24181 tablet_replica.cc:330] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:03.725390 24181 raft_consensus.cc:2229] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:03.725749 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet e2792c45255c42adaeefb7aef4e25715 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.725950 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet d16110e1b1664802a98182c73f1e41d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.726821 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet bb4b4a4d6a364de8af3b54d99a0ae138 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.727722 24181 pending_rounds.cc:62] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5: Trying to abort 1 pending ops.
I20240502 08:17:03.731141 24180 tablet_replica.cc:330] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:03.731405 24180 raft_consensus.cc:2229] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:03.731377 23610 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet fba8f6e017e848d59239ae98432b1837 (table default.Table_c7de500cdfa341798dbbd1cc9ab5e4fb [id=3deaa06a1e88491f9dda6abda2cbdd9f]). Expected version 1 got 0
I20240502 08:17:03.731592 23610 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 6088f160a13c4ff09f72bed72f69eaf8 (table default.Table_851f246623304504aa9ddf2e6c5e9082 [id=4fecac7b0d9042b3b4f674f3fc1338ba]). Expected version 1 got 0
I20240502 08:17:03.731724 24180 pending_rounds.cc:62] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:17:03.731788 24180 pending_rounds.cc:68] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156523416592384 op_type: NO_OP noop_request { }
I20240502 08:17:03.731978 24180 raft_consensus.cc:2866] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:03.732064 24180 raft_consensus.cc:2258] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.733109 24180 ts_tablet_manager.cc:1875] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.734115 24173 raft_consensus.cc:374] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.734278 24173 raft_consensus.cc:729] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.734661 24173 consensus_queue.cc:260] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:03.735249 24173 ts_tablet_manager.cc:1398] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.149s user 0.001s sys 0.001s
I20240502 08:17:03.735627 24173 tablet_bootstrap.cc:492] T fddac683eda14ec1a0140926380f2c07 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:03.737761 24173 tablet_bootstrap.cc:654] T fddac683eda14ec1a0140926380f2c07 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.740295 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f83bcf640f465bb95d87a991cc5ff6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9cb9cb0164164b43bf08c2a8532b766b) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.740770 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ee52bcc61b4816acda5b8140b63504 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.741151 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet fddac683eda14ec1a0140926380f2c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.741465 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet e9171ebf74424c9aae5e1c8a5118f768 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.741890 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3c00e5393c47df924bcb5dcf2a0f19 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:51 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.742278 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 76e0f767c6534a608023e6e6d33ca7a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.742616 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a74a6eb6d214d858903b3307e09e9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.742929 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet e2792c45255c42adaeefb7aef4e25715 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:58 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.743203 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a3455e5931488da3134d6b5c18aef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.743487 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 0281c8123eab4c48bddc377f5695c6e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.743762 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 5e56af7b42cb481a8363d60f49ee3f61 (table default.Table_94efd28ada5b4024ab65e4d5feda0b38 [id=15b42ca7619645c8ac3cff50044f069b]) successfully deleted
W20240502 08:17:03.751320 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d607bb20596946e495cf749fb7e2552d consensus state has no leader
I20240502 08:17:03.752007 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet f13bd33dd7c44ac19fa4e0bba3083483
I20240502 08:17:03.754130 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet f13bd33dd7c44ac19fa4e0bba3083483 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.755970 23602 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet fba8f6e017e848d59239ae98432b1837 (table default.Table_c7de500cdfa341798dbbd1cc9ab5e4fb [id=3deaa06a1e88491f9dda6abda2cbdd9f]). Expected version 1 got 0
I20240502 08:17:03.758141 25902 raft_consensus.cc:482] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:03.758280 25902 raft_consensus.cc:504] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.759274 25902 leader_election.cc:288] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:03.760506 26008 tablet.cc:1722] T ac50774b2dd241de9cae8096578d183e P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:17:03.765967 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_15a09f13f7024d2cbe6d5c0fec986fd6" } new_table_name: "default.Table_39508ff6f1ec454dbbb39a729353140f" modify_external_catalogs: true
W20240502 08:17:03.766889 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 129188us
I20240502 08:17:03.767571 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3c00e5393c47df924bcb5dcf2a0f19" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:17:03.767808 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eaea443ae9044b76b48b3b5f92a9ae7e consensus state has no leader
I20240502 08:17:03.767813 23922 raft_consensus.cc:2454] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:03.771317 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a3455e5931488da3134d6b5c18aef2" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:03.771551 23922 raft_consensus.cc:2454] T 91a3455e5931488da3134d6b5c18aef2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:03.772114 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e88b8a2ebdba4f77922330a6da96ec51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.772732 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e88b8a2ebdba4f77922330a6da96ec51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.772795 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee764398b364d029ae7f0f575a30d64" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:03.773031 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a52ae049081249eeb49eeb8d8620da1e" } modify_external_catalogs: true
I20240502 08:17:03.776203 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_a52ae049081249eeb49eeb8d8620da1e" } modify_external_catalogs: true
I20240502 08:17:03.776970 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b4a4d6a364de8af3b54d99a0ae138" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:03.777140 23786 raft_consensus.cc:2454] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:03.777931 23602 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 90d4e2f23c9649a4b1dfb6a179ca7be5 (table default.Table_b2e1c08bfc404c7ebb2720631ad10495 [id=9480d59a76fc401e959a1e41caafc845]). Expected version 3 got 0
I20240502 08:17:03.779099 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9a2fa4232ad64879b69ae596cdfd2bed" } modify_external_catalogs: true
I20240502 08:17:03.779311 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9a2fa4232ad64879b69ae596cdfd2bed" } modify_external_catalogs: true
I20240502 08:17:03.780851 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6689e6f205644f8091743ad64544240a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:03.784890 23602 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet fba8f6e017e848d59239ae98432b1837 (table default.Table_c7de500cdfa341798dbbd1cc9ab5e4fb [id=3deaa06a1e88491f9dda6abda2cbdd9f]). Expected version 1 got 0
I20240502 08:17:03.786247 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet f13bd33dd7c44ac19fa4e0bba3083483 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.792086 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 66ea6743a3a44f9b95e1ca3d50a3dd40
I20240502 08:17:03.792940 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ea6743a3a44f9b95e1ca3d50a3dd40 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.772742 23723 leader_election.cc:302] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
W20240502 08:17:03.801806 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 128c37802f0347518003a0fdea3798bb has no consensus state
I20240502 08:17:03.801949 25903 raft_consensus.cc:2781] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:03.802042 25903 raft_consensus.cc:482] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:03.802354 25903 raft_consensus.cc:3037] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.805068 25903 raft_consensus.cc:504] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:17:03.806066 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6689e6f205644f8091743ad64544240a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.809051 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ea6743a3a44f9b95e1ca3d50a3dd40 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.809348 25903 leader_election.cc:288] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:03.809468 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d8410ba7cc4b03bb64a138b1f9bb14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.809671 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet b2eac3bde41c4415bdb97fc53a5b395f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.810056 23723 leader_election.cc:302] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:17:03.810259 26000 raft_consensus.cc:2781] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:03.810328 26000 raft_consensus.cc:482] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:03.810384 26000 raft_consensus.cc:3037] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:17:03.810420 23858 leader_election.cc:341] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.810545 23858 leader_election.cc:302] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:03.811288 23858 leader_election.cc:302] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:03.811347 25494 raft_consensus.cc:2726] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:03.811543 25494 consensus_queue.cc:1035] T fba8f6e017e848d59239ae98432b1837 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:03.811923 25494 raft_consensus.cc:2781] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:03.727782 24181 pending_rounds.cc:68] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156519607046144 op_type: NO_OP noop_request { }
I20240502 08:17:03.811975 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3c00e5393c47df924bcb5dcf2a0f19" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:03.812122 23920 raft_consensus.cc:3037] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.812116 24181 raft_consensus.cc:2866] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:03.812253 24181 raft_consensus.cc:2258] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.764454 26009 tablet.cc:1722] T ac50774b2dd241de9cae8096578d183e P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:17:03.816268 26000 raft_consensus.cc:504] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.816838 26000 leader_election.cc:288] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:03.816941 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae9c817c7be42da8c689267c7eb5a06 with delete_type TABLET_DATA_DELETED (replaced by tablet b3b0c0dcb2184ad39b984f718f8a2082) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.817809 24181 ts_tablet_manager.cc:1875] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.819794 23760 tablet_service.cc:1467] Processing CreateTablet for tablet e82b97c0c1564ddf9946831fc3096b85 (DEFAULT_TABLE table=default.Table_df7e0a371a964acc812387e7d12dd45a [id=964597ff9f084977944f9a45a0aa73d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.820240 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82b97c0c1564ddf9946831fc3096b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.822448 23608 master_service.cc:946] ReplaceTablet: received request to replace tablet 4e3570787d8242338d1ce01fb0299eb2 from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:03.822888 24175 tablet_bootstrap.cc:700] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent opening tablet: real 0.181s user 0.000s sys 0.000s
I20240502 08:17:03.823320 25867 webserver.cc:460] Webserver started at http://127.22.225.126:34035/ using document root <none> and password file <none>
I20240502 08:17:03.822140 24174 tablet_bootstrap.cc:700] T 981b7a7292884b1fb48af99db6e762c0 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent opening tablet: real 0.146s user 0.001s sys 0.000s
I20240502 08:17:03.824330 24174 tablet_bootstrap.cc:654] T 981b7a7292884b1fb48af99db6e762c0 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.824468 25867 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:03.824625 25867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:03.824736 24181 ts_tablet_manager.cc:1888] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:03.824821 24181 log.cc:1196] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/2b98cc8728c042f4b7f6496fcc5ef913
I20240502 08:17:03.825284 24181 ts_tablet_manager.cc:1909] T 2b98cc8728c042f4b7f6496fcc5ef913 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:03.823190 23920 raft_consensus.cc:2454] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:17:03.811985 25494 raft_consensus.cc:482] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:03.826833 25494 raft_consensus.cc:3037] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.829095 25012 ts_tablet_manager.cc:1888] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20240502 08:17:03.829221 25012 log.cc:1196] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/b9f83bcf640f465bb95d87a991cc5ff6
I20240502 08:17:03.829576 25494 raft_consensus.cc:504] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:17:03.851286 23595 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet d607bb20596946e495cf749fb7e2552d (table default.Table_94fc9fc0e27b43c8807dd10d6c3f94a6 [id=0f561a70639e4e27a7d192270e8d608f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d607bb20596946e495cf749fb7e2552d consensus state has no leader
I20240502 08:17:03.825438 26017 tablet.cc:1722] T ac50774b2dd241de9cae8096578d183e P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240502 08:17:03.830708 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet a91e2b50321e4241847d97a558b419f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:03.854135 23723 leader_election.cc:302] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:17:03.856864 25903 raft_consensus.cc:2781] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:03.857005 25903 raft_consensus.cc:686] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:17:03.857246 25903 consensus_queue.cc:237] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.870762 25903 raft_consensus.cc:482] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:03.870896 25903 raft_consensus.cc:504] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.871431 25903 leader_election.cc:288] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:03.872166 25867 fs_manager.cc:709] Time spent opening directory manager: real 0.029s user 0.002s sys 0.005s
I20240502 08:17:03.874980 24055 raft_consensus.cc:3037] T 665bbee2d15b4112b651844431535d0e P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.829564 25012 ts_tablet_manager.cc:1909] T b9f83bcf640f465bb95d87a991cc5ff6 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:03.880990 25012 tablet_replica.cc:330] T d16110e1b1664802a98182c73f1e41d3 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:03.881227 25012 raft_consensus.cc:2229] T d16110e1b1664802a98182c73f1e41d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:03.881393 25012 raft_consensus.cc:2258] T d16110e1b1664802a98182c73f1e41d3 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.882011 25012 ts_tablet_manager.cc:1875] T d16110e1b1664802a98182c73f1e41d3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.885764 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet b9f83bcf640f465bb95d87a991cc5ff6 (table default.Table_e00515baeaae4ce9ad93000143f2a53a [id=13293a9ab7004e8a965e004f2fb5a14a]) successfully deleted
W20240502 08:17:03.886267 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 30de4d9cb40e4037b99b71bff5667e8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30de4d9cb40e4037b99b71bff5667e8c already in progress: creating tablet
I20240502 08:17:03.886466 26028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20240502 08:17:03.886649 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 397003e47c78418182e0167c3b876d4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 397003e47c78418182e0167c3b876d4e already in progress: creating tablet
W20240502 08:17:03.887012 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 6e35efdc0434429aa093fec7770782ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e35efdc0434429aa093fec7770782ef already in progress: creating tablet
I20240502 08:17:03.888617 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e56af7b42cb481a8363d60f49ee3f61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:03.889848 25867 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.003s
W20240502 08:17:03.890179 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 5604212dd32c430ab35f8c50fb9d7023 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5604212dd32c430ab35f8c50fb9d7023 already in progress: creating tablet
W20240502 08:17:03.890662 23597 catalog_manager.cc:4810] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet ea1ceb374483425194f962b5b5b87438 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea1ceb374483425194f962b5b5b87438 already in progress: creating tablet
I20240502 08:17:03.890045 25867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data,/tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/wal
uuid: "bcc8a323e6b9429eaaf4e0c1698ad077"
format_stamp: "Formatted at 2024-05-02 08:16:30 on dist-test-slave-m44h"
I20240502 08:17:03.891170 24173 tablet_bootstrap.cc:492] T fddac683eda14ec1a0140926380f2c07 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:03.891300 24173 ts_tablet_manager.cc:1367] T fddac683eda14ec1a0140926380f2c07 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.156s user 0.009s sys 0.001s
I20240502 08:17:03.891283 25867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:03.891988 24173 raft_consensus.cc:348] T fddac683eda14ec1a0140926380f2c07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:03.892167 24173 raft_consensus.cc:374] T fddac683eda14ec1a0140926380f2c07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.892220 24173 raft_consensus.cc:729] T fddac683eda14ec1a0140926380f2c07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.892400 24173 consensus_queue.cc:260] T fddac683eda14ec1a0140926380f2c07 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:03.846191 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a3455e5931488da3134d6b5c18aef2" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:03.846278 23893 tablet_service.cc:1467] Processing CreateTablet for tablet e82b97c0c1564ddf9946831fc3096b85 (DEFAULT_TABLE table=default.Table_df7e0a371a964acc812387e7d12dd45a [id=964597ff9f084977944f9a45a0aa73d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.893349 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82b97c0c1564ddf9946831fc3096b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.857266 23890 tablet_service.cc:1467] Processing CreateTablet for tablet c02e1f0df76744c3b4f47319f35920b0 (DEFAULT_TABLE table=default.Table_7410c0888be547e69500e037718e3861 [id=2c30d6e8dc794603aa979411accff78d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.897835 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02e1f0df76744c3b4f47319f35920b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.901412 26027 raft_consensus.cc:482] T 6088f160a13c4ff09f72bed72f69eaf8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:03.901460 26009 tablet.cc:1722] T c7aa03e73acd4a9da5574b9ba93415ae P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:03.901531 26027 raft_consensus.cc:504] T 6088f160a13c4ff09f72bed72f69eaf8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.902375 25649 raft_consensus.cc:482] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:03.902513 25649 raft_consensus.cc:504] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.903000 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088f160a13c4ff09f72bed72f69eaf8" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:03.903124 25649 leader_election.cc:288] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:17:03.904063 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e0f767c6534a608023e6e6d33ca7a8" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:03.904354 23920 raft_consensus.cc:3037] T 91a3455e5931488da3134d6b5c18aef2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:17:03.904593 23861 leader_election.cc:341] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.906836 23920 raft_consensus.cc:2454] T 91a3455e5931488da3134d6b5c18aef2 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:17:03.877851 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae9c817c7be42da8c689267c7eb5a06" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:03.878165 26018 consensus_queue.cc:1035] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:03.830650 24175 tablet_bootstrap.cc:654] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.912704 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b4a4d6a364de8af3b54d99a0ae138" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:17:03.913251 23861 leader_election.cc:341] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:03.915024 23767 tablet_service.cc:1467] Processing CreateTablet for tablet c02e1f0df76744c3b4f47319f35920b0 (DEFAULT_TABLE table=default.Table_7410c0888be547e69500e037718e3861 [id=2c30d6e8dc794603aa979411accff78d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.915496 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02e1f0df76744c3b4f47319f35920b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.922065 26027 leader_election.cc:288] T 6088f160a13c4ff09f72bed72f69eaf8 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:03.924839 26008 tablet.cc:1722] T c7aa03e73acd4a9da5574b9ba93415ae P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:03.925783 26003 raft_consensus.cc:482] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:17:03.926039 23861 consensus_peers.cc:485] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:03.926374 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6088f160a13c4ff09f72bed72f69eaf8" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:03.926496 25903 consensus_queue.cc:1035] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:03.926649 23922 raft_consensus.cc:2379] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 149ce744feab49b5aa5bcaa14bd6aba5 in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:17:03.927057 25494 leader_election.cc:288] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:17:03.929085 23996 leader_election.cc:302] T 6088f160a13c4ff09f72bed72f69eaf8 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:03.929260 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 42e11d405a2e4e518a10bf383e9ae6c4 (table default.Table_d790560c82534057ab7061352d8c71a2 [id=15cd2d29f86a49e295158ba17e364a23]) successfully deleted
I20240502 08:17:03.929735 26026 raft_consensus.cc:3037] T 6088f160a13c4ff09f72bed72f69eaf8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.930720 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 1e57882a0a0a4ec5819a4108635062f0 (DEFAULT_TABLE table=default.Table_7c9232b45ef64cb58387478ac1b14b45 [id=92bda79c30d940ddb02950c6ade6837f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.930816 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 2b98cc8728c042f4b7f6496fcc5ef913 (table default.Table_5fe628d3da624233941f440e3743da88 [id=ca161ce6780b446fa5743d3974cfa27b]) successfully deleted
I20240502 08:17:03.931183 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e57882a0a0a4ec5819a4108635062f0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:03.931365 24108 tablet.cc:2369] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 149ce744feab49b5aa5bcaa14bd6aba5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:03.933545 26026 raft_consensus.cc:2726] T 6088f160a13c4ff09f72bed72f69eaf8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:03.939532 26003 raft_consensus.cc:504] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.940311 26003 leader_election.cc:288] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:03.941530 24174 tablet_bootstrap.cc:492] T 981b7a7292884b1fb48af99db6e762c0 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:03.941731 24174 ts_tablet_manager.cc:1367] T 981b7a7292884b1fb48af99db6e762c0 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.267s user 0.002s sys 0.001s
I20240502 08:17:03.942412 23602 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet fba8f6e017e848d59239ae98432b1837 (table default.Table_c7de500cdfa341798dbbd1cc9ab5e4fb [id=3deaa06a1e88491f9dda6abda2cbdd9f]). Expected version 1 got 0
I20240502 08:17:03.942534 24174 raft_consensus.cc:348] T 981b7a7292884b1fb48af99db6e762c0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.942700 24174 raft_consensus.cc:374] T 981b7a7292884b1fb48af99db6e762c0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.942698 23602 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet b0cf1a6de3d649b984c50f45393ec0f1 (table default.Table_d416b40fe3d24dafa8db5ee61d13acc0 [id=fa5100e581a7430ab8f0c19b2daae994]). Expected version 1 got 0
I20240502 08:17:03.942745 24174 raft_consensus.cc:729] T 981b7a7292884b1fb48af99db6e762c0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.943426 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_39b97fc571c04e5eb6c403c080d75a87" } modify_external_catalogs: true
I20240502 08:17:03.943579 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_39b97fc571c04e5eb6c403c080d75a87" } modify_external_catalogs: true
I20240502 08:17:03.944788 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_77228df00cdf4938acabcf03c8b4525e" } new_table_name: "default.Table_a524374b2bc042d189accf5d6ecdd701" modify_external_catalogs: true
I20240502 08:17:03.945273 24174 consensus_queue.cc:260] T 981b7a7292884b1fb48af99db6e762c0 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.945933 25903 raft_consensus.cc:482] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:03.946053 25903 raft_consensus.cc:504] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.946099 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7a24f13034944055bb412fa1b79c70c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:03.946586 25903 leader_election.cc:288] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:03.947150 24174 ts_tablet_manager.cc:1398] T 981b7a7292884b1fb48af99db6e762c0 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:17:03.947415 24174 tablet_bootstrap.cc:492] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
W20240502 08:17:03.948202 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a24f13034944055bb412fa1b79c70c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.948199 23603 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 90d4e2f23c9649a4b1dfb6a179ca7be5 (table default.Table_b2e1c08bfc404c7ebb2720631ad10495 [id=9480d59a76fc401e959a1e41caafc845]). Expected version 3 got 0
I20240502 08:17:03.952493 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_40d7974e43e447dea4a527c9468becd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:03.953133 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40d7974e43e447dea4a527c9468becd9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:03.953603 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3c00e5393c47df924bcb5dcf2a0f19" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:03.954784 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a3455e5931488da3134d6b5c18aef2" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:03.955550 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3c00e5393c47df924bcb5dcf2a0f19" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:03.955935 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a3455e5931488da3134d6b5c18aef2" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:03.956997 24174 tablet_bootstrap.cc:654] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.958743 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae9c817c7be42da8c689267c7eb5a06" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:03.962806 24174 tablet_bootstrap.cc:492] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:03.962941 24174 ts_tablet_manager.cc:1367] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20240502 08:17:03.963217 24051 raft_consensus.cc:3037] T 07a9d13d3a2d4e6dbc04998c3dd7e1a8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:03.963615 24174 raft_consensus.cc:348] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:03.963786 24174 raft_consensus.cc:374] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.963836 24174 raft_consensus.cc:729] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.967980 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b175e1b55bf4102804666b6fffccead with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:59 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:03.968302 24181 tablet_replica.cc:330] T 1b175e1b55bf4102804666b6fffccead P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:03.968538 24181 raft_consensus.cc:2229] T 1b175e1b55bf4102804666b6fffccead P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:03.968690 24181 raft_consensus.cc:2258] T 1b175e1b55bf4102804666b6fffccead P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:03.964015 24174 consensus_queue.cc:260] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:03.969242 24181 ts_tablet_manager.cc:1875] T 1b175e1b55bf4102804666b6fffccead P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:03.970078 24174 ts_tablet_manager.cc:1398] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240502 08:17:03.970302 24174 tablet_bootstrap.cc:492] T 045b0882a40f4eaba94935d9bc90da0c P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:03.971279 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e0f767c6534a608023e6e6d33ca7a8" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:03.971453 23784 raft_consensus.cc:2454] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:03.971967 23723 leader_election.cc:302] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:17:03.972179 25903 raft_consensus.cc:2781] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:03.972316 25903 raft_consensus.cc:686] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:17:03.972579 25903 consensus_queue.cc:237] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.977766 24181 ts_tablet_manager.cc:1888] T 1b175e1b55bf4102804666b6fffccead P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:03.977924 24181 log.cc:1196] T 1b175e1b55bf4102804666b6fffccead P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/1b175e1b55bf4102804666b6fffccead
I20240502 08:17:03.977952 24174 tablet_bootstrap.cc:654] T 045b0882a40f4eaba94935d9bc90da0c P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.978282 24181 ts_tablet_manager.cc:1909] T 1b175e1b55bf4102804666b6fffccead P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:03.979532 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet f13bd33dd7c44ac19fa4e0bba3083483 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:03.983067 25832 consensus_queue.cc:1035] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:03.983522 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ea6743a3a44f9b95e1ca3d50a3dd40 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:03.984269 24033 tablet_service.cc:1467] Processing CreateTablet for tablet e82b97c0c1564ddf9946831fc3096b85 (DEFAULT_TABLE table=default.Table_df7e0a371a964acc812387e7d12dd45a [id=964597ff9f084977944f9a45a0aa73d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.984588 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82b97c0c1564ddf9946831fc3096b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.984953 24175 tablet_bootstrap.cc:492] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:03.985085 24175 ts_tablet_manager.cc:1367] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.345s user 0.003s sys 0.000s
I20240502 08:17:03.985894 24173 ts_tablet_manager.cc:1398] T fddac683eda14ec1a0140926380f2c07 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.095s user 0.009s sys 0.005s
I20240502 08:17:03.986143 24173 tablet_bootstrap.cc:492] T 0f306a288b90413b9d8ea8b76d23363a P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:03.986402 25832 consensus_queue.cc:1035] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:03.987603 24173 tablet_bootstrap.cc:654] T 0f306a288b90413b9d8ea8b76d23363a P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.988238 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 1e57882a0a0a4ec5819a4108635062f0 (DEFAULT_TABLE table=default.Table_7c9232b45ef64cb58387478ac1b14b45 [id=92bda79c30d940ddb02950c6ade6837f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:03.992386 24173 tablet_bootstrap.cc:492] T 0f306a288b90413b9d8ea8b76d23363a P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:03.992506 24173 ts_tablet_manager.cc:1367] T 0f306a288b90413b9d8ea8b76d23363a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240502 08:17:03.993146 24173 raft_consensus.cc:348] T 0f306a288b90413b9d8ea8b76d23363a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.993371 24173 raft_consensus.cc:374] T 0f306a288b90413b9d8ea8b76d23363a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:03.993417 24173 raft_consensus.cc:729] T 0f306a288b90413b9d8ea8b76d23363a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.993647 24173 consensus_queue.cc:260] T 0f306a288b90413b9d8ea8b76d23363a P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:03.994138 24173 ts_tablet_manager.cc:1398] T 0f306a288b90413b9d8ea8b76d23363a P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:03.994431 24173 tablet_bootstrap.cc:492] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:03.994470 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 3449d7d1e8044b6cbf20766a5597e5ab
I20240502 08:17:03.995819 24173 tablet_bootstrap.cc:654] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:03.998726 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet 3449d7d1e8044b6cbf20766a5597e5ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.002135 23601 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet fba8f6e017e848d59239ae98432b1837 (table default.Table_c7de500cdfa341798dbbd1cc9ab5e4fb [id=3deaa06a1e88491f9dda6abda2cbdd9f]). Expected version 1 got 0
I20240502 08:17:04.002524 23601 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 981b7a7292884b1fb48af99db6e762c0 (table default.Table_2911b4ae0383471eafe8761c4674beed [id=a681d30fb3e5469aad1bc27dfdd56ef7]). Expected version 3 got 0
I20240502 08:17:04.003609 24173 tablet_bootstrap.cc:492] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:04.003749 24173 ts_tablet_manager.cc:1367] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 08:17:04.003775 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e57882a0a0a4ec5819a4108635062f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.004387 24173 raft_consensus.cc:348] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.004561 24173 raft_consensus.cc:374] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.004606 24173 raft_consensus.cc:729] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.004805 24173 consensus_queue.cc:260] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.005244 24173 ts_tablet_manager.cc:1398] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 08:17:04.005481 24173 tablet_bootstrap.cc:492] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:04.006969 24173 tablet_bootstrap.cc:654] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.014165 24033 tablet_service.cc:1467] Processing CreateTablet for tablet c02e1f0df76744c3b4f47319f35920b0 (DEFAULT_TABLE table=default.Table_7410c0888be547e69500e037718e3861 [id=2c30d6e8dc794603aa979411accff78d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.014601 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02e1f0df76744c3b4f47319f35920b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.021565 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d8410ba7cc4b03bb64a138b1f9bb14" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:04.021811 23916 raft_consensus.cc:2454] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:04.024310 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 1e57882a0a0a4ec5819a4108635062f0 (DEFAULT_TABLE table=default.Table_7c9232b45ef64cb58387478ac1b14b45 [id=92bda79c30d940ddb02950c6ade6837f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.024641 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e57882a0a0a4ec5819a4108635062f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.028801 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 0e9de33c88984befb056647b4c31de6f
I20240502 08:17:04.030447 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 3449d7d1e8044b6cbf20766a5597e5ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.031226 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 816efe6a50374a3cb0103a28909e0ce4 (DEFAULT_TABLE table=default.Table_8da903edc43e49b98cb1e4f287d45633 [id=93dd365eda6742158209de9cd84cf2b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.031558 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816efe6a50374a3cb0103a28909e0ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.031498 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_a783dfa350f04101b7ead760676cb153"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.032300 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a783dfa350f04101b7ead760676cb153 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.036051 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a91e2b50321e4241847d97a558b419f2" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:04.037760 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 1b175e1b55bf4102804666b6fffccead (table default.Table_8c96a3fc08e94e36af627e4c8dddc7ae [id=5a9755c4cebd4ad2ab00ce225b25334c]) successfully deleted
I20240502 08:17:04.040107 24175 raft_consensus.cc:348] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.040340 24175 raft_consensus.cc:374] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.040400 24175 raft_consensus.cc:729] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
W20240502 08:17:04.041246 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet f13bd33dd7c44ac19fa4e0bba3083483 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f13bd33dd7c44ac19fa4e0bba3083483 already in progress: creating tablet
W20240502 08:17:04.041767 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 66ea6743a3a44f9b95e1ca3d50a3dd40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66ea6743a3a44f9b95e1ca3d50a3dd40 already in progress: creating tablet
I20240502 08:17:04.043462 24180 ts_tablet_manager.cc:1888] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240502 08:17:04.044289 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 3449d7d1e8044b6cbf20766a5597e5ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3449d7d1e8044b6cbf20766a5597e5ab already in progress: creating tablet
I20240502 08:17:04.045943 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d8410ba7cc4b03bb64a138b1f9bb14" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:04.047700 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 3449d7d1e8044b6cbf20766a5597e5ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:04.048354 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 816efe6a50374a3cb0103a28909e0ce4 (DEFAULT_TABLE table=default.Table_8da903edc43e49b98cb1e4f287d45633 [id=93dd365eda6742158209de9cd84cf2b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.048727 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816efe6a50374a3cb0103a28909e0ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.055366 24175 consensus_queue.cc:260] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.057219 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a91e2b50321e4241847d97a558b419f2" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:04.057288 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 5805e9fcf8cd42abae2e52e262627e20 (DEFAULT_TABLE table=default.Table_517b4d2b22924609b825632eaf88a0bf [id=d4a4f61a07a046108a5961702b8369d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.057742 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5805e9fcf8cd42abae2e52e262627e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.070881 23758 tablet_service.cc:1467] Processing CreateTablet for tablet aec17736da3b43d0b79091adb79365a1 (DEFAULT_TABLE table=default.Table_42afc33a0fee436b9658e8a3e51eedfd [id=bb5a81f9103e4f1eb2c2e9dfdab1e761]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.073810 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 5805e9fcf8cd42abae2e52e262627e20 (DEFAULT_TABLE table=default.Table_517b4d2b22924609b825632eaf88a0bf [id=d4a4f61a07a046108a5961702b8369d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.074226 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5805e9fcf8cd42abae2e52e262627e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.075109 24029 tablet_service.cc:1467] Processing CreateTablet for tablet aec17736da3b43d0b79091adb79365a1 (DEFAULT_TABLE table=default.Table_42afc33a0fee436b9658e8a3e51eedfd [id=bb5a81f9103e4f1eb2c2e9dfdab1e761]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.075510 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec17736da3b43d0b79091adb79365a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.077975 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6881b68c76417bba71c8426abc4768 (DEFAULT_TABLE table=default.Table_9dff2385175e41de9631cd27fbd848cb [id=aae90f81081b4992beffcaa2fb27289b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.078291 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a6881b68c76417bba71c8426abc4768. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.081801 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 524b604a536a48c18221560baf81b7d6 (DEFAULT_TABLE table=default.Table_ab7e9343fdc74b38bb98f2ead6c8eeb3 [id=2ff010494a374af78fe16a59cd35af5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.082142 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 524b604a536a48c18221560baf81b7d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.082434 24031 tablet_service.cc:1467] Processing CreateTablet for tablet c0976fc5fd29413e9b27d05ad6ab4965 (DEFAULT_TABLE table=default.Table_19eec114db694534a82e5f2f867c02b2 [id=2544ec09ec614537856652009e070d56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.082509 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_986dd87cffc9409da955a904714d5e56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:04.082706 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0976fc5fd29413e9b27d05ad6ab4965. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:04.083011 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_986dd87cffc9409da955a904714d5e56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.090201 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e9de33c88984befb056647b4c31de6f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:17:04.090874 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 0e9de33c88984befb056647b4c31de6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e9de33c88984befb056647b4c31de6f already in progress: creating tablet
I20240502 08:17:04.092339 24173 tablet_bootstrap.cc:492] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:04.092471 24173 ts_tablet_manager.cc:1367] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.087s user 0.015s sys 0.023s
I20240502 08:17:04.093257 24173 raft_consensus.cc:348] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.093516 24173 raft_consensus.cc:374] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.093617 24173 raft_consensus.cc:729] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.093816 24173 consensus_queue.cc:260] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.095003 24173 ts_tablet_manager.cc:1398] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:04.095499 24173 tablet_bootstrap.cc:492] T b0cf1a6de3d649b984c50f45393ec0f1 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:04.097010 24173 tablet_bootstrap.cc:654] T b0cf1a6de3d649b984c50f45393ec0f1 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.105592 24175 ts_tablet_manager.cc:1398] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.120s user 0.003s sys 0.000s
I20240502 08:17:04.106138 24175 tablet_bootstrap.cc:492] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:04.111644 24173 tablet_bootstrap.cc:492] T b0cf1a6de3d649b984c50f45393ec0f1 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:04.111825 24173 ts_tablet_manager.cc:1367] T b0cf1a6de3d649b984c50f45393ec0f1 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.005s
I20240502 08:17:04.112541 24173 raft_consensus.cc:348] T b0cf1a6de3d649b984c50f45393ec0f1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.112753 24173 raft_consensus.cc:374] T b0cf1a6de3d649b984c50f45393ec0f1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.112699 23898 tablet_service.cc:1467] Processing CreateTablet for tablet aec17736da3b43d0b79091adb79365a1 (DEFAULT_TABLE table=default.Table_42afc33a0fee436b9658e8a3e51eedfd [id=bb5a81f9103e4f1eb2c2e9dfdab1e761]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.112866 24173 raft_consensus.cc:729] T b0cf1a6de3d649b984c50f45393ec0f1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.113070 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec17736da3b43d0b79091adb79365a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.113086 24173 consensus_queue.cc:260] T b0cf1a6de3d649b984c50f45393ec0f1 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.113616 24173 ts_tablet_manager.cc:1398] T b0cf1a6de3d649b984c50f45393ec0f1 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:04.114120 23890 tablet_service.cc:1467] Processing CreateTablet for tablet c0976fc5fd29413e9b27d05ad6ab4965 (DEFAULT_TABLE table=default.Table_19eec114db694534a82e5f2f867c02b2 [id=2544ec09ec614537856652009e070d56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.114542 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0976fc5fd29413e9b27d05ad6ab4965. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.117827 24173 tablet_bootstrap.cc:492] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:04.119632 24173 tablet_bootstrap.cc:654] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.087276 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e9de33c88984befb056647b4c31de6f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:04.124258 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 524b604a536a48c18221560baf81b7d6 (DEFAULT_TABLE table=default.Table_ab7e9343fdc74b38bb98f2ead6c8eeb3 [id=2ff010494a374af78fe16a59cd35af5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.124858 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 524b604a536a48c18221560baf81b7d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.127022 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6881b68c76417bba71c8426abc4768 (DEFAULT_TABLE table=default.Table_9dff2385175e41de9631cd27fbd848cb [id=aae90f81081b4992beffcaa2fb27289b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.127470 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a6881b68c76417bba71c8426abc4768. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.132239 26029 raft_consensus.cc:482] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:04.132375 26029 raft_consensus.cc:504] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.133018 26029 leader_election.cc:288] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:17:04.134994 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 661f7748cf74413cbed6f6bb0d46fef4 consensus state has no leader
I20240502 08:17:04.136976 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d4e2f23c9649a4b1dfb6a179ca7be5" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:04.139483 25832 raft_consensus.cc:482] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:04.139609 25832 raft_consensus.cc:504] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.140357 25832 leader_election.cc:288] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:17:04.143872 25494 raft_consensus.cc:482] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:04.144021 25494 raft_consensus.cc:504] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.144582 25494 leader_election.cc:288] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:17:04.145222 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2792c45255c42adaeefb7aef4e25715" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:04.149024 23861 leader_election.cc:341] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.152973 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ce67caa9ca8a4bd092ecf98b26fd3afb" } new_table_name: "default.Table_e0ad47aad16e4f37a8900e8d86098fde" modify_external_catalogs: true
I20240502 08:17:04.155184 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a74a6eb6d214d858903b3307e09e9e6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:04.155694 23861 leader_election.cc:341] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.156819 23761 tablet_service.cc:1467] Processing CreateTablet for tablet c0976fc5fd29413e9b27d05ad6ab4965 (DEFAULT_TABLE table=default.Table_19eec114db694534a82e5f2f867c02b2 [id=2544ec09ec614537856652009e070d56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.156899 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e9de33c88984befb056647b4c31de6f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.156945 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 816efe6a50374a3cb0103a28909e0ce4 (DEFAULT_TABLE table=default.Table_8da903edc43e49b98cb1e4f287d45633 [id=93dd365eda6742158209de9cd84cf2b2]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:17:04.157070 23723 leader_election.cc:341] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.157287 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 5805e9fcf8cd42abae2e52e262627e20 (DEFAULT_TABLE table=default.Table_517b4d2b22924609b825632eaf88a0bf [id=d4a4f61a07a046108a5961702b8369d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.157996 24180 log.cc:1196] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/d16110e1b1664802a98182c73f1e41d3
I20240502 08:17:04.158016 23602 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_c36327219de24751921b49cc4beed5fc [id=30defee3de2445f7bc5017fb23ab2ee7] alter complete (version 2)
I20240502 08:17:04.158102 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec17736da3b43d0b79091adb79365a1. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:04.158167 23836 tablet.cc:2369] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:04.162631 26017 tablet.cc:1722] T c7aa03e73acd4a9da5574b9ba93415ae P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:04.162742 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6881b68c76417bba71c8426abc4768 (DEFAULT_TABLE table=default.Table_9dff2385175e41de9631cd27fbd848cb [id=aae90f81081b4992beffcaa2fb27289b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.158026 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d4e2f23c9649a4b1dfb6a179ca7be5" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:04.163641 23993 leader_election.cc:302] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:17:04.163849 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 151927us
I20240502 08:17:04.164162 26029 raft_consensus.cc:2726] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:04.164780 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5805e9fcf8cd42abae2e52e262627e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.166210 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816efe6a50374a3cb0103a28909e0ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.169229 23723 leader_election.cc:302] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:17:04.169663 24180 ts_tablet_manager.cc:1909] T d16110e1b1664802a98182c73f1e41d3 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:04.170790 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet d16110e1b1664802a98182c73f1e41d3 (table default.Table_9feed0e1d82c472aaf0e9b92cd9173d5 [id=0f45bbbc044d4e15b40e0a7f9edf6ef2]) successfully deleted
W20240502 08:17:04.176321 26001 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.038s sys 0.094s
W20240502 08:17:04.176447 26001 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.547s user 0.038s sys 0.094s
I20240502 08:17:04.177013 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a6881b68c76417bba71c8426abc4768. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.177215 24031 tablet_service.cc:1467] Processing CreateTablet for tablet db070085f9d94f86a25fc798cc2b8039 (DEFAULT_TABLE table=default.Table_f7cad6a15dbc4e7ab7e60c73a2102d0c [id=420e72b7a00c466289884e2a121ec8b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.177682 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db070085f9d94f86a25fc798cc2b8039. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.178185 25832 raft_consensus.cc:482] T 5e56af7b42cb481a8363d60f49ee3f61 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:04.178296 25832 raft_consensus.cc:504] T 5e56af7b42cb481a8363d60f49ee3f61 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.178844 25832 leader_election.cc:288] T 5e56af7b42cb481a8363d60f49ee3f61 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:04.179298 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e56af7b42cb481a8363d60f49ee3f61" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:04.179791 23861 leader_election.cc:341] T 5e56af7b42cb481a8363d60f49ee3f61 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.181588 26003 raft_consensus.cc:2781] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:04.181684 26003 raft_consensus.cc:482] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:04.181764 26003 raft_consensus.cc:3037] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:04.181870 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_86bd849a645b4a2b923b48963b9f51f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.182371 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86bd849a645b4a2b923b48963b9f51f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.183491 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_f1fdf44a6fa14967a3599402a901e520"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.183907 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1fdf44a6fa14967a3599402a901e520 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.183992 26003 raft_consensus.cc:504] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.184576 26003 leader_election.cc:288] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:17:04.184810 23723 leader_election.cc:341] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.184813 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 665bbee2d15b4112b651844431535d0e (table default.Table_64f173e9aa5947069cae45a6cb8dd18f [id=427121bc94c247eaa315b6c830770d61]). Expected version 3 got 0
I20240502 08:17:04.185004 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 07a9d13d3a2d4e6dbc04998c3dd7e1a8 (table default.Table_d9f68c6b26d442bbbd485bb89ee5564c [id=7a49ccfdd6884281ba9b882debd04871]). Expected version 1 got 0
I20240502 08:17:04.185313 23601 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet b0cf1a6de3d649b984c50f45393ec0f1 (table default.Table_d416b40fe3d24dafa8db5ee61d13acc0 [id=fa5100e581a7430ab8f0c19b2daae994]). Expected version 1 got 0
W20240502 08:17:04.186625 23723 leader_election.cc:341] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:17:04.186945 23723 leader_election.cc:341] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:17:04.187389 23723 leader_election.cc:341] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:17:04.187682 23723 leader_election.cc:341] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.188643 26000 consensus_queue.cc:1035] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:04.188984 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d8410ba7cc4b03bb64a138b1f9bb14" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:04.189155 23922 raft_consensus.cc:3037] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:04.190246 24051 raft_consensus.cc:3037] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:04.191412 24180 tablet_replica.cc:330] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:04.187711 23780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2792c45255c42adaeefb7aef4e25715" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:04.192145 23922 raft_consensus.cc:2454] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:17:04.192381 23858 leader_election.cc:302] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:04.193017 25494 raft_consensus.cc:2781] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:04.193101 25494 raft_consensus.cc:482] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:04.193154 25494 raft_consensus.cc:3037] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:04.193081 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d8410ba7cc4b03bb64a138b1f9bb14" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:04.195361 25494 raft_consensus.cc:504] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.195901 25494 leader_election.cc:288] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:17:04.196054 24173 tablet_bootstrap.cc:492] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:04.196202 24173 ts_tablet_manager.cc:1367] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.078s user 0.016s sys 0.017s
I20240502 08:17:04.196929 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e0f767c6534a608023e6e6d33ca7a8" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:17:04.197453 23861 leader_election.cc:341] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.198403 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 4e3570787d8242338d1ce01fb0299eb2
I20240502 08:17:04.198817 23608 catalog_manager.cc:6554] ReplaceTablet: tablet 4e3570787d8242338d1ce01fb0299eb2 deleted and replaced by tablet f3ae926889814758813d19a5fd44adf4
I20240502 08:17:04.199427 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3570787d8242338d1ce01fb0299eb2 with delete_type TABLET_DATA_DELETED (replaced by tablet f3ae926889814758813d19a5fd44adf4) from {username='slave'} at 127.0.0.1:51782
W20240502 08:17:04.199990 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 4e3570787d8242338d1ce01fb0299eb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e3570787d8242338d1ce01fb0299eb2 already in progress: creating tablet
I20240502 08:17:04.201151 23608 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 665bbee2d15b4112b651844431535d0e (table default.Table_64f173e9aa5947069cae45a6cb8dd18f [id=427121bc94c247eaa315b6c830770d61]). Expected version 3 got 0
I20240502 08:17:04.201346 23608 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 6088f160a13c4ff09f72bed72f69eaf8 (table default.Table_851f246623304504aa9ddf2e6c5e9082 [id=4fecac7b0d9042b3b4f674f3fc1338ba]). Expected version 1 got 0
I20240502 08:17:04.201726 23890 tablet_service.cc:1467] Processing CreateTablet for tablet db070085f9d94f86a25fc798cc2b8039 (DEFAULT_TABLE table=default.Table_f7cad6a15dbc4e7ab7e60c73a2102d0c [id=420e72b7a00c466289884e2a121ec8b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.202077 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db070085f9d94f86a25fc798cc2b8039. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.206485 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 524b604a536a48c18221560baf81b7d6 (DEFAULT_TABLE table=default.Table_ab7e9343fdc74b38bb98f2ead6c8eeb3 [id=2ff010494a374af78fe16a59cd35af5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.207239 25012 ts_tablet_manager.cc:1888] T d16110e1b1664802a98182c73f1e41d3 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:04.207363 25012 log.cc:1196] T d16110e1b1664802a98182c73f1e41d3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/d16110e1b1664802a98182c73f1e41d3
I20240502 08:17:04.207661 25012 ts_tablet_manager.cc:1909] T d16110e1b1664802a98182c73f1e41d3 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:04.207777 24173 raft_consensus.cc:348] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.208104 25012 tablet_replica.cc:330] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:04.208086 24173 raft_consensus.cc:374] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.208408 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet d16110e1b1664802a98182c73f1e41d3 (table default.Table_9feed0e1d82c472aaf0e9b92cd9173d5 [id=0f45bbbc044d4e15b40e0a7f9edf6ef2]) successfully deleted
I20240502 08:17:04.209405 24175 tablet_bootstrap.cc:654] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.211434 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6a09f0e331ba4a51bbecca6ddaf0bf51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.211968 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a09f0e331ba4a51bbecca6ddaf0bf51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.214205 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_24fea7b9f9a44b11a31e7eb63eafb557"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.214694 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24fea7b9f9a44b11a31e7eb63eafb557 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.209306 26029 raft_consensus.cc:482] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:04.215499 26029 raft_consensus.cc:504] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.216162 26029 leader_election.cc:288] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:04.208300 24173 raft_consensus.cc:729] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.216876 24173 consensus_queue.cc:260] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.216987 23922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d96cc17d9b4bcf8ce7c4414a63f9cb" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:04.219930 24173 ts_tablet_manager.cc:1398] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20240502 08:17:04.220213 24173 tablet_bootstrap.cc:492] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:17:04.220913 23723 consensus_peers.cc:485] T b0cf1a6de3d649b984c50f45393ec0f1 P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:04.220969 26011 consensus_queue.cc:1035] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20240502 08:17:04.221475 23783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b4a4d6a364de8af3b54d99a0ae138" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:04.221936 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0976fc5fd29413e9b27d05ad6ab4965. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.222141 24173 tablet_bootstrap.cc:654] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.223238 23780 raft_consensus.cc:2454] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:04.223385 24180 raft_consensus.cc:2229] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:04.225288 24173 tablet_bootstrap.cc:492] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:04.225427 24173 ts_tablet_manager.cc:1367] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:17:04.225684 23754 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3570787d8242338d1ce01fb0299eb2 with delete_type TABLET_DATA_DELETED (replaced by tablet f3ae926889814758813d19a5fd44adf4) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.226014 24174 tablet_bootstrap.cc:492] T 045b0882a40f4eaba94935d9bc90da0c P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:04.226282 24173 raft_consensus.cc:348] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.226467 24173 raft_consensus.cc:374] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.226521 24173 raft_consensus.cc:729] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.226712 24173 consensus_queue.cc:260] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.227059 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 524b604a536a48c18221560baf81b7d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.227049 23785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d96cc17d9b4bcf8ce7c4414a63f9cb" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:17:04.227488 23723 leader_election.cc:341] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.229723 24173 ts_tablet_manager.cc:1398] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:04.229723 23993 leader_election.cc:302] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:04.229979 24173 tablet_bootstrap.cc:492] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:04.230120 26029 raft_consensus.cc:2726] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:04.231351 24173 tablet_bootstrap.cc:654] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.232782 24180 raft_consensus.cc:2258] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.221814 23782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a74a6eb6d214d858903b3307e09e9e6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:04.238263 23782 raft_consensus.cc:2454] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:04.240695 24174 ts_tablet_manager.cc:1367] T 045b0882a40f4eaba94935d9bc90da0c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.256s user 0.003s sys 0.003s
I20240502 08:17:04.244515 23723 leader_election.cc:302] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:04.248993 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3570787d8242338d1ce01fb0299eb2 with delete_type TABLET_DATA_DELETED (replaced by tablet f3ae926889814758813d19a5fd44adf4) from {username='slave'} at 127.0.0.1:33620
W20240502 08:17:04.249080 23973 tablet.cc:2369] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:04.251686 23605 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet fba8f6e017e848d59239ae98432b1837 (table default.Table_c7de500cdfa341798dbbd1cc9ab5e4fb [id=3deaa06a1e88491f9dda6abda2cbdd9f]). Expected version 1 got 0
I20240502 08:17:04.253293 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c2a25d0d51cd40769e5045ff80c9c3fc" } modify_external_catalogs: true
I20240502 08:17:04.253511 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_c2a25d0d51cd40769e5045ff80c9c3fc" } modify_external_catalogs: true
I20240502 08:17:04.254746 24180 ts_tablet_manager.cc:1875] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:17:04.257709 23723 leader_election.cc:341] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.257745 25012 raft_consensus.cc:2229] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:04.257932 25012 raft_consensus.cc:2258] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.258419 26011 raft_consensus.cc:2726] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:04.258574 25012 ts_tablet_manager.cc:1875] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:04.261651 23609 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d26ea2d33fbc414e84ffcc7193fa3a52" } new_table_name: "default.Table_9fe575ed3b4549b28142df213b8baebc" modify_external_catalogs: true
W20240502 08:17:04.262426 23723 leader_election.cc:341] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.263201 23723 leader_election.cc:302] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
W20240502 08:17:04.263653 23723 consensus_peers.cc:485] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:04.264106 23723 leader_election.cc:341] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.265323 25903 raft_consensus.cc:2726] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:04.265655 26011 consensus_queue.cc:1035] T 87c36b622cd94f23b6e7ebdd2ab5b6b3 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:04.267107 23858 leader_election.cc:302] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:04.268213 25832 raft_consensus.cc:2781] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:04.268306 25832 raft_consensus.cc:482] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:04.268378 25832 raft_consensus.cc:3037] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:17:04.269349 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82a83530b77c4f8badd9c36c024823be consensus state has no leader
I20240502 08:17:04.269685 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_04f57727640b4aceabd37ba02a21237a" } new_table_name: "default.Table_c67bc16cd18d4585b827421062b583ef" modify_external_catalogs: true
I20240502 08:17:04.269889 24174 raft_consensus.cc:348] T 045b0882a40f4eaba94935d9bc90da0c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.270081 24174 raft_consensus.cc:374] T 045b0882a40f4eaba94935d9bc90da0c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.270152 24174 raft_consensus.cc:729] T 045b0882a40f4eaba94935d9bc90da0c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.270362 24174 consensus_queue.cc:260] T 045b0882a40f4eaba94935d9bc90da0c P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.270720 25832 raft_consensus.cc:504] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.270821 24174 ts_tablet_manager.cc:1398] T 045b0882a40f4eaba94935d9bc90da0c P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20240502 08:17:04.271096 24174 tablet_bootstrap.cc:492] T 19e4cd31e7774546a54704ccd405ffe6 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:04.272977 24174 tablet_bootstrap.cc:654] T 19e4cd31e7774546a54704ccd405ffe6 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.274523 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2792c45255c42adaeefb7aef4e25715" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:17:04.275403 23861 leader_election.cc:341] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.276386 25832 leader_election.cc:288] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:17:04.277321 24052 raft_consensus.cc:3037] T 90d4e2f23c9649a4b1dfb6a179ca7be5 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:04.265467 23723 leader_election.cc:302] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:04.278126 24180 ts_tablet_manager.cc:1888] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:04.278236 24180 log.cc:1196] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/bb4b4a4d6a364de8af3b54d99a0ae138
I20240502 08:17:04.278553 24180 ts_tablet_manager.cc:1909] T bb4b4a4d6a364de8af3b54d99a0ae138 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:04.278797 26011 raft_consensus.cc:2781] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:04.278915 26011 raft_consensus.cc:686] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:17:04.279165 26011 consensus_queue.cc:237] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.281714 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet bb4b4a4d6a364de8af3b54d99a0ae138 (table default.Table_86918f47dba74a9eb36be5145a017f5b [id=8dbcb2bdd1b14db192a3bd5dcc32fc9a]) successfully deleted
W20240502 08:17:04.282706 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet b9f83bcf640f465bb95d87a991cc5ff6 because the tablet was not found. No further retry: Not found: Tablet not found: b9f83bcf640f465bb95d87a991cc5ff6
I20240502 08:17:04.283069 24180 tablet_replica.cc:330] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:04.283401 24180 raft_consensus.cc:2229] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:04.283684 24180 pending_rounds.cc:62] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:17:04.283798 24180 pending_rounds.cc:68] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156524842135552 op_type: NO_OP noop_request { }
I20240502 08:17:04.284032 24180 raft_consensus.cc:2866] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:04.284183 24180 raft_consensus.cc:2258] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.285445 24175 tablet_bootstrap.cc:492] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:04.285676 24175 ts_tablet_manager.cc:1367] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.180s user 0.004s sys 0.000s
I20240502 08:17:04.286640 24175 raft_consensus.cc:348] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.289741 23760 tablet_service.cc:1467] Processing CreateTablet for tablet e1fbbcdd18b045728bbc110a5e616105 (DEFAULT_TABLE table=default.Table_312f784e0e4b43ec8666607e58f62ca3 [id=cb3c4778a91f4730ba5b62ce19c935b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.290707 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1fbbcdd18b045728bbc110a5e616105. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.291297 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 975a2ad2b32c4c3e864f0086c69b0551 (DEFAULT_TABLE table=default.Table_299008ee28694f2395585f3ce567a641 [id=f096c67d2d334f7b8d1abd0e02eb8474]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.291638 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975a2ad2b32c4c3e864f0086c69b0551. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.292778 24180 ts_tablet_manager.cc:1875] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:04.304756 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 975a2ad2b32c4c3e864f0086c69b0551 (DEFAULT_TABLE table=default.Table_299008ee28694f2395585f3ce567a641 [id=f096c67d2d334f7b8d1abd0e02eb8474]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.305260 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975a2ad2b32c4c3e864f0086c69b0551. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.305507 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e0f767c6534a608023e6e6d33ca7a8" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:04.305735 23787 raft_consensus.cc:3037] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:04.308244 23787 raft_consensus.cc:2454] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:04.311025 24031 tablet_service.cc:1467] Processing CreateTablet for tablet e1fbbcdd18b045728bbc110a5e616105 (DEFAULT_TABLE table=default.Table_312f784e0e4b43ec8666607e58f62ca3 [id=cb3c4778a91f4730ba5b62ce19c935b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.311420 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1fbbcdd18b045728bbc110a5e616105. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.321593 24173 tablet_bootstrap.cc:492] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:04.321738 24173 ts_tablet_manager.cc:1367] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.092s user 0.009s sys 0.005s
I20240502 08:17:04.322085 23761 tablet_service.cc:1467] Processing CreateTablet for tablet db070085f9d94f86a25fc798cc2b8039 (DEFAULT_TABLE table=default.Table_f7cad6a15dbc4e7ab7e60c73a2102d0c [id=420e72b7a00c466289884e2a121ec8b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.322541 24173 raft_consensus.cc:348] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.322755 24173 raft_consensus.cc:374] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.322809 24173 raft_consensus.cc:729] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.323899 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db070085f9d94f86a25fc798cc2b8039. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:04.300166 23858 leader_election.cc:341] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.320837 25012 ts_tablet_manager.cc:1888] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:04.324925 25012 log.cc:1196] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/bb4b4a4d6a364de8af3b54d99a0ae138
I20240502 08:17:04.325052 24175 raft_consensus.cc:374] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.325160 24175 raft_consensus.cc:729] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.325250 25012 ts_tablet_manager.cc:1909] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:04.325522 24175 consensus_queue.cc:260] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.324818 23858 leader_election.cc:302] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:04.325968 25012 tablet_replica.cc:330] T 72ee52bcc61b4816acda5b8140b63504 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
W20240502 08:17:04.326122 23858 raft_consensus.cc:2670] T bb4b4a4d6a364de8af3b54d99a0ae138 P 5b89a1f0485a44b7ac1710e2369a809b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:17:04.326375 25012 raft_consensus.cc:2229] T 72ee52bcc61b4816acda5b8140b63504 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:04.326547 25012 raft_consensus.cc:2258] T 72ee52bcc61b4816acda5b8140b63504 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.326603 23858 leader_election.cc:302] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
W20240502 08:17:04.327186 23858 leader_election.cc:341] T 5e56af7b42cb481a8363d60f49ee3f61 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 5e56af7b42cb481a8363d60f49ee3f61
I20240502 08:17:04.327356 23858 leader_election.cc:302] T 5e56af7b42cb481a8363d60f49ee3f61 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:04.327610 25832 raft_consensus.cc:2726] T 5e56af7b42cb481a8363d60f49ee3f61 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:04.328619 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8bbe9f5123cc4dacb9fd2749d84f9988" } modify_external_catalogs: true
I20240502 08:17:04.328790 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8bbe9f5123cc4dacb9fd2749d84f9988" } modify_external_catalogs: true
I20240502 08:17:04.330065 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3964effad9bb411dab9ba4e158fe33fd" } new_table_name: "default.Table_4bb4046caa9e462a92c3af8f5f3042d0" modify_external_catalogs: true
I20240502 08:17:04.327265 26052 raft_consensus.cc:2781] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:04.331864 26052 raft_consensus.cc:482] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:04.331955 26052 raft_consensus.cc:3037] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:04.334908 23858 leader_election.cc:302] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:04.335254 25649 raft_consensus.cc:2781] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:04.335335 25649 raft_consensus.cc:686] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:17:04.335565 25649 consensus_queue.cc:237] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.341315 23601 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_45691f9b0fd44328a709aa6a3e3b1f0d [id=d37b40e0c2d942369def844a0a6db568] alter complete (version 1)
I20240502 08:17:04.343487 25012 ts_tablet_manager.cc:1875] T 72ee52bcc61b4816acda5b8140b63504 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:04.344681 23601 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 665bbee2d15b4112b651844431535d0e (table default.Table_64f173e9aa5947069cae45a6cb8dd18f [id=427121bc94c247eaa315b6c830770d61]). Expected version 3 got 0
I20240502 08:17:04.346012 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d7579e78c8b34bb199ee967a0fe0afa5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.346561 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7579e78c8b34bb199ee967a0fe0afa5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.346571 24052 raft_consensus.cc:3037] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:04.346930 26003 consensus_queue.cc:1035] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:04.347397 23890 tablet_service.cc:1467] Processing CreateTablet for tablet e1fbbcdd18b045728bbc110a5e616105 (DEFAULT_TABLE table=default.Table_312f784e0e4b43ec8666607e58f62ca3 [id=cb3c4778a91f4730ba5b62ce19c935b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.347800 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1fbbcdd18b045728bbc110a5e616105. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.348459 23608 master_service.cc:946] ReplaceTablet: received request to replace tablet 0001b9d68e0a4b21a9bfd584dcdf99ea from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:04.349795 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 975a2ad2b32c4c3e864f0086c69b0551 (DEFAULT_TABLE table=default.Table_299008ee28694f2395585f3ce567a641 [id=f096c67d2d334f7b8d1abd0e02eb8474]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.349910 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet bb4b4a4d6a364de8af3b54d99a0ae138 (table default.Table_86918f47dba74a9eb36be5145a017f5b [id=8dbcb2bdd1b14db192a3bd5dcc32fc9a]) successfully deleted
I20240502 08:17:04.350281 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975a2ad2b32c4c3e864f0086c69b0551. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:04.350294 23597 catalog_manager.cc:4797] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): delete failed for tablet 0501575a84ec454780f48574bf3155fe because the tablet was not found. No further retry: Not found: Tablet not found: 0501575a84ec454780f48574bf3155fe
I20240502 08:17:04.352216 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ae314c4445fb416f97e233054cd2cb3a" } modify_external_catalogs: true
I20240502 08:17:04.352717 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_ae314c4445fb416f97e233054cd2cb3a" } modify_external_catalogs: true
I20240502 08:17:04.352953 26052 raft_consensus.cc:504] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.359033 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f306a288b90413b9d8ea8b76d23363a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.359323 24181 tablet_replica.cc:330] T 0f306a288b90413b9d8ea8b76d23363a P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:04.359529 24181 raft_consensus.cc:2229] T 0f306a288b90413b9d8ea8b76d23363a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:04.359668 24181 raft_consensus.cc:2258] T 0f306a288b90413b9d8ea8b76d23363a P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.360396 24181 ts_tablet_manager.cc:1875] T 0f306a288b90413b9d8ea8b76d23363a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:04.365155 23603 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 6088f160a13c4ff09f72bed72f69eaf8 (table default.Table_851f246623304504aa9ddf2e6c5e9082 [id=4fecac7b0d9042b3b4f674f3fc1338ba]). Expected version 1 got 0
I20240502 08:17:04.365409 23603 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet b0cf1a6de3d649b984c50f45393ec0f1 (table default.Table_d416b40fe3d24dafa8db5ee61d13acc0 [id=fa5100e581a7430ab8f0c19b2daae994]). Expected version 1 got 0
I20240502 08:17:04.365839 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet fddac683eda14ec1a0140926380f2c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.367010 23603 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 665bbee2d15b4112b651844431535d0e (table default.Table_64f173e9aa5947069cae45a6cb8dd18f [id=427121bc94c247eaa315b6c830770d61]). Expected version 3 got 0
I20240502 08:17:04.367208 23603 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 045b0882a40f4eaba94935d9bc90da0c (table default.Table_7c20ea5847184b48b0bd45fd5751b992 [id=abb72d2b04794e4aab7acee7a3a0925b]). Expected version 3 got 0
I20240502 08:17:04.367781 24181 ts_tablet_manager.cc:1888] T 0f306a288b90413b9d8ea8b76d23363a P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:04.367899 24181 log.cc:1196] T 0f306a288b90413b9d8ea8b76d23363a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/0f306a288b90413b9d8ea8b76d23363a
I20240502 08:17:04.368263 24181 ts_tablet_manager.cc:1909] T 0f306a288b90413b9d8ea8b76d23363a P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:04.368578 23603 master_service.cc:946] ReplaceTablet: received request to replace tablet 17d6f4f6e9ee46c28b7e7513f791c81f from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:04.368711 23754 tablet_service.cc:1467] Processing CreateTablet for tablet dd4d2571803643c88ffbd93d7909ac44 (DEFAULT_TABLE table=default.Table_a889270ae3d442d29f570221d6d5494f [id=0be2d1db3a8d4709bf7c21ddb6e735d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.369001 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_962cbf6a1b424252a27be241d7c3173f" } modify_external_catalogs: true
I20240502 08:17:04.369098 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd4d2571803643c88ffbd93d7909ac44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.369091 24181 tablet_replica.cc:330] T fddac683eda14ec1a0140926380f2c07 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:04.369150 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_962cbf6a1b424252a27be241d7c3173f" } modify_external_catalogs: true
I20240502 08:17:04.369253 24181 raft_consensus.cc:2229] T fddac683eda14ec1a0140926380f2c07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:04.369375 24181 raft_consensus.cc:2258] T fddac683eda14ec1a0140926380f2c07 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.370436 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae9c817c7be42da8c689267c7eb5a06 with delete_type TABLET_DATA_DELETED (replaced by tablet b3b0c0dcb2184ad39b984f718f8a2082) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:04.370631 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd9ac17dc714a4b87135b994eb50912 (DEFAULT_TABLE table=default.Table_98b47a2ad25448c6b8db287180fbff73 [id=19f7edcacb8442b784edafc047079040]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.371026 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd9ac17dc714a4b87135b994eb50912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.373373 24180 ts_tablet_manager.cc:1888] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:04.373605 24173 consensus_queue.cc:260] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:17:04.375789 23609 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a60eae19da824562961dba8d52920a97 consensus state has no leader
I20240502 08:17:04.375861 24173 ts_tablet_manager.cc:1398] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.054s user 0.002s sys 0.000s
I20240502 08:17:04.376010 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet 81a8101212df4f3e88bd322c23c17fd1 from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:04.376040 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 0f306a288b90413b9d8ea8b76d23363a (table default.Table_9bb2be9b511a49eca4228b582c5ac739 [id=97c877d344bd44d896a989e63768579b]) successfully deleted
I20240502 08:17:04.376927 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 7a23c06cb3704a7b83143a02efe2a8b3
I20240502 08:17:04.377292 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd9ac17dc714a4b87135b994eb50912 (DEFAULT_TABLE table=default.Table_98b47a2ad25448c6b8db287180fbff73 [id=19f7edcacb8442b784edafc047079040]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.377801 24023 tablet_service.cc:1467] Processing CreateTablet for tablet dd4d2571803643c88ffbd93d7909ac44 (DEFAULT_TABLE table=default.Table_a889270ae3d442d29f570221d6d5494f [id=0be2d1db3a8d4709bf7c21ddb6e735d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.378178 24175 ts_tablet_manager.cc:1398] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.092s user 0.002s sys 0.000s
I20240502 08:17:04.378430 24175 tablet_bootstrap.cc:492] T 23ba091d8efb44858bf90a104c0f442f P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:04.379570 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet d1218fd56b8a4e88827f275f952ad782
I20240502 08:17:04.380506 24180 log.cc:1196] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/72ee52bcc61b4816acda5b8140b63504
I20240502 08:17:04.380954 24180 ts_tablet_manager.cc:1909] T 72ee52bcc61b4816acda5b8140b63504 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:04.381589 24181 ts_tablet_manager.cc:1875] T fddac683eda14ec1a0140926380f2c07 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:04.382941 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a23c06cb3704a7b83143a02efe2a8b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.383040 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 72ee52bcc61b4816acda5b8140b63504 (table default.Table_d035b24dfb644b7aa77b160ee28d36e5 [id=19ef62dadf8943bdb06f82727634f764]) successfully deleted
I20240502 08:17:04.383102 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet d1218fd56b8a4e88827f275f952ad782 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.379092 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2792c45255c42adaeefb7aef4e25715" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:04.381350 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet d1218fd56b8a4e88827f275f952ad782 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.383152 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a23c06cb3704a7b83143a02efe2a8b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.383397 23787 raft_consensus.cc:3037] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:04.383760 24173 tablet_bootstrap.cc:492] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:04.383760 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd9ac17dc714a4b87135b994eb50912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.384620 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6e85449b13c249ed9c64e3a26eec0212"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.385133 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e85449b13c249ed9c64e3a26eec0212 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.385154 24173 tablet_bootstrap.cc:654] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.385630 23787 raft_consensus.cc:2454] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:04.385915 26029 raft_consensus.cc:482] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:04.386022 26029 raft_consensus.cc:504] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.386339 25832 consensus_queue.cc:1035] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:04.386788 23594 catalog_manager.cc:4797] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet fddac683eda14ec1a0140926380f2c07 because the tablet was not found. No further retry: Not found: Tablet not found: fddac683eda14ec1a0140926380f2c07
I20240502 08:17:04.387106 26029 leader_election.cc:288] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:04.387300 24180 tablet_replica.cc:330] T e9171ebf74424c9aae5e1c8a5118f768 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:04.387512 24180 raft_consensus.cc:2229] T e9171ebf74424c9aae5e1c8a5118f768 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:04.387490 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9171ebf74424c9aae5e1c8a5118f768" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:04.387694 24180 raft_consensus.cc:2258] T e9171ebf74424c9aae5e1c8a5118f768 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.388371 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9171ebf74424c9aae5e1c8a5118f768" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:04.388485 24180 ts_tablet_manager.cc:1875] T e9171ebf74424c9aae5e1c8a5118f768 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:17:04.388805 23993 leader_election.cc:341] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.388911 23993 leader_election.cc:302] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:04.391927 24033 tablet_service.cc:1467] Processing CreateTablet for tablet dc74b34f6cb74b248c54a8b21c352595 (DEFAULT_TABLE table=default.Table_1dcd211756884e2abc59e20c039a4566 [id=e2f5e4717e4d40a094762f2ef68fdf8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.392275 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc74b34f6cb74b248c54a8b21c352595. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.392683 23760 tablet_service.cc:1467] Processing CreateTablet for tablet dc74b34f6cb74b248c54a8b21c352595 (DEFAULT_TABLE table=default.Table_1dcd211756884e2abc59e20c039a4566 [id=e2f5e4717e4d40a094762f2ef68fdf8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.393015 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc74b34f6cb74b248c54a8b21c352595. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.394186 23758 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7e8b340d0f40e88930324eb76c0ead (DEFAULT_TABLE table=default.Table_f58d9956fcb946a8821dda27ca3bbe20 [id=bc3821ae3b4e4f3081b5bd0d00f81459]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.394487 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7e8b340d0f40e88930324eb76c0ead. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.395336 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7e8b340d0f40e88930324eb76c0ead (DEFAULT_TABLE table=default.Table_f58d9956fcb946a8821dda27ca3bbe20 [id=bc3821ae3b4e4f3081b5bd0d00f81459]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.395628 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7e8b340d0f40e88930324eb76c0ead. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:04.398813 23604 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e1f57ce662044b385369d95952cb0ac consensus state has no leader
I20240502 08:17:04.399178 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 255f69f56d344ce7805eca0bfdf8a063
I20240502 08:17:04.400445 23754 tablet_service.cc:1514] Processing DeleteTablet for tablet 255f69f56d344ce7805eca0bfdf8a063 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.403219 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd4d2571803643c88ffbd93d7909ac44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.405953 23607 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5d3aa312157c4ed7baafaa4864bcc205" } new_table_name: "default.Table_6bb940e8dece4a1387007d4c8feacf3f" modify_external_catalogs: true
I20240502 08:17:04.406803 24180 ts_tablet_manager.cc:1888] T e9171ebf74424c9aae5e1c8a5118f768 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:04.406987 24180 log.cc:1196] T e9171ebf74424c9aae5e1c8a5118f768 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/e9171ebf74424c9aae5e1c8a5118f768
I20240502 08:17:04.409639 24180 ts_tablet_manager.cc:1909] T e9171ebf74424c9aae5e1c8a5118f768 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
W20240502 08:17:04.411568 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed428ef1be8a49c19e4b6b48025b9cac consensus state has no leader
I20240502 08:17:04.411871 23902 tablet_service.cc:1467] Processing CreateTablet for tablet dc74b34f6cb74b248c54a8b21c352595 (DEFAULT_TABLE table=default.Table_1dcd211756884e2abc59e20c039a4566 [id=e2f5e4717e4d40a094762f2ef68fdf8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.412007 23890 tablet_service.cc:1467] Processing CreateTablet for tablet dd4d2571803643c88ffbd93d7909ac44 (DEFAULT_TABLE table=default.Table_a889270ae3d442d29f570221d6d5494f [id=0be2d1db3a8d4709bf7c21ddb6e735d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.412222 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc74b34f6cb74b248c54a8b21c352595. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.412482 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd9ac17dc714a4b87135b994eb50912 (DEFAULT_TABLE table=default.Table_98b47a2ad25448c6b8db287180fbff73 [id=19f7edcacb8442b784edafc047079040]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.412801 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd9ac17dc714a4b87135b994eb50912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.413102 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a23c06cb3704a7b83143a02efe2a8b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:04.413803 25012 ts_tablet_manager.cc:1888] T 72ee52bcc61b4816acda5b8140b63504 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:04.413856 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd4d2571803643c88ffbd93d7909ac44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.413901 25012 log.cc:1196] T 72ee52bcc61b4816acda5b8140b63504 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/72ee52bcc61b4816acda5b8140b63504
I20240502 08:17:04.414206 25012 ts_tablet_manager.cc:1909] T 72ee52bcc61b4816acda5b8140b63504 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:04.414608 25012 tablet_replica.cc:330] T 0f306a288b90413b9d8ea8b76d23363a P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:04.414765 25012 raft_consensus.cc:2229] T 0f306a288b90413b9d8ea8b76d23363a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:04.414899 25012 raft_consensus.cc:2258] T 0f306a288b90413b9d8ea8b76d23363a P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.414947 24180 tablet_replica.cc:330] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:04.415135 24180 raft_consensus.cc:2229] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:04.415333 24180 pending_rounds.cc:62] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:17:04.415364 25012 ts_tablet_manager.cc:1875] T 0f306a288b90413b9d8ea8b76d23363a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:04.415386 24180 pending_rounds.cc:68] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156526523322368 op_type: NO_OP noop_request { }
I20240502 08:17:04.415493 24180 raft_consensus.cc:2866] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:04.415575 24180 raft_consensus.cc:2258] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.416277 24180 ts_tablet_manager.cc:1875] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:04.417906 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7e8b340d0f40e88930324eb76c0ead (DEFAULT_TABLE table=default.Table_f58d9956fcb946a8821dda27ca3bbe20 [id=bc3821ae3b4e4f3081b5bd0d00f81459]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.418164 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7e8b340d0f40e88930324eb76c0ead. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.419325 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d49f09ff91c349f1b72e94bf914ae862"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.419762 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d49f09ff91c349f1b72e94bf914ae862 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.420953 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet e9171ebf74424c9aae5e1c8a5118f768 (table default.Table_dcbb721cb3904016a2866ef34f6bf557 [id=3711d24a9e544db98c2bda42becc3f81]) successfully deleted
I20240502 08:17:04.421387 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 255f69f56d344ce7805eca0bfdf8a063 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.421540 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 255f69f56d344ce7805eca0bfdf8a063 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:04.422607 26052 leader_election.cc:288] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:17:04.422761 24180 ts_tablet_manager.cc:1888] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:04.422871 24180 log.cc:1196] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/9f3c00e5393c47df924bcb5dcf2a0f19
I20240502 08:17:04.423885 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a74a6eb6d214d858903b3307e09e9e6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:04.425101 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet d1218fd56b8a4e88827f275f952ad782 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:04.425148 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 72ee52bcc61b4816acda5b8140b63504 (table default.Table_d035b24dfb644b7aa77b160ee28d36e5 [id=19ef62dadf8943bdb06f82727634f764]) successfully deleted
I20240502 08:17:04.426358 23858 leader_election.cc:302] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:04.426363 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_efc8673aacce43208f8c36e92f913555"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.426800 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_efc8673aacce43208f8c36e92f913555 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.427423 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a74a6eb6d214d858903b3307e09e9e6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:04.427623 23787 raft_consensus.cc:3037] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:17:04.428119 23861 leader_election.cc:341] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.430320 23787 raft_consensus.cc:2454] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:04.431071 23858 leader_election.cc:302] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:04.431334 25832 raft_consensus.cc:2781] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:04.431457 25832 raft_consensus.cc:686] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:17:04.432310 26018 raft_consensus.cc:2781] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:04.432694 26018 raft_consensus.cc:686] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:17:04.432870 23608 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 0001b9d68e0a4b21a9bfd584dcdf99ea
I20240502 08:17:04.433246 23608 catalog_manager.cc:6554] ReplaceTablet: tablet 0001b9d68e0a4b21a9bfd584dcdf99ea deleted and replaced by tablet f60d658c5c0c4cb2b205e031a3012059
I20240502 08:17:04.433682 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 0001b9d68e0a4b21a9bfd584dcdf99ea with delete_type TABLET_DATA_DELETED (replaced by tablet f60d658c5c0c4cb2b205e031a3012059) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.435079 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet 0001b9d68e0a4b21a9bfd584dcdf99ea with delete_type TABLET_DATA_DELETED (replaced by tablet f60d658c5c0c4cb2b205e031a3012059) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:04.436108 24173 tablet_bootstrap.cc:492] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:04.436246 24173 ts_tablet_manager.cc:1367] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.001s
I20240502 08:17:04.436976 24173 raft_consensus.cc:348] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.437155 24173 raft_consensus.cc:374] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.437199 24173 raft_consensus.cc:729] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.437394 24173 consensus_queue.cc:260] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.437466 25934 raft_consensus.cc:482] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:04.437568 25934 raft_consensus.cc:504] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.438422 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981b7a7292884b1fb48af99db6e762c0" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:04.438616 24173 ts_tablet_manager.cc:1398] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:04.438907 24173 tablet_bootstrap.cc:492] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:17:04.438905 23861 leader_election.cc:341] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.440647 24173 tablet_bootstrap.cc:654] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.442270 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 0001b9d68e0a4b21a9bfd584dcdf99ea with delete_type TABLET_DATA_DELETED (replaced by tablet f60d658c5c0c4cb2b205e031a3012059) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.443385 24180 ts_tablet_manager.cc:1909] T 9f3c00e5393c47df924bcb5dcf2a0f19 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:04.431722 25832 consensus_queue.cc:237] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.444761 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 9f3c00e5393c47df924bcb5dcf2a0f19 (table default.Table_a44933d855bd48aa8865c11eaaa5cbe4 [id=7227b8f1e8164cc8aee8ded7c5917ac2]) successfully deleted
I20240502 08:17:04.445096 25934 leader_election.cc:288] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:17:04.445287 24180 tablet_replica.cc:330] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:04.445652 24180 raft_consensus.cc:2229] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:04.445920 24180 raft_consensus.cc:2258] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.446390 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981b7a7292884b1fb48af99db6e762c0" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:04.446609 23787 raft_consensus.cc:2454] T 981b7a7292884b1fb48af99db6e762c0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:04.446847 24180 ts_tablet_manager.cc:1875] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:04.446967 23858 leader_election.cc:302] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:04.447221 25934 raft_consensus.cc:2781] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:04.447290 25934 raft_consensus.cc:482] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:04.447351 25934 raft_consensus.cc:3037] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:04.454330 24180 ts_tablet_manager.cc:1888] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:04.454536 24180 log.cc:1196] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/76e0f767c6534a608023e6e6d33ca7a8
I20240502 08:17:04.454943 24180 ts_tablet_manager.cc:1909] T 76e0f767c6534a608023e6e6d33ca7a8 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:04.432900 26018 consensus_queue.cc:237] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.456015 24174 tablet_bootstrap.cc:492] T 19e4cd31e7774546a54704ccd405ffe6 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:04.456135 24174 ts_tablet_manager.cc:1367] T 19e4cd31e7774546a54704ccd405ffe6 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.185s user 0.002s sys 0.008s
I20240502 08:17:04.456107 23760 tablet_service.cc:1467] Processing CreateTablet for tablet fe0040ae6d3e464c8cdc31edc70516b0 (DEFAULT_TABLE table=default.Table_76ef356c8f6543ec93536b3079814cff [id=c67aff10f32a419f9b84e744ad201c05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.456598 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0040ae6d3e464c8cdc31edc70516b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.458596 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 76e0f767c6534a608023e6e6d33ca7a8 (table default.Table_7929569b25404ea08efa4875a3c241a3 [id=51db0a1374374028806594b722a42fcc]) successfully deleted
I20240502 08:17:04.462749 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 0fb9750041a44138bcf463da241e5e3a
I20240502 08:17:04.462725 24174 raft_consensus.cc:348] T 19e4cd31e7774546a54704ccd405ffe6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.462888 24174 raft_consensus.cc:374] T 19e4cd31e7774546a54704ccd405ffe6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.462925 24174 raft_consensus.cc:729] T 19e4cd31e7774546a54704ccd405ffe6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.463073 24174 consensus_queue.cc:260] T 19e4cd31e7774546a54704ccd405ffe6 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.463469 24174 ts_tablet_manager.cc:1398] T 19e4cd31e7774546a54704ccd405ffe6 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 08:17:04.463573 24180 tablet_replica.cc:330] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:04.463690 24180 raft_consensus.cc:2229] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:04.463815 24180 raft_consensus.cc:2258] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.463928 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb9750041a44138bcf463da241e5e3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.464037 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb9750041a44138bcf463da241e5e3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.464561 24180 ts_tablet_manager.cc:1875] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:04.465420 23890 tablet_service.cc:1467] Processing CreateTablet for tablet fe0040ae6d3e464c8cdc31edc70516b0 (DEFAULT_TABLE table=default.Table_76ef356c8f6543ec93536b3079814cff [id=c67aff10f32a419f9b84e744ad201c05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.467803 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet cc4c8aea662149148032a1f13adddc5f
I20240502 08:17:04.468534 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 81a8101212df4f3e88bd322c23c17fd1
I20240502 08:17:04.468890 23609 catalog_manager.cc:6554] ReplaceTablet: tablet 81a8101212df4f3e88bd322c23c17fd1 deleted and replaced by tablet e8f78713c7d14703a0874bcb664ae655
I20240502 08:17:04.468919 24174 tablet_bootstrap.cc:492] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:04.470356 24174 tablet_bootstrap.cc:654] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.471343 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet cc4c8aea662149148032a1f13adddc5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.471594 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet 81a8101212df4f3e88bd322c23c17fd1 with delete_type TABLET_DATA_DELETED (replaced by tablet e8f78713c7d14703a0874bcb664ae655) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.472551 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet cc4c8aea662149148032a1f13adddc5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:04.472643 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb9750041a44138bcf463da241e5e3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:04.472643 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet 81a8101212df4f3e88bd322c23c17fd1 with delete_type TABLET_DATA_DELETED (replaced by tablet e8f78713c7d14703a0874bcb664ae655) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:04.473846 24180 ts_tablet_manager.cc:1888] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:04.473934 24180 log.cc:1196] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/3a74a6eb6d214d858903b3307e09e9e6
I20240502 08:17:04.474196 24180 ts_tablet_manager.cc:1909] T 3a74a6eb6d214d858903b3307e09e9e6 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:04.474236 25934 raft_consensus.cc:504] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.474823 24175 tablet_bootstrap.cc:654] T 23ba091d8efb44858bf90a104c0f442f P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.475106 24180 tablet_replica.cc:330] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:04.475179 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981b7a7292884b1fb48af99db6e762c0" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:04.475283 24180 raft_consensus.cc:2229] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:04.475309 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 3a74a6eb6d214d858903b3307e09e9e6 (table default.Table_c22209bdcbfd4dfa9e643ab9c98dc005 [id=232529911c8d43e2860c52d5527597d1]) successfully deleted
I20240502 08:17:04.475414 24180 raft_consensus.cc:2258] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.475107 25934 leader_election.cc:288] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
W20240502 08:17:04.475762 23861 leader_election.cc:341] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.475867 24180 ts_tablet_manager.cc:1875] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:04.476186 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0040ae6d3e464c8cdc31edc70516b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.476445 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981b7a7292884b1fb48af99db6e762c0" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:04.476600 23787 raft_consensus.cc:3037] T 981b7a7292884b1fb48af99db6e762c0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:04.479637 23787 raft_consensus.cc:2454] T 981b7a7292884b1fb48af99db6e762c0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:04.480067 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet 81a8101212df4f3e88bd322c23c17fd1 with delete_type TABLET_DATA_DELETED (replaced by tablet e8f78713c7d14703a0874bcb664ae655) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.480065 23858 leader_election.cc:302] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:04.480396 25934 raft_consensus.cc:2781] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:04.480552 25934 raft_consensus.cc:686] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:17:04.481381 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet cc4c8aea662149148032a1f13adddc5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.482810 24180 ts_tablet_manager.cc:1888] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:04.482976 24023 tablet_service.cc:1467] Processing CreateTablet for tablet fe0040ae6d3e464c8cdc31edc70516b0 (DEFAULT_TABLE table=default.Table_76ef356c8f6543ec93536b3079814cff [id=c67aff10f32a419f9b84e744ad201c05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.483424 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0040ae6d3e464c8cdc31edc70516b0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:04.483424 23607 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1d46ced6ac2e439b99fd6fa29e1a5a52 consensus state has no leader
I20240502 08:17:04.486804 24180 log.cc:1196] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/e2792c45255c42adaeefb7aef4e25715
I20240502 08:17:04.487416 24180 ts_tablet_manager.cc:1909] T e2792c45255c42adaeefb7aef4e25715 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:04.480834 25934 consensus_queue.cc:237] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:17:04.490600 23858 consensus_peers.cc:485] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9171ebf74424c9aae5e1c8a5118f768. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:04.490645 26029 raft_consensus.cc:2726] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:17:04.507261 24181 ts_tablet_manager.cc:1888] T fddac683eda14ec1a0140926380f2c07 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:04.507758 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet e2792c45255c42adaeefb7aef4e25715 (table default.Table_4c264de039c8469284d9921dc3bdf875 [id=e1658213b68a4cbc91a9435d0b238cb0]) successfully deleted
I20240502 08:17:04.507836 24180 tablet_replica.cc:330] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:04.508216 24180 raft_consensus.cc:2229] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:04.508447 24180 pending_rounds.cc:62] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:17:04.508507 24180 pending_rounds.cc:68] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156526993637376 op_type: NO_OP noop_request { }
I20240502 08:17:04.508605 24180 raft_consensus.cc:2866] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:04.508671 24180 raft_consensus.cc:2258] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.509102 24181 log.cc:1196] T fddac683eda14ec1a0140926380f2c07 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/fddac683eda14ec1a0140926380f2c07
I20240502 08:17:04.509326 24180 ts_tablet_manager.cc:1875] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:04.509886 24181 ts_tablet_manager.cc:1909] T fddac683eda14ec1a0140926380f2c07 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:04.515108 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet fddac683eda14ec1a0140926380f2c07 (table default.Table_4b967ef2b2fc4e2caa4219afcae841a6 [id=c32fc96889554c32bdf1d4550633e91f]) successfully deleted
W20240502 08:17:04.515538 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 7a23c06cb3704a7b83143a02efe2a8b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a23c06cb3704a7b83143a02efe2a8b3 already in progress: creating tablet
W20240502 08:17:04.516628 23723 consensus_peers.cc:485] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:04.520637 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet d1218fd56b8a4e88827f275f952ad782 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1218fd56b8a4e88827f275f952ad782 already in progress: creating tablet
I20240502 08:17:04.521725 24180 ts_tablet_manager.cc:1888] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:04.521975 24180 log.cc:1196] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/91a3455e5931488da3134d6b5c18aef2
I20240502 08:17:04.522457 24180 ts_tablet_manager.cc:1909] T 91a3455e5931488da3134d6b5c18aef2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:04.523730 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 91a3455e5931488da3134d6b5c18aef2 (table default.Table_d3c66c36f4cd494585b1e4cb114b81a9 [id=c0408e826b4d4bcd819c828dca1bd9cb]) successfully deleted
I20240502 08:17:04.524811 24180 tablet_replica.cc:330] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:04.525313 24180 raft_consensus.cc:2229] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:04.525964 24180 pending_rounds.cc:62] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:17:04.526443 24180 pending_rounds.cc:68] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156522402144256 op_type: NO_OP noop_request { }
I20240502 08:17:04.526748 24180 raft_consensus.cc:2866] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:04.526983 24180 raft_consensus.cc:2258] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.527761 24180 ts_tablet_manager.cc:1875] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:04.535912 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e7d36f8ce9ad4afeaf61010b5b2513bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:04.536078 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_36bbcbb0bb934222b8c149b2da3c7957"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.536492 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7d36f8ce9ad4afeaf61010b5b2513bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.536710 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7c74c34063104d54b3ab77c878892069" } new_table_name: "default.Table_353ed567df1d4ae2b174cc32bbc8e634" modify_external_catalogs: true
I20240502 08:17:04.535912 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1fd5707963d04b938961fa6fa33e84c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:04.539196 23601 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 6088f160a13c4ff09f72bed72f69eaf8 (table default.Table_851f246623304504aa9ddf2e6c5e9082 [id=4fecac7b0d9042b3b4f674f3fc1338ba]). Expected version 1 got 0
W20240502 08:17:04.539274 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fd5707963d04b938961fa6fa33e84c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.543249 23603 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 90d4e2f23c9649a4b1dfb6a179ca7be5 (table default.Table_b2e1c08bfc404c7ebb2720631ad10495 [id=9480d59a76fc401e959a1e41caafc845]). Expected version 3 got 0
I20240502 08:17:04.543416 23603 catalog_manager.cc:5576] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163) does not have the latest schema for tablet 6088f160a13c4ff09f72bed72f69eaf8 (table default.Table_851f246623304504aa9ddf2e6c5e9082 [id=4fecac7b0d9042b3b4f674f3fc1338ba]). Expected version 1 got 0
I20240502 08:17:04.545172 24180 ts_tablet_manager.cc:1888] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:04.545277 24180 log.cc:1196] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/0281c8123eab4c48bddc377f5695c6e0
I20240502 08:17:04.546315 24180 ts_tablet_manager.cc:1909] T 0281c8123eab4c48bddc377f5695c6e0 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:04.547421 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 0281c8123eab4c48bddc377f5695c6e0 (table default.Table_3e0123590a92473dabaacc9c158859e7 [id=d1a317371a5b43f185eb6453861e112a]) successfully deleted
I20240502 08:17:04.548713 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet e9171ebf74424c9aae5e1c8a5118f768 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:43 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.548756 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ee3de52e0dcb4bfea556a83fee836104"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:04.548921 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3c00e5393c47df924bcb5dcf2a0f19 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:51 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:17:04.549168 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee3de52e0dcb4bfea556a83fee836104 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:17:04.549651 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet f13bd33dd7c44ac19fa4e0bba3083483 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f13bd33dd7c44ac19fa4e0bba3083483 already in progress: creating tablet
W20240502 08:17:04.550472 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 66ea6743a3a44f9b95e1ca3d50a3dd40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66ea6743a3a44f9b95e1ca3d50a3dd40 already in progress: creating tablet
I20240502 08:17:04.550892 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet bb4b4a4d6a364de8af3b54d99a0ae138 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:47 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:17:04.536492 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36bbcbb0bb934222b8c149b2da3c7957 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.551260 24180 tablet_replica.cc:330] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:04.551543 24180 raft_consensus.cc:2229] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:04.552006 24180 pending_rounds.cc:62] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:17:04.552294 24180 pending_rounds.cc:68] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156528249659392 op_type: NO_OP noop_request { }
I20240502 08:17:04.552551 24180 raft_consensus.cc:2866] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:04.552729 24180 raft_consensus.cc:2258] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.554360 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 0281c8123eab4c48bddc377f5695c6e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:55 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.554492 26029 raft_consensus.cc:482] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:04.554594 26029 raft_consensus.cc:504] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.555126 26029 leader_election.cc:288] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:17:04.556172 23993 leader_election.cc:341] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 9f3c00e5393c47df924bcb5dcf2a0f19
I20240502 08:17:04.557732 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3c00e5393c47df924bcb5dcf2a0f19" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:04.558197 23916 raft_consensus.cc:2379] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 149ce744feab49b5aa5bcaa14bd6aba5 in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:17:04.559108 23996 leader_election.cc:302] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:04.559448 24173 tablet_bootstrap.cc:492] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:04.559600 24173 ts_tablet_manager.cc:1367] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.121s user 0.002s sys 0.002s
I20240502 08:17:04.560504 24173 raft_consensus.cc:348] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.560895 24173 raft_consensus.cc:374] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.560963 24173 raft_consensus.cc:729] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.561223 24173 consensus_queue.cc:260] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.561549 26027 raft_consensus.cc:3037] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:04.564344 26027 raft_consensus.cc:2726] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:04.565573 24173 ts_tablet_manager.cc:1398] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:17:04.566243 24173 tablet_bootstrap.cc:492] T b2eac3bde41c4415bdb97fc53a5b395f P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:17:04.568899 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 255f69f56d344ce7805eca0bfdf8a063 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 255f69f56d344ce7805eca0bfdf8a063 already in progress: creating tablet
W20240502 08:17:04.569109 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 0001b9d68e0a4b21a9bfd584dcdf99ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0001b9d68e0a4b21a9bfd584dcdf99ea already in progress: creating tablet
W20240502 08:17:04.569399 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 0fb9750041a44138bcf463da241e5e3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fb9750041a44138bcf463da241e5e3a already in progress: creating tablet
W20240502 08:17:04.569969 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet cc4c8aea662149148032a1f13adddc5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc4c8aea662149148032a1f13adddc5f already in progress: creating tablet
W20240502 08:17:04.570159 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 81a8101212df4f3e88bd322c23c17fd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81a8101212df4f3e88bd322c23c17fd1 already in progress: creating tablet
I20240502 08:17:04.574280 25012 ts_tablet_manager.cc:1888] T 0f306a288b90413b9d8ea8b76d23363a P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:04.574385 25012 log.cc:1196] T 0f306a288b90413b9d8ea8b76d23363a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/0f306a288b90413b9d8ea8b76d23363a
I20240502 08:17:04.574613 25012 ts_tablet_manager.cc:1909] T 0f306a288b90413b9d8ea8b76d23363a P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:04.574983 25012 tablet_replica.cc:330] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:04.575088 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 0f306a288b90413b9d8ea8b76d23363a (table default.Table_9bb2be9b511a49eca4228b582c5ac739 [id=97c877d344bd44d896a989e63768579b]) successfully deleted
I20240502 08:17:04.575163 25012 raft_consensus.cc:2229] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:04.575285 25012 pending_rounds.cc:62] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b: Trying to abort 1 pending ops.
I20240502 08:17:04.575330 25012 pending_rounds.cc:68] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156522402144256 op_type: NO_OP noop_request { }
I20240502 08:17:04.575431 25012 raft_consensus.cc:2866] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:04.575506 25012 raft_consensus.cc:2258] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.568045 24181 tablet_replica.cc:330] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:04.568323 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet d16110e1b1664802a98182c73f1e41d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.577546 24181 raft_consensus.cc:2229] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:04.577971 24181 raft_consensus.cc:2258] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.581468 25012 ts_tablet_manager.cc:1875] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:04.576113 24174 tablet_bootstrap.cc:492] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:04.583276 26000 raft_consensus.cc:482] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:04.586746 24173 tablet_bootstrap.cc:654] T b2eac3bde41c4415bdb97fc53a5b395f P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.586730 26000 raft_consensus.cc:504] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.587455 26000 leader_election.cc:288] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:04.588415 24174 ts_tablet_manager.cc:1367] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.120s user 0.009s sys 0.019s
I20240502 08:17:04.588443 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee764398b364d029ae7f0f575a30d64" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:04.588944 23723 leader_election.cc:341] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.589498 24174 raft_consensus.cc:348] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.589733 24174 raft_consensus.cc:374] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.589820 24174 raft_consensus.cc:729] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.590049 24174 consensus_queue.cc:260] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.595840 24174 ts_tablet_manager.cc:1398] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 08:17:04.600611 24174 tablet_bootstrap.cc:492] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:04.602046 24174 tablet_bootstrap.cc:654] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.603861 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee764398b364d029ae7f0f575a30d64" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:04.604470 24174 tablet_bootstrap.cc:492] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:04.604578 24174 ts_tablet_manager.cc:1367] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:17:04.604135 23914 raft_consensus.cc:2454] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:04.605262 23723 leader_election.cc:302] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:04.605726 26000 raft_consensus.cc:2781] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:04.605801 26000 raft_consensus.cc:482] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:04.605854 26000 raft_consensus.cc:3037] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:04.606125 24174 raft_consensus.cc:348] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.606288 24174 raft_consensus.cc:374] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.606330 24174 raft_consensus.cc:729] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.606252 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_921ba2dcfb7a4c51bc2f12ac85328ef9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:04.606642 24174 consensus_queue.cc:260] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:17:04.606885 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_921ba2dcfb7a4c51bc2f12ac85328ef9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.607127 24174 ts_tablet_manager.cc:1398] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:04.607211 23603 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_f0a7174ad23742efad1bbd0a285e2731" } new_table_name: "default.Table_5904e9a9ca9f4e9eabf14ec7e39e9062" modify_external_catalogs: true
I20240502 08:17:04.607574 24175 tablet_bootstrap.cc:492] T 23ba091d8efb44858bf90a104c0f442f P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:04.607872 24174 tablet_bootstrap.cc:492] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:04.608330 24175 ts_tablet_manager.cc:1367] T 23ba091d8efb44858bf90a104c0f442f P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.230s user 0.007s sys 0.000s
I20240502 08:17:04.608366 26000 raft_consensus.cc:504] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.608566 23610 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1bdb95f894a04d6bbeafc73022a2283c" } modify_external_catalogs: true
I20240502 08:17:04.608952 26000 leader_election.cc:288] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:04.608937 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_b92c1042c74e4e16b0298b65923e32d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.609324 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b92c1042c74e4e16b0298b65923e32d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.609548 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_7a5da2bba6f64c658d98d56c9465e80f" } new_table_name: "default.Table_65264907568d4852a3a32e936797fda3" modify_external_catalogs: true
I20240502 08:17:04.609700 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_308171ca5d9c4bfdb1d31a3b6e92b62f" } modify_external_catalogs: true
I20240502 08:17:04.609999 26001 consensus_queue.cc:1035] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:04.610714 24173 tablet_bootstrap.cc:492] T b2eac3bde41c4415bdb97fc53a5b395f P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:04.610847 24173 ts_tablet_manager.cc:1367] T b2eac3bde41c4415bdb97fc53a5b395f P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.003s
I20240502 08:17:04.610822 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee764398b364d029ae7f0f575a30d64" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:17:04.611449 23723 leader_election.cc:341] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.611599 24173 raft_consensus.cc:348] T b2eac3bde41c4415bdb97fc53a5b395f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.611769 24173 raft_consensus.cc:374] T b2eac3bde41c4415bdb97fc53a5b395f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.611829 24173 raft_consensus.cc:729] T b2eac3bde41c4415bdb97fc53a5b395f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.612031 24173 consensus_queue.cc:260] T b2eac3bde41c4415bdb97fc53a5b395f P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.612695 24173 ts_tablet_manager.cc:1398] T b2eac3bde41c4415bdb97fc53a5b395f P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:04.614213 24173 tablet_bootstrap.cc:492] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:04.615571 24181 ts_tablet_manager.cc:1875] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:04.617501 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee764398b364d029ae7f0f575a30d64" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:04.617679 23916 raft_consensus.cc:3037] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:04.615615 24173 tablet_bootstrap.cc:654] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.619161 24174 tablet_bootstrap.cc:654] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.620982 23916 raft_consensus.cc:2454] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:17:04.621495 23723 leader_election.cc:302] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:04.621848 26001 raft_consensus.cc:2781] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:04.621934 26001 raft_consensus.cc:686] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:17:04.622149 26001 consensus_queue.cc:237] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.643043 24174 tablet_bootstrap.cc:492] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:04.651170 24174 ts_tablet_manager.cc:1367] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.002s
I20240502 08:17:04.654256 24174 raft_consensus.cc:348] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.654425 24174 raft_consensus.cc:374] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.654475 24174 raft_consensus.cc:729] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.654649 24174 consensus_queue.cc:260] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.660769 24174 ts_tablet_manager.cc:1398] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 08:17:04.661175 24174 tablet_bootstrap.cc:492] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:04.662600 24174 tablet_bootstrap.cc:654] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.663659 26000 raft_consensus.cc:482] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:04.663775 26000 raft_consensus.cc:504] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.664288 26000 leader_election.cc:288] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:04.664920 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e6a1ba23dd4f82bc721a60c0526871" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:04.665325 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e6a1ba23dd4f82bc721a60c0526871" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:04.665409 23723 leader_election.cc:341] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:17:04.665786 23723 leader_election.cc:341] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.665894 23723 leader_election.cc:302] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:04.666410 26000 raft_consensus.cc:2726] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:04.671808 23610 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1bdb95f894a04d6bbeafc73022a2283c" } modify_external_catalogs: true
I20240502 08:17:04.681677 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet b2eac3bde41c4415bdb97fc53a5b395f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.683013 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_33297782b5e64503b4186fe4a347f8e9" } modify_external_catalogs: true
I20240502 08:17:04.683208 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_33297782b5e64503b4186fe4a347f8e9" } modify_external_catalogs: true
I20240502 08:17:04.685425 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_308171ca5d9c4bfdb1d31a3b6e92b62f" } modify_external_catalogs: true
I20240502 08:17:04.686503 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_fc4d6fa243f34d899165684a654feb3e" } new_table_name: "default.Table_0d35709e973f412bb0980fc6ffeba001" modify_external_catalogs: true
I20240502 08:17:04.687639 23609 master_service.cc:946] ReplaceTablet: received request to replace tablet 121668ea2e2943c4b61354b8c165000e from {username='slave'} at 127.0.0.1:36668
W20240502 08:17:04.691206 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f5496fa258de4988b9502136f0baf779 consensus state has no leader
I20240502 08:17:04.691578 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_ae752acc232641abb68b65f3530c58d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:04.691735 24174 tablet_bootstrap.cc:492] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:04.691861 24174 ts_tablet_manager.cc:1367] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
W20240502 08:17:04.691990 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae752acc232641abb68b65f3530c58d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.694243 23604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_de22e733a5464ad5830bf5f3a5d41cc9" } modify_external_catalogs: true
I20240502 08:17:04.694386 23604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_de22e733a5464ad5830bf5f3a5d41cc9" } modify_external_catalogs: true
I20240502 08:17:04.695089 24180 ts_tablet_manager.cc:1875] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:04.696194 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5a9deff891ad40538ea9f2574ee0ff55" } modify_external_catalogs: true
I20240502 08:17:04.696348 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5a9deff891ad40538ea9f2574ee0ff55" } modify_external_catalogs: true
W20240502 08:17:04.698477 23603 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 586fc16fa81f40c1b8554f9894be4092 consensus state has no leader
I20240502 08:17:04.698772 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_92101faa794b4c918cf635eb3e135147"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.699193 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92101faa794b4c918cf635eb3e135147 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.699944 26017 tablet.cc:1722] T 665bbee2d15b4112b651844431535d0e P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:17:04.701073 23610 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 338fa14b1e004c36accccb3b671103c1
W20240502 08:17:04.702107 26079 log.cc:924] Time spent T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20240502 08:17:04.704335 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet 338fa14b1e004c36accccb3b671103c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:04.704820 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 338fa14b1e004c36accccb3b671103c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.707741 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 338fa14b1e004c36accccb3b671103c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.710047 24174 raft_consensus.cc:348] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.710301 24173 tablet_bootstrap.cc:492] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:04.710443 24173 ts_tablet_manager.cc:1367] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.096s user 0.002s sys 0.006s
I20240502 08:17:04.711202 24173 raft_consensus.cc:348] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.711391 24173 raft_consensus.cc:374] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.711445 24173 raft_consensus.cc:729] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.711289 23610 catalog_manager.cc:5484] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65). New cstate: current_term: 1 leader_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:04.711486 24174 raft_consensus.cc:374] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.711621 24174 raft_consensus.cc:729] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.711664 24173 consensus_queue.cc:260] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.711892 24174 consensus_queue.cc:260] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.712044 23610 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 07a9d13d3a2d4e6dbc04998c3dd7e1a8 (table default.Table_d9f68c6b26d442bbbd485bb89ee5564c [id=7a49ccfdd6884281ba9b882debd04871]). Expected version 1 got 0
I20240502 08:17:04.712538 24174 ts_tablet_manager.cc:1398] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240502 08:17:04.712855 24174 tablet_bootstrap.cc:492] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:04.713097 23608 catalog_manager.cc:5484] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66). New cstate: current_term: 1 leader_uuid: "5b89a1f0485a44b7ac1710e2369a809b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:04.713485 23608 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 981b7a7292884b1fb48af99db6e762c0 (table default.Table_2911b4ae0383471eafe8761c4674beed [id=a681d30fb3e5469aad1bc27dfdd56ef7]). Expected version 3 got 0
I20240502 08:17:04.714604 24174 tablet_bootstrap.cc:654] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.718298 24175 raft_consensus.cc:348] T 23ba091d8efb44858bf90a104c0f442f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.718497 24175 raft_consensus.cc:374] T 23ba091d8efb44858bf90a104c0f442f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.718556 24175 raft_consensus.cc:729] T 23ba091d8efb44858bf90a104c0f442f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.718791 24175 consensus_queue.cc:260] T 23ba091d8efb44858bf90a104c0f442f P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.719408 24173 ts_tablet_manager.cc:1398] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 08:17:04.720243 24173 tablet_bootstrap.cc:492] T 72ee52bcc61b4816acda5b8140b63504 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:04.720381 24175 ts_tablet_manager.cc:1398] T 23ba091d8efb44858bf90a104c0f442f P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.112s user 0.002s sys 0.000s
I20240502 08:17:04.720863 26001 raft_consensus.cc:482] T 045b0882a40f4eaba94935d9bc90da0c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:04.721189 24175 tablet_bootstrap.cc:492] T 2ebf55c8024c483f9ef0afe6945a02ff P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:04.721089 26001 raft_consensus.cc:504] T 045b0882a40f4eaba94935d9bc90da0c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.722455 26001 leader_election.cc:288] T 045b0882a40f4eaba94935d9bc90da0c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:04.722505 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045b0882a40f4eaba94935d9bc90da0c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:04.722978 23723 leader_election.cc:341] T 045b0882a40f4eaba94935d9bc90da0c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.722669 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045b0882a40f4eaba94935d9bc90da0c" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:17:04.723558 23723 leader_election.cc:341] T 045b0882a40f4eaba94935d9bc90da0c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.723665 23723 leader_election.cc:302] T 045b0882a40f4eaba94935d9bc90da0c P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:04.724171 26001 raft_consensus.cc:2726] T 045b0882a40f4eaba94935d9bc90da0c P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:04.730548 24181 ts_tablet_manager.cc:1888] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:04.730564 24174 tablet_bootstrap.cc:492] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:04.730650 24181 log.cc:1196] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/e9171ebf74424c9aae5e1c8a5118f768
I20240502 08:17:04.730690 24174 ts_tablet_manager.cc:1367] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.002s
I20240502 08:17:04.731519 24174 raft_consensus.cc:348] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.731734 24174 raft_consensus.cc:374] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.731782 24174 raft_consensus.cc:729] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.732003 24174 consensus_queue.cc:260] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.732882 24174 ts_tablet_manager.cc:1398] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:04.723052 24175 tablet_bootstrap.cc:654] T 2ebf55c8024c483f9ef0afe6945a02ff P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.733162 24174 tablet_bootstrap.cc:492] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:04.734732 24174 tablet_bootstrap.cc:654] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.738843 24173 tablet_bootstrap.cc:654] T 72ee52bcc61b4816acda5b8140b63504 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.747404 24174 tablet_bootstrap.cc:492] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:04.747557 24174 ts_tablet_manager.cc:1367] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240502 08:17:04.748445 24174 raft_consensus.cc:348] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.748616 24174 raft_consensus.cc:374] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.748665 24174 raft_consensus.cc:729] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.748936 24174 consensus_queue.cc:260] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.749665 24174 ts_tablet_manager.cc:1398] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:17:04.750291 26017 tablet.cc:1722] T 55f3f09fc1f94c00a47b9087b98e221d P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:17:04.751863 26090 tablet.cc:1722] T 665bbee2d15b4112b651844431535d0e P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:17:04.752671 26091 tablet.cc:1722] T 665bbee2d15b4112b651844431535d0e P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:17:04.753015 24174 tablet_bootstrap.cc:492] T 6e35efdc0434429aa093fec7770782ef P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:04.754494 24174 tablet_bootstrap.cc:654] T 6e35efdc0434429aa093fec7770782ef P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.757918 24175 tablet_bootstrap.cc:492] T 2ebf55c8024c483f9ef0afe6945a02ff P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:04.758109 24175 ts_tablet_manager.cc:1367] T 2ebf55c8024c483f9ef0afe6945a02ff P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.000s
I20240502 08:17:04.758726 24180 ts_tablet_manager.cc:1888] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:04.758828 24180 log.cc:1196] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/b6d8410ba7cc4b03bb64a138b1f9bb14
I20240502 08:17:04.758937 24175 raft_consensus.cc:348] T 2ebf55c8024c483f9ef0afe6945a02ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.759150 24180 ts_tablet_manager.cc:1909] T b6d8410ba7cc4b03bb64a138b1f9bb14 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:04.759157 24175 raft_consensus.cc:374] T 2ebf55c8024c483f9ef0afe6945a02ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.759266 24175 raft_consensus.cc:729] T 2ebf55c8024c483f9ef0afe6945a02ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.759518 24175 consensus_queue.cc:260] T 2ebf55c8024c483f9ef0afe6945a02ff P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.759938 24175 ts_tablet_manager.cc:1398] T 2ebf55c8024c483f9ef0afe6945a02ff P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:04.760031 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet b6d8410ba7cc4b03bb64a138b1f9bb14 (table default.Table_b9b22cc686024294ab1843b25cb74af7 [id=2c5d4e40edf045719a60cb12646c5868]) successfully deleted
I20240502 08:17:04.760850 24175 tablet_bootstrap.cc:492] T 045b0882a40f4eaba94935d9bc90da0c P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:04.763077 24175 tablet_bootstrap.cc:654] T 045b0882a40f4eaba94935d9bc90da0c P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.769351 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet ae2b20ae3a014c85af33d5ee51b704e1
I20240502 08:17:04.771414 23604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet e34873a3f08043098f2740641b3dc41f
I20240502 08:17:04.772022 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_98da23f792224757a75f9527a604c037" } new_table_name: "default.Table_7349fc7ffdbb4fbd964d7f0da67c54af" modify_external_catalogs: true
I20240502 08:17:04.772079 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_88f82d9d71d2401da70f910b87546fe2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.772529 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88f82d9d71d2401da70f910b87546fe2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.773433 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet ae2b20ae3a014c85af33d5ee51b704e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.773762 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 79a36b64095f445ba21a6304b887a112
I20240502 08:17:04.773939 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet ed545e81a6934cf8972ff84181048156
I20240502 08:17:04.774307 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_cc84cf8bfcbb448fb07e0025acdddb22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.774706 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc84cf8bfcbb448fb07e0025acdddb22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.775200 23754 tablet_service.cc:1514] Processing DeleteTablet for tablet ed545e81a6934cf8972ff84181048156 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.775375 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 79a36b64095f445ba21a6304b887a112 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:43544
W20240502 08:17:04.775986 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2b7393897c54a1fa790ef47daa33525 consensus state has no leader
I20240502 08:17:04.775897 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_89434cc32c4f4d31af47e426eeecb2c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.776268 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89434cc32c4f4d31af47e426eeecb2c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.776350 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_299008ee28694f2395585f3ce567a641" } new_table_name: "default.Table_24c6cae05759463681b9ca845e7a04ed" modify_external_catalogs: true
I20240502 08:17:04.778455 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet ae2b20ae3a014c85af33d5ee51b704e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.781042 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet e34873a3f08043098f2740641b3dc41f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.785413 26091 tablet.cc:1722] T 55f3f09fc1f94c00a47b9087b98e221d P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240502 08:17:04.785692 23609 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 121668ea2e2943c4b61354b8c165000e
I20240502 08:17:04.786056 23609 catalog_manager.cc:6554] ReplaceTablet: tablet 121668ea2e2943c4b61354b8c165000e deleted and replaced by tablet 5ce7f3c381704fc0937bcea120d7b3d7
I20240502 08:17:04.786538 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet e34873a3f08043098f2740641b3dc41f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.786732 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet 79a36b64095f445ba21a6304b887a112 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.777884 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet ae2b20ae3a014c85af33d5ee51b704e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:04.786897 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet ed545e81a6934cf8972ff84181048156 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.788712 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet e34873a3f08043098f2740641b3dc41f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:04.789350 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 121668ea2e2943c4b61354b8c165000e with delete_type TABLET_DATA_DELETED (replaced by tablet 5ce7f3c381704fc0937bcea120d7b3d7) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.790164 24174 tablet_bootstrap.cc:492] T 6e35efdc0434429aa093fec7770782ef P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:04.790272 24174 ts_tablet_manager.cc:1367] T 6e35efdc0434429aa093fec7770782ef P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.000s
I20240502 08:17:04.790531 23760 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ebf55c8024c483f9ef0afe6945a02ff with delete_type TABLET_DATA_DELETED (replaced by tablet ed5916a2229545b4ba13515219d459cb) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.790691 23754 tablet_service.cc:1514] Processing DeleteTablet for tablet 02e6a1ba23dd4f82bc721a60c0526871 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.790843 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet 121668ea2e2943c4b61354b8c165000e with delete_type TABLET_DATA_DELETED (replaced by tablet 5ce7f3c381704fc0937bcea120d7b3d7) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.790904 24174 raft_consensus.cc:348] T 6e35efdc0434429aa093fec7770782ef P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.791067 24174 raft_consensus.cc:374] T 6e35efdc0434429aa093fec7770782ef P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.791121 24174 raft_consensus.cc:729] T 6e35efdc0434429aa093fec7770782ef P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.791291 24174 consensus_queue.cc:260] T 6e35efdc0434429aa093fec7770782ef P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.791735 24174 ts_tablet_manager.cc:1398] T 6e35efdc0434429aa093fec7770782ef P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:04.791949 24174 tablet_bootstrap.cc:492] T b97b5ba571754aac8434d0b5ceb37bd7 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:04.793313 24174 tablet_bootstrap.cc:654] T b97b5ba571754aac8434d0b5ceb37bd7 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.793685 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet 14075c65ed7a4aa78166943aa32f904f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.797525 24173 tablet_bootstrap.cc:492] T 72ee52bcc61b4816acda5b8140b63504 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:04.797904 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet 79a36b64095f445ba21a6304b887a112 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:04.797967 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a74a6eb6d214d858903b3307e09e9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:04.798040 23893 tablet_service.cc:1514] Processing DeleteTablet for tablet ed545e81a6934cf8972ff84181048156 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:04.798116 23902 tablet_service.cc:1514] Processing DeleteTablet for tablet a91e2b50321e4241847d97a558b419f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:04.799845 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet 121668ea2e2943c4b61354b8c165000e with delete_type TABLET_DATA_DELETED (replaced by tablet 5ce7f3c381704fc0937bcea120d7b3d7) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:04.774358 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_699d9f15dcfc42cf9b943d1b3967106c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.802294 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_699d9f15dcfc42cf9b943d1b3967106c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.823485 24174 tablet_bootstrap.cc:492] T b97b5ba571754aac8434d0b5ceb37bd7 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:04.823628 24174 ts_tablet_manager.cc:1367] T b97b5ba571754aac8434d0b5ceb37bd7 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.006s
I20240502 08:17:04.824353 24174 raft_consensus.cc:348] T b97b5ba571754aac8434d0b5ceb37bd7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.824541 24174 raft_consensus.cc:374] T b97b5ba571754aac8434d0b5ceb37bd7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.824587 24174 raft_consensus.cc:729] T b97b5ba571754aac8434d0b5ceb37bd7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.824786 24174 consensus_queue.cc:260] T b97b5ba571754aac8434d0b5ceb37bd7 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.826064 24174 ts_tablet_manager.cc:1398] T b97b5ba571754aac8434d0b5ceb37bd7 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:04.831734 24174 tablet_bootstrap.cc:492] T 1be95e0576c143e5a94c7aaf8d3e0625 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:04.833168 24174 tablet_bootstrap.cc:654] T 1be95e0576c143e5a94c7aaf8d3e0625 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.834825 26090 tablet.cc:1722] T 55f3f09fc1f94c00a47b9087b98e221d P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20240502 08:17:04.851441 23594 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet f5496fa258de4988b9502136f0baf779 (table default.Table_65264907568d4852a3a32e936797fda3 [id=68e36934ceb0416a86dc24c4fc50804b], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet f5496fa258de4988b9502136f0baf779 consensus state has no leader
I20240502 08:17:04.851764 24173 ts_tablet_manager.cc:1367] T 72ee52bcc61b4816acda5b8140b63504 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.132s user 0.004s sys 0.000s
I20240502 08:17:04.852618 24173 raft_consensus.cc:348] T 72ee52bcc61b4816acda5b8140b63504 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.852802 24173 raft_consensus.cc:374] T 72ee52bcc61b4816acda5b8140b63504 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.852855 24173 raft_consensus.cc:729] T 72ee52bcc61b4816acda5b8140b63504 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.853075 24173 consensus_queue.cc:260] T 72ee52bcc61b4816acda5b8140b63504 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.853765 24173 ts_tablet_manager.cc:1398] T 72ee52bcc61b4816acda5b8140b63504 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:04.860967 24173 tablet_bootstrap.cc:492] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:04.862295 26095 tablet.cc:1722] T 6088f160a13c4ff09f72bed72f69eaf8 P 149ce744feab49b5aa5bcaa14bd6aba5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:04.862800 26098 tablet.cc:1722] T 6088f160a13c4ff09f72bed72f69eaf8 P a297a6c7c2504ca68e87b24f6691ffb2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:04.863276 25012 ts_tablet_manager.cc:1888] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:04.863456 25012 log.cc:1196] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/0281c8123eab4c48bddc377f5695c6e0
I20240502 08:17:04.863921 25012 ts_tablet_manager.cc:1909] T 0281c8123eab4c48bddc377f5695c6e0 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:04.864285 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1321) took 1243 ms. Trace:
I20240502 08:17:04.864430 25012 rpcz_store.cc:276] 0502 08:17:03.620593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:03.620647 (+ 54us) service_pool.cc:225] Handling call
0502 08:17:04.574943 (+954296us) ts_tablet_manager.cc:1120] Deleting tablet 0281c8123eab4c48bddc377f5695c6e0
0502 08:17:04.574950 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:04.723114 (+148164us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:04.726117 (+ 3003us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:04.864266 (+138149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":954035}
I20240502 08:17:04.868011 24181 ts_tablet_manager.cc:1909] T e9171ebf74424c9aae5e1c8a5118f768 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:04.868983 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet e9171ebf74424c9aae5e1c8a5118f768 (table default.Table_dcbb721cb3904016a2866ef34f6bf557 [id=3711d24a9e544db98c2bda42becc3f81]) successfully deleted
I20240502 08:17:04.869279 26090 tablet.cc:1722] T 6088f160a13c4ff09f72bed72f69eaf8 P 5b89a1f0485a44b7ac1710e2369a809b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:17:04.870797 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet 0281c8123eab4c48bddc377f5695c6e0 (table default.Table_3e0123590a92473dabaacc9c158859e7 [id=d1a317371a5b43f185eb6453861e112a]) successfully deleted
I20240502 08:17:04.871111 25012 tablet_replica.cc:330] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:04.871341 25012 raft_consensus.cc:2229] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:04.874375 24173 tablet_bootstrap.cc:654] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.875283 23604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_d95b1a6e19ef46958e311e6c87b06bb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.875743 23604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d95b1a6e19ef46958e311e6c87b06bb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:17:04.878592 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 975a2ad2b32c4c3e864f0086c69b0551 consensus state has no leader
I20240502 08:17:04.881736 25012 raft_consensus.cc:2258] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.882409 25012 ts_tablet_manager.cc:1875] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:04.882750 24181 tablet_replica.cc:330] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:04.883752 24180 tablet_replica.cc:330] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:04.883988 24181 raft_consensus.cc:2229] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:04.884232 24180 raft_consensus.cc:2229] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:04.885753 24181 raft_consensus.cc:2258] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.886101 24180 raft_consensus.cc:2258] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.886672 24181 ts_tablet_manager.cc:1875] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:17:04.887331 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ef622696fe7435abe07808f5e8e918d consensus state has no leader
I20240502 08:17:04.887605 24180 ts_tablet_manager.cc:1875] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:04.888186 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_99cb7bc53f4a44ea93503add4cddf5d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.888644 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99cb7bc53f4a44ea93503add4cddf5d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.894537 24173 tablet_bootstrap.cc:492] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:04.894503 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3a2e4270c3d04bb39c94940fb2adb74a" } new_table_name: "default.Table_ffdd09c54c12402895e96ae153bd30e4" modify_external_catalogs: true
I20240502 08:17:04.894682 24173 ts_tablet_manager.cc:1367] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20240502 08:17:04.895524 24173 raft_consensus.cc:348] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.895763 24173 raft_consensus.cc:374] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.895828 24173 raft_consensus.cc:729] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.896711 24173 consensus_queue.cc:260] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.897714 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e73e0921e2154d34aef2a2f3061b8b0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:04.898002 24173 ts_tablet_manager.cc:1398] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240502 08:17:04.898151 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e73e0921e2154d34aef2a2f3061b8b0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.900096 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_c888f6a56a0f4bda8415ba9c572747e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.900645 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c888f6a56a0f4bda8415ba9c572747e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.902061 24181 ts_tablet_manager.cc:1888] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:04.902163 24181 log.cc:1196] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/9f3c00e5393c47df924bcb5dcf2a0f19
I20240502 08:17:04.902271 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_6723fdf3d7de45429c461fc29b99783f" } new_table_name: "default.Table_2fb33579180145bc89bfe4a1664595d1" modify_external_catalogs: true
I20240502 08:17:04.903492 23608 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_64f173e9aa5947069cae45a6cb8dd18f [id=427121bc94c247eaa315b6c830770d61] alter complete (version 3)
I20240502 08:17:04.907096 24173 tablet_bootstrap.cc:492] T e2792c45255c42adaeefb7aef4e25715 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:04.907150 23606 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8608727bcced4974881d8876c1e1db84" } modify_external_catalogs: true
I20240502 08:17:04.907302 23606 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_8608727bcced4974881d8876c1e1db84" } modify_external_catalogs: true
I20240502 08:17:04.909919 24173 tablet_bootstrap.cc:654] T e2792c45255c42adaeefb7aef4e25715 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.912681 24180 ts_tablet_manager.cc:1888] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:04.912803 24180 log.cc:1196] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/b2eac3bde41c4415bdb97fc53a5b395f
I20240502 08:17:04.913195 24180 ts_tablet_manager.cc:1909] T b2eac3bde41c4415bdb97fc53a5b395f P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:04.921525 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1364) took 1208 ms. Trace:
I20240502 08:17:04.921633 24180 rpcz_store.cc:276] 0502 08:17:03.713547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:03.809635 (+ 96088us) service_pool.cc:225] Handling call
0502 08:17:04.883697 (+1074062us) ts_tablet_manager.cc:1120] Deleting tablet b2eac3bde41c4415bdb97fc53a5b395f
0502 08:17:04.883709 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:04.890371 (+ 6662us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:04.912587 (+ 22216us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:04.921507 (+ 8920us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1060320}
I20240502 08:17:04.922051 24180 tablet_replica.cc:330] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:04.923130 24180 raft_consensus.cc:2229] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:04.923310 24180 raft_consensus.cc:2258] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.923892 24180 ts_tablet_manager.cc:1875] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:04.925786 24173 tablet_bootstrap.cc:492] T e2792c45255c42adaeefb7aef4e25715 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:04.925953 24173 ts_tablet_manager.cc:1367] T e2792c45255c42adaeefb7aef4e25715 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
I20240502 08:17:04.926101 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet b2eac3bde41c4415bdb97fc53a5b395f (table default.Table_2d9b44d9d9d34aa184855169eb278e94 [id=f19f154c950e4a228766d5189c345eb6]) successfully deleted
I20240502 08:17:04.926671 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ebf55c8024c483f9ef0afe6945a02ff with delete_type TABLET_DATA_DELETED (replaced by tablet ed5916a2229545b4ba13515219d459cb) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:04.927489 24173 raft_consensus.cc:348] T e2792c45255c42adaeefb7aef4e25715 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.927803 24173 raft_consensus.cc:374] T e2792c45255c42adaeefb7aef4e25715 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.927930 24173 raft_consensus.cc:729] T e2792c45255c42adaeefb7aef4e25715 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.928282 24173 consensus_queue.cc:260] T e2792c45255c42adaeefb7aef4e25715 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.928670 23609 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_353ed567df1d4ae2b174cc32bbc8e634 [id=106dd88de5eb49e89f0042beb7920736] alter complete (version 3)
I20240502 08:17:04.929773 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet 23ba091d8efb44858bf90a104c0f442f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.929461 24173 ts_tablet_manager.cc:1398] T e2792c45255c42adaeefb7aef4e25715 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:17:04.929983 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b03d759a354a39aea44fd5d4eeb089 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:45 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.930558 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet b2eac3bde41c4415bdb97fc53a5b395f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:04.930846 24173 tablet_bootstrap.cc:492] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:04.933236 24173 tablet_bootstrap.cc:654] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.933475 26001 raft_consensus.cc:482] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:04.933652 26001 raft_consensus.cc:504] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.934334 26001 leader_election.cc:288] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:04.935045 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ba091d8efb44858bf90a104c0f442f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:04.935585 23723 leader_election.cc:341] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.943287 25867 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:04.944458 25867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:04.945192 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ba091d8efb44858bf90a104c0f442f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:04.945430 23917 raft_consensus.cc:2454] T 23ba091d8efb44858bf90a104c0f442f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:04.946717 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet 23ba091d8efb44858bf90a104c0f442f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:44 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:04.947686 23723 leader_election.cc:302] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:04.948210 26002 raft_consensus.cc:2781] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:04.948315 26002 raft_consensus.cc:482] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:04.948385 26002 raft_consensus.cc:3037] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:04.950510 26002 raft_consensus.cc:504] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.951053 26002 leader_election.cc:288] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:04.951218 26002 raft_consensus.cc:482] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:04.951305 26002 raft_consensus.cc:504] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.951680 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ba091d8efb44858bf90a104c0f442f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:04.951779 26002 leader_election.cc:288] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:04.952381 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ba091d8efb44858bf90a104c0f442f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:04.952523 23917 raft_consensus.cc:3037] T 23ba091d8efb44858bf90a104c0f442f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:17:04.953814 23723 leader_election.cc:341] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.954217 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14075c65ed7a4aa78166943aa32f904f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:04.954767 23723 leader_election.cc:341] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:04.958119 26027 raft_consensus.cc:482] T b0cf1a6de3d649b984c50f45393ec0f1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:04.958241 26027 raft_consensus.cc:504] T b0cf1a6de3d649b984c50f45393ec0f1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.958598 24180 ts_tablet_manager.cc:1888] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:04.958725 24180 log.cc:1196] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/7ae9c817c7be42da8c689267c7eb5a06
I20240502 08:17:04.958753 26027 leader_election.cc:288] T b0cf1a6de3d649b984c50f45393ec0f1 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:04.959053 24180 ts_tablet_manager.cc:1909] T 7ae9c817c7be42da8c689267c7eb5a06 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:04.959183 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0cf1a6de3d649b984c50f45393ec0f1" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:04.959908 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1381) took 1149 ms. Trace:
I20240502 08:17:04.960011 24180 rpcz_store.cc:276] 0502 08:17:03.809986 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:03.816893 (+ 6907us) service_pool.cc:225] Handling call
0502 08:17:04.922011 (+1105118us) ts_tablet_manager.cc:1120] Deleting tablet 7ae9c817c7be42da8c689267c7eb5a06
0502 08:17:04.922019 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:04.937091 (+ 15072us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:04.939916 (+ 2825us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:04.959868 (+ 19952us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":838,"tablet-delete.queue_time_us":1103990}
I20240502 08:17:04.960181 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0cf1a6de3d649b984c50f45393ec0f1" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:04.960489 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 7ae9c817c7be42da8c689267c7eb5a06 (table default.Table_d718a36474964fc09d2813f1210c5ce2 [id=d9f38d2a946c498f86eddd3d3277464d]) successfully deleted
I20240502 08:17:04.962162 23993 leader_election.cc:302] T b0cf1a6de3d649b984c50f45393ec0f1 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:04.962272 24173 tablet_bootstrap.cc:492] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:04.962406 24173 ts_tablet_manager.cc:1367] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20240502 08:17:04.962616 26027 raft_consensus.cc:2726] T b0cf1a6de3d649b984c50f45393ec0f1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:04.963016 24173 raft_consensus.cc:348] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.963176 24173 raft_consensus.cc:374] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.963217 24173 raft_consensus.cc:729] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.963366 24173 consensus_queue.cc:260] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:04.963810 24173 ts_tablet_manager.cc:1398] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:04.964035 24173 tablet_bootstrap.cc:492] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:04.965653 24173 tablet_bootstrap.cc:654] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.966588 24180 tablet_replica.cc:330] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:04.967772 24180 raft_consensus.cc:2229] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:04.967923 24180 raft_consensus.cc:2258] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:04.968374 24180 ts_tablet_manager.cc:1875] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240502 08:17:04.971082 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3519b9c0ff2546c882ac9a92add72b40 consensus state has no leader
I20240502 08:17:04.974287 24174 tablet_bootstrap.cc:492] T 1be95e0576c143e5a94c7aaf8d3e0625 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:04.974421 24174 ts_tablet_manager.cc:1367] T 1be95e0576c143e5a94c7aaf8d3e0625 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.143s user 0.047s sys 0.016s
I20240502 08:17:04.975108 25867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:04.975078 24174 raft_consensus.cc:348] T 1be95e0576c143e5a94c7aaf8d3e0625 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.975230 24174 raft_consensus.cc:374] T 1be95e0576c143e5a94c7aaf8d3e0625 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.975268 24174 raft_consensus.cc:729] T 1be95e0576c143e5a94c7aaf8d3e0625 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.975464 24174 consensus_queue.cc:260] T 1be95e0576c143e5a94c7aaf8d3e0625 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.976259 24174 ts_tablet_manager.cc:1398] T 1be95e0576c143e5a94c7aaf8d3e0625 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20240502 08:17:04.977005 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3f3de0cffda4ee2b43a88470d2e9795 consensus state has no leader
I20240502 08:17:04.977037 24174 tablet_bootstrap.cc:492] T 2085ced52fff46ebaa98324cbb2d8c11 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:04.978432 24174 tablet_bootstrap.cc:654] T 2085ced52fff46ebaa98324cbb2d8c11 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.981488 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1159b00d82f4483c8209c879e2d3c06a" } modify_external_catalogs: true
I20240502 08:17:04.981693 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1159b00d82f4483c8209c879e2d3c06a" } modify_external_catalogs: true
I20240502 08:17:04.983075 24173 tablet_bootstrap.cc:492] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:04.982998 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_0632d2e7212e481daaf9b7195c423de8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:04.983179 24173 ts_tablet_manager.cc:1367] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
W20240502 08:17:04.983479 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0632d2e7212e481daaf9b7195c423de8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.983764 24173 raft_consensus.cc:348] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.983932 24173 raft_consensus.cc:374] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:04.983978 24173 raft_consensus.cc:729] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:04.984139 24173 consensus_queue.cc:260] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:04.984542 24173 ts_tablet_manager.cc:1398] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:04.984764 24173 tablet_bootstrap.cc:492] T 6ee764398b364d029ae7f0f575a30d64 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:04.985375 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9f3ea2fdcdc2477b89d509e868ffe4de" } modify_external_catalogs: true
I20240502 08:17:04.985518 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9f3ea2fdcdc2477b89d509e868ffe4de" } modify_external_catalogs: true
I20240502 08:17:04.985559 23601 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9dff2385175e41de9631cd27fbd848cb" } modify_external_catalogs: true
I20240502 08:17:04.985886 23601 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_9dff2385175e41de9631cd27fbd848cb" } modify_external_catalogs: true
I20240502 08:17:04.985759 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1ce14ae5f46f4278ad1667cf245cd2ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:04.986608 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ce14ae5f46f4278ad1667cf245cd2ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:04.986960 24173 tablet_bootstrap.cc:654] T 6ee764398b364d029ae7f0f575a30d64 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.987600 23605 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2e569d61a7b94fb89013bed2831ec5c1" } modify_external_catalogs: true
I20240502 08:17:04.987728 23605 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_2e569d61a7b94fb89013bed2831ec5c1" } modify_external_catalogs: true
I20240502 08:17:04.994971 23606 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet efe9d0674e1547679895ef48ed9d53ce
I20240502 08:17:04.996563 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_42afc33a0fee436b9658e8a3e51eedfd" } new_table_name: "default.Table_3a718dec171546839c4b90619666bb74" modify_external_catalogs: true
I20240502 08:17:05.001456 24052 raft_consensus.cc:3037] T a4d96cc17d9b4bcf8ce7c4414a63f9cb P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.002606 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 4c395a29c7a1485eab39ca107dd91fca
I20240502 08:17:05.003203 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c395a29c7a1485eab39ca107dd91fca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:05.003710 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c395a29c7a1485eab39ca107dd91fca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:05.005295 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c395a29c7a1485eab39ca107dd91fca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:05.005466 26001 raft_consensus.cc:482] T b97b5ba571754aac8434d0b5ceb37bd7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.005952 26001 raft_consensus.cc:504] T b97b5ba571754aac8434d0b5ceb37bd7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.006614 26001 leader_election.cc:288] T b97b5ba571754aac8434d0b5ceb37bd7 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:05.007894 24180 ts_tablet_manager.cc:1888] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:05.008380 24180 log.cc:1196] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/a91e2b50321e4241847d97a558b419f2
I20240502 08:17:05.008778 24180 ts_tablet_manager.cc:1909] T a91e2b50321e4241847d97a558b419f2 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:05.010776 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97b5ba571754aac8434d0b5ceb37bd7" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:05.011655 24173 tablet_bootstrap.cc:492] T 6ee764398b364d029ae7f0f575a30d64 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:05.011821 24173 ts_tablet_manager.cc:1367] T 6ee764398b364d029ae7f0f575a30d64 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.002s
I20240502 08:17:05.012485 24173 raft_consensus.cc:348] T 6ee764398b364d029ae7f0f575a30d64 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.012699 24173 raft_consensus.cc:374] T 6ee764398b364d029ae7f0f575a30d64 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.012768 24173 raft_consensus.cc:729] T 6ee764398b364d029ae7f0f575a30d64 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.013031 24173 consensus_queue.cc:260] T 6ee764398b364d029ae7f0f575a30d64 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.013178 23917 raft_consensus.cc:2454] T 23ba091d8efb44858bf90a104c0f442f P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 1.
I20240502 08:17:05.013597 24173 ts_tablet_manager.cc:1398] T 6ee764398b364d029ae7f0f575a30d64 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:05.014066 24173 tablet_bootstrap.cc:492] T 981b7a7292884b1fb48af99db6e762c0 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:05.016120 24173 tablet_bootstrap.cc:654] T 981b7a7292884b1fb48af99db6e762c0 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:05.016551 26052 raft_consensus.cc:482] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.016683 26052 raft_consensus.cc:504] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.017266 26052 leader_election.cc:288] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:17:05.017277 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6e21621360404c8aaa08ac5646edcc95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:05.017810 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e21621360404c8aaa08ac5646edcc95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:05.018859 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae9c817c7be42da8c689267c7eb5a06" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:05.019635 23861 leader_election.cc:341] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.022184 24174 tablet_bootstrap.cc:492] T 2085ced52fff46ebaa98324cbb2d8c11 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:05.022330 24174 ts_tablet_manager.cc:1367] T 2085ced52fff46ebaa98324cbb2d8c11 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.004s
I20240502 08:17:05.023029 24174 raft_consensus.cc:348] T 2085ced52fff46ebaa98324cbb2d8c11 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.023187 24174 raft_consensus.cc:374] T 2085ced52fff46ebaa98324cbb2d8c11 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.023236 24174 raft_consensus.cc:729] T 2085ced52fff46ebaa98324cbb2d8c11 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.023413 24174 consensus_queue.cc:260] T 2085ced52fff46ebaa98324cbb2d8c11 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.023650 26011 raft_consensus.cc:482] T 6e35efdc0434429aa093fec7770782ef P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.023764 26011 raft_consensus.cc:504] T 6e35efdc0434429aa093fec7770782ef P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:17:05.023895 23723 leader_election.cc:341] T b97b5ba571754aac8434d0b5ceb37bd7 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.024091 23723 leader_election.cc:302] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:05.024230 26011 leader_election.cc:288] T 6e35efdc0434429aa093fec7770782ef P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:05.024374 26011 raft_consensus.cc:2781] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:05.024469 26011 raft_consensus.cc:686] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Running, Role: LEADER
I20240502 08:17:05.024684 26011 consensus_queue.cc:237] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:17:05.025949 23858 leader_election.cc:341] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 7ae9c817c7be42da8c689267c7eb5a06
I20240502 08:17:05.026058 23858 leader_election.cc:302] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:05.026360 25649 raft_consensus.cc:2726] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:05.028107 24174 ts_tablet_manager.cc:1398] T 2085ced52fff46ebaa98324cbb2d8c11 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:17:05.030359 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97b5ba571754aac8434d0b5ceb37bd7" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:05.031092 24051 raft_consensus.cc:3037] T b0cf1a6de3d649b984c50f45393ec0f1 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.032191 24173 tablet_bootstrap.cc:492] T 981b7a7292884b1fb48af99db6e762c0 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:05.032303 24173 ts_tablet_manager.cc:1367] T 981b7a7292884b1fb48af99db6e762c0 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20240502 08:17:05.032989 24173 raft_consensus.cc:348] T 981b7a7292884b1fb48af99db6e762c0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.033187 24173 raft_consensus.cc:374] T 981b7a7292884b1fb48af99db6e762c0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.033237 24173 raft_consensus.cc:729] T 981b7a7292884b1fb48af99db6e762c0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.033440 24173 consensus_queue.cc:260] T 981b7a7292884b1fb48af99db6e762c0 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.034089 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e35efdc0434429aa093fec7770782ef" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:05.034713 24173 ts_tablet_manager.cc:1398] T 981b7a7292884b1fb48af99db6e762c0 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:05.035046 24173 tablet_bootstrap.cc:492] T a91e2b50321e4241847d97a558b419f2 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:05.037992 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_aed335510eae46a89e254b954ca8ad08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:05.038579 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aed335510eae46a89e254b954ca8ad08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:17:05.039714 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aec17736da3b43d0b79091adb79365a1 consensus state has no leader
I20240502 08:17:05.039831 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 63afd6e4fee24edeaba1209105f1c3e6
I20240502 08:17:05.040078 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_edd6a0d33e854c2487fc0df22cda4524"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:05.040488 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edd6a0d33e854c2487fc0df22cda4524 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:05.040884 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e35efdc0434429aa093fec7770782ef" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:05.041860 23890 tablet_service.cc:1514] Processing DeleteTablet for tablet 63afd6e4fee24edeaba1209105f1c3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:05.043248 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet 63afd6e4fee24edeaba1209105f1c3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:05.044967 24173 tablet_bootstrap.cc:654] T a91e2b50321e4241847d97a558b419f2 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:05.061012 24173 tablet_bootstrap.cc:492] T a91e2b50321e4241847d97a558b419f2 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:05.061175 24173 ts_tablet_manager.cc:1367] T a91e2b50321e4241847d97a558b419f2 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.002s
I20240502 08:17:05.062727 24173 raft_consensus.cc:348] T a91e2b50321e4241847d97a558b419f2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.062919 24173 raft_consensus.cc:374] T a91e2b50321e4241847d97a558b419f2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.062965 24173 raft_consensus.cc:729] T a91e2b50321e4241847d97a558b419f2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.063162 24173 consensus_queue.cc:260] T a91e2b50321e4241847d97a558b419f2 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.063588 24173 ts_tablet_manager.cc:1398] T a91e2b50321e4241847d97a558b419f2 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:05.063832 24173 tablet_bootstrap.cc:492] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:05.067620 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 63afd6e4fee24edeaba1209105f1c3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:51782
W20240502 08:17:05.073660 23723 leader_election.cc:341] T 6e35efdc0434429aa093fec7770782ef P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:17:05.074995 23723 leader_election.cc:341] T b97b5ba571754aac8434d0b5ceb37bd7 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.075111 23723 leader_election.cc:302] T b97b5ba571754aac8434d0b5ceb37bd7 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:05.075326 26011 raft_consensus.cc:2726] T b97b5ba571754aac8434d0b5ceb37bd7 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:17:05.075793 23723 leader_election.cc:341] T 6e35efdc0434429aa093fec7770782ef P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.075896 23723 leader_election.cc:302] T 6e35efdc0434429aa093fec7770782ef P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:05.076030 24173 tablet_bootstrap.cc:654] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:05.076141 26011 raft_consensus.cc:2726] T 6e35efdc0434429aa093fec7770782ef P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:05.077247 26011 raft_consensus.cc:482] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.077406 26011 raft_consensus.cc:504] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.078142 26011 leader_election.cc:288] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:05.079958 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "255f69f56d344ce7805eca0bfdf8a063" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:05.080410 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_56944338c8ef4b8a98aff14eef9fa611" } new_table_name: "default.Table_d0660289d487485b88ae54235771b52e" modify_external_catalogs: true
I20240502 08:17:05.081092 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "255f69f56d344ce7805eca0bfdf8a063" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:05.087823 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet abcc0296b3c941fbad86bafa8f7965fc consensus state has no leader
I20240502 08:17:05.089159 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_1692280f84c44944b308c28c5918f6f8" } new_table_name: "default.Table_906777bcd18b416baa4a868713742b18" modify_external_catalogs: true
I20240502 08:17:05.089483 24173 tablet_bootstrap.cc:492] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:05.089617 24173 ts_tablet_manager.cc:1367] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.002s
W20240502 08:17:05.093927 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1cf9fc9e13d840fdbbf2fe498410c288 consensus state has no leader
I20240502 08:17:05.094853 24173 raft_consensus.cc:348] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.095029 24173 raft_consensus.cc:374] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.095072 24173 raft_consensus.cc:729] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.095269 24173 consensus_queue.cc:260] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.095712 24173 ts_tablet_manager.cc:1398] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 08:17:05.095970 24173 tablet_bootstrap.cc:492] T 91a3455e5931488da3134d6b5c18aef2 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:05.098567 24173 tablet_bootstrap.cc:654] T 91a3455e5931488da3134d6b5c18aef2 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:05.099172 26052 raft_consensus.cc:482] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.099365 26052 raft_consensus.cc:504] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.100538 26052 leader_election.cc:288] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
I20240502 08:17:05.100790 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a91e2b50321e4241847d97a558b419f2" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:05.101033 24051 raft_consensus.cc:2454] T a91e2b50321e4241847d97a558b419f2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:05.101470 23861 leader_election.cc:302] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:17:05.101927 26052 raft_consensus.cc:2781] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:05.102047 26052 raft_consensus.cc:482] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:05.102180 26052 raft_consensus.cc:3037] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.105193 26052 raft_consensus.cc:504] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.106734 26027 raft_consensus.cc:482] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.106925 26027 raft_consensus.cc:504] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.108533 26027 leader_election.cc:288] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:17:05.109166 23993 leader_election.cc:341] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: d16110e1b1664802a98182c73f1e41d3
W20240502 08:17:05.109848 23996 leader_election.cc:341] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Not found: Tablet not found: d16110e1b1664802a98182c73f1e41d3
I20240502 08:17:05.109947 23996 leader_election.cc:302] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:05.110548 26027 raft_consensus.cc:2726] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:05.113619 26027 raft_consensus.cc:482] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.113792 26027 raft_consensus.cc:504] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.115116 26027 leader_election.cc:288] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:17:05.115736 23993 leader_election.cc:341] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 0281c8123eab4c48bddc377f5695c6e0
W20240502 08:17:05.116264 23996 leader_election.cc:341] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Not found: Tablet not found: 0281c8123eab4c48bddc377f5695c6e0
I20240502 08:17:05.116421 23996 leader_election.cc:302] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:05.117326 26027 raft_consensus.cc:2726] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:05.126175 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a91e2b50321e4241847d97a558b419f2" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
W20240502 08:17:05.126828 23723 leader_election.cc:341] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:17:05.129882 23723 leader_election.cc:341] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.130939 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a91e2b50321e4241847d97a558b419f2" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
I20240502 08:17:05.131168 24051 raft_consensus.cc:3037] T a91e2b50321e4241847d97a558b419f2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.112115 25012 ts_tablet_manager.cc:1888] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:05.133349 25012 log.cc:1196] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/e9171ebf74424c9aae5e1c8a5118f768
I20240502 08:17:05.133816 25012 ts_tablet_manager.cc:1909] T e9171ebf74424c9aae5e1c8a5118f768 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:05.137804 23723 leader_election.cc:302] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:05.124080 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14075c65ed7a4aa78166943aa32f904f" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:05.138846 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1316) took 1519 ms. Trace:
I20240502 08:17:05.139349 26011 raft_consensus.cc:2726] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:05.140703 24051 raft_consensus.cc:2454] T a91e2b50321e4241847d97a558b419f2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:05.143249 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_79d6fd82fcf8426b8b0eb1a3fe0d00f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:05.143769 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79d6fd82fcf8426b8b0eb1a3fe0d00f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:17:05.144980 23723 leader_election.cc:341] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.145076 23723 leader_election.cc:302] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:05.145277 26011 raft_consensus.cc:2726] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:05.153601 25012 rpcz_store.cc:276] 0502 08:17:03.619621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:03.621630 (+ 2009us) service_pool.cc:225] Handling call
0502 08:17:04.871059 (+1249429us) ts_tablet_manager.cc:1120] Deleting tablet e9171ebf74424c9aae5e1c8a5118f768
0502 08:17:04.871069 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:05.051103 (+180034us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:05.111987 (+ 60884us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:05.138830 (+ 26843us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1249302}
I20240502 08:17:05.153908 25012 tablet_replica.cc:330] T fddac683eda14ec1a0140926380f2c07 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:05.154079 25012 raft_consensus.cc:2229] T fddac683eda14ec1a0140926380f2c07 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.154209 25012 raft_consensus.cc:2258] T fddac683eda14ec1a0140926380f2c07 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.154474 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1383) took 1324 ms. Trace:
I20240502 08:17:05.154543 24180 rpcz_store.cc:276] 0502 08:17:03.830134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:03.830636 (+ 502us) service_pool.cc:225] Handling call
0502 08:17:04.966538 (+1135902us) ts_tablet_manager.cc:1120] Deleting tablet a91e2b50321e4241847d97a558b419f2
0502 08:17:04.966548 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:04.988263 (+ 21715us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:05.006616 (+ 18353us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:05.154455 (+147839us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1109849}
I20240502 08:17:05.154651 25012 ts_tablet_manager.cc:1875] T fddac683eda14ec1a0140926380f2c07 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:05.141752 26052 leader_election.cc:288] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163), a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709)
W20240502 08:17:05.145998 23858 leader_election.cc:341] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Illegal state: Tablet not RUNNING: SHUTDOWN
I20240502 08:17:05.154915 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1386) took 1156 ms. Trace:
I20240502 08:17:05.154979 24180 rpcz_store.cc:276] 0502 08:17:03.998595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:03.998660 (+ 65us) service_pool.cc:225] Handling call
0502 08:17:05.154774 (+1156114us) ts_tablet_manager.cc:1120] Deleting tablet 3449d7d1e8044b6cbf20766a5597e5ab
0502 08:17:05.154782 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:05.154900 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1155881}
I20240502 08:17:05.155109 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet e9171ebf74424c9aae5e1c8a5118f768 (table default.Table_dcbb721cb3904016a2866ef34f6bf557 [id=3711d24a9e544db98c2bda42becc3f81]) successfully deleted
I20240502 08:17:05.155284 24180 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43544 (request call id 1387) took 1124 ms. Trace:
I20240502 08:17:05.155342 24180 rpcz_store.cc:276] 0502 08:17:04.030640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:04.030702 (+ 62us) service_pool.cc:225] Handling call
0502 08:17:05.155192 (+1124490us) ts_tablet_manager.cc:1120] Deleting tablet 0e9de33c88984befb056647b4c31de6f
0502 08:17:05.155200 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:05.155271 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":998144}
I20240502 08:17:05.155696 23861 leader_election.cc:302] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:17:05.155987 26052 raft_consensus.cc:2781] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:05.156013 24180 tablet_replica.cc:330] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:05.156090 26052 raft_consensus.cc:686] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:17:05.156180 24180 raft_consensus.cc:2229] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.156311 24180 raft_consensus.cc:2258] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.156327 26052 consensus_queue.cc:237] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.156711 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet a91e2b50321e4241847d97a558b419f2 (table default.Table_5ad32437311f45d3a50a7fe4b134adb9 [id=243d930b07314a519d68fa40bd636226]) successfully deleted
W20240502 08:17:05.157002 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 3449d7d1e8044b6cbf20766a5597e5ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3449d7d1e8044b6cbf20766a5597e5ab already in progress: creating tablet
W20240502 08:17:05.157155 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 0e9de33c88984befb056647b4c31de6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e9de33c88984befb056647b4c31de6f already in progress: creating tablet
W20240502 08:17:05.157282 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 4e3570787d8242338d1ce01fb0299eb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e3570787d8242338d1ce01fb0299eb2 already in progress: creating tablet
W20240502 08:17:05.157414 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet d1218fd56b8a4e88827f275f952ad782 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1218fd56b8a4e88827f275f952ad782 already in progress: creating tablet
W20240502 08:17:05.157580 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 7a23c06cb3704a7b83143a02efe2a8b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a23c06cb3704a7b83143a02efe2a8b3 already in progress: creating tablet
I20240502 08:17:05.153471 24175 tablet_bootstrap.cc:492] T 045b0882a40f4eaba94935d9bc90da0c P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:05.158435 24175 ts_tablet_manager.cc:1367] T 045b0882a40f4eaba94935d9bc90da0c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.398s user 0.020s sys 0.036s
W20240502 08:17:05.158691 23858 leader_election.cc:341] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: a91e2b50321e4241847d97a558b419f2
I20240502 08:17:05.160065 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_5dc37698aa0741cc8f5292288f5e4532"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:05.160347 24174 tablet_bootstrap.cc:492] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
W20240502 08:17:05.160545 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5dc37698aa0741cc8f5292288f5e4532 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:05.162127 24180 ts_tablet_manager.cc:1875] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:05.167665 24174 tablet_bootstrap.cc:654] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:05.169191 24180 ts_tablet_manager.cc:1888] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:05.169339 24180 log.cc:1196] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/255f69f56d344ce7805eca0bfdf8a063
I20240502 08:17:05.169716 24180 ts_tablet_manager.cc:1909] T 255f69f56d344ce7805eca0bfdf8a063 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:05.170792 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 255f69f56d344ce7805eca0bfdf8a063 (table default.Table_8bbe9f5123cc4dacb9fd2749d84f9988 [id=6c492ddfa9aa4d53b14c0cb7c0e116f2]) successfully deleted
W20240502 08:17:05.171412 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 0001b9d68e0a4b21a9bfd584dcdf99ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0001b9d68e0a4b21a9bfd584dcdf99ea already in progress: creating tablet
W20240502 08:17:05.171980 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 0fb9750041a44138bcf463da241e5e3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fb9750041a44138bcf463da241e5e3a already in progress: creating tablet
W20240502 08:17:05.172807 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 81a8101212df4f3e88bd322c23c17fd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81a8101212df4f3e88bd322c23c17fd1 already in progress: creating tablet
I20240502 08:17:05.173243 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_cb97a20720ec4b2384ff5373d1629576"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:05.173621 26027 raft_consensus.cc:482] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 08:17:05.173790 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb97a20720ec4b2384ff5373d1629576 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:05.173756 26027 raft_consensus.cc:504] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.175125 26027 leader_election.cc:288] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:05.175395 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_71806954494040a98a8f1a388a869a66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:05.175606 23993 leader_election.cc:341] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: bb4b4a4d6a364de8af3b54d99a0ae138
W20240502 08:17:05.175869 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71806954494040a98a8f1a388a869a66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:17:05.176352 23996 leader_election.cc:341] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Not found: Tablet not found: bb4b4a4d6a364de8af3b54d99a0ae138
I20240502 08:17:05.176456 23996 leader_election.cc:302] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:05.176936 26027 raft_consensus.cc:2726] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:05.176996 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5ffe8637fd9b4cd3b5b5aa0eb542b267" } new_table_name: "default.Table_3a05000b55904d2b9d561bc7883c2e98" modify_external_catalogs: true
I20240502 08:17:05.178697 26011 raft_consensus.cc:482] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.178820 26011 raft_consensus.cc:504] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.179378 26011 leader_election.cc:288] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:05.180141 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_e63bcdd98eae4bef904eb59804455320"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:05.180567 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e63bcdd98eae4bef904eb59804455320 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:05.181567 23602 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_e883d2bea1d049fea7b1486d4de57d81" } new_table_name: "default.Table_f6c9d7dc691c487cba1d0d744d3104d8" modify_external_catalogs: true
I20240502 08:17:05.182608 26011 raft_consensus.cc:482] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.182715 26011 raft_consensus.cc:504] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.183180 26011 leader_election.cc:288] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:17:05.185058 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet cc4c8aea662149148032a1f13adddc5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc4c8aea662149148032a1f13adddc5f already in progress: creating tablet
W20240502 08:17:05.185608 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 338fa14b1e004c36accccb3b671103c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 338fa14b1e004c36accccb3b671103c1 already in progress: creating tablet
W20240502 08:17:05.188020 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 069ee3844b514038a6cdf9313766b702 consensus state has no leader
I20240502 08:17:05.189875 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebf55c8024c483f9ef0afe6945a02ff" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:05.190651 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b03d759a354a39aea44fd5d4eeb089" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:05.191295 23723 leader_election.cc:341] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:17:05.191535 23723 leader_election.cc:341] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:17:05.192854 23602 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8d8a96668e14c088de6c82d549d35ec consensus state has no leader
I20240502 08:17:05.185985 24175 raft_consensus.cc:348] T 045b0882a40f4eaba94935d9bc90da0c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.189359 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebf55c8024c483f9ef0afe6945a02ff" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:05.195163 23917 raft_consensus.cc:2454] T 2ebf55c8024c483f9ef0afe6945a02ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a297a6c7c2504ca68e87b24f6691ffb2 in term 0.
I20240502 08:17:05.190451 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b03d759a354a39aea44fd5d4eeb089" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:05.195008 24175 raft_consensus.cc:374] T 045b0882a40f4eaba94935d9bc90da0c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.195775 24175 raft_consensus.cc:729] T 045b0882a40f4eaba94935d9bc90da0c P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.196117 24175 consensus_queue.cc:260] T 045b0882a40f4eaba94935d9bc90da0c P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.196852 24175 ts_tablet_manager.cc:1398] T 045b0882a40f4eaba94935d9bc90da0c P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.038s user 0.001s sys 0.001s
I20240502 08:17:05.197000 26027 raft_consensus.cc:482] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.197165 26027 raft_consensus.cc:504] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:17:05.198565 23993 leader_election.cc:341] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 3a74a6eb6d214d858903b3307e09e9e6
I20240502 08:17:05.198706 26029 raft_consensus.cc:482] T 6ee764398b364d029ae7f0f575a30d64 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.198792 26029 raft_consensus.cc:504] T 6ee764398b364d029ae7f0f575a30d64 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.199218 26029 leader_election.cc:288] T 6ee764398b364d029ae7f0f575a30d64 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:05.200050 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee764398b364d029ae7f0f575a30d64" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:05.200605 26011 raft_consensus.cc:482] T 1be95e0576c143e5a94c7aaf8d3e0625 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.200706 26011 raft_consensus.cc:504] T 1be95e0576c143e5a94c7aaf8d3e0625 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.201117 26011 leader_election.cc:288] T 1be95e0576c143e5a94c7aaf8d3e0625 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:05.201879 23602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_33f2ee613cae4e848bb987e9041391c1" } modify_external_catalogs: true
I20240502 08:17:05.202008 23602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_33f2ee613cae4e848bb987e9041391c1" } modify_external_catalogs: true
I20240502 08:17:05.202975 26027 leader_election.cc:288] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:05.203637 26011 raft_consensus.cc:482] T 19e4cd31e7774546a54704ccd405ffe6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.203730 26011 raft_consensus.cc:504] T 19e4cd31e7774546a54704ccd405ffe6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.204106 26011 leader_election.cc:288] T 19e4cd31e7774546a54704ccd405ffe6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:05.204794 23607 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cbac61918e774f599192caeeaef51577" } modify_external_catalogs: true
I20240502 08:17:05.204921 23607 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_cbac61918e774f599192caeeaef51577" } modify_external_catalogs: true
I20240502 08:17:05.202862 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a74a6eb6d214d858903b3307e09e9e6" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:05.205785 23996 leader_election.cc:302] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:05.206185 26027 raft_consensus.cc:2726] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:05.203202 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee764398b364d029ae7f0f575a30d64" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:05.206553 23916 raft_consensus.cc:2379] T 6ee764398b364d029ae7f0f575a30d64 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 149ce744feab49b5aa5bcaa14bd6aba5 in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:17:05.206925 23996 leader_election.cc:302] T 6ee764398b364d029ae7f0f575a30d64 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:05.207346 26027 raft_consensus.cc:3037] T 6ee764398b364d029ae7f0f575a30d64 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:17:05.208359 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet ae2b20ae3a014c85af33d5ee51b704e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae2b20ae3a014c85af33d5ee51b704e1 already in progress: creating tablet
W20240502 08:17:05.208961 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet ed545e81a6934cf8972ff84181048156 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed545e81a6934cf8972ff84181048156 already in progress: creating tablet
W20240502 08:17:05.209455 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 79a36b64095f445ba21a6304b887a112 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79a36b64095f445ba21a6304b887a112 already in progress: creating tablet
I20240502 08:17:05.210242 24175 tablet_bootstrap.cc:492] T 14075c65ed7a4aa78166943aa32f904f P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:05.210851 26027 raft_consensus.cc:2726] T 6ee764398b364d029ae7f0f575a30d64 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:05.212244 24175 tablet_bootstrap.cc:654] T 14075c65ed7a4aa78166943aa32f904f P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
W20240502 08:17:05.212900 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet e34873a3f08043098f2740641b3dc41f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e34873a3f08043098f2740641b3dc41f already in progress: creating tablet
I20240502 08:17:05.214248 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be95e0576c143e5a94c7aaf8d3e0625" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:05.215911 26011 raft_consensus.cc:482] T 2085ced52fff46ebaa98324cbb2d8c11 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.216045 26011 raft_consensus.cc:504] T 2085ced52fff46ebaa98324cbb2d8c11 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.216573 26011 leader_election.cc:288] T 2085ced52fff46ebaa98324cbb2d8c11 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:05.217058 23723 leader_election.cc:302] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:05.217191 24180 tablet_replica.cc:330] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:05.217438 24180 raft_consensus.cc:2229] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.218403 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e4cd31e7774546a54704ccd405ffe6" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:05.219851 26011 raft_consensus.cc:2684] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Received pre-election callback for term 1 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20240502 08:17:05.213532 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be95e0576c143e5a94c7aaf8d3e0625" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:05.218048 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e4cd31e7774546a54704ccd405ffe6" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:05.220254 24180 raft_consensus.cc:2258] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240502 08:17:05.219285 23723 leader_election.cc:341] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.220829 23723 leader_election.cc:302] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:05.221602 24180 ts_tablet_manager.cc:1875] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:05.222483 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2085ced52fff46ebaa98324cbb2d8c11" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:05.221812 26011 raft_consensus.cc:2726] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:17:05.225384 23723 leader_election.cc:341] T 1be95e0576c143e5a94c7aaf8d3e0625 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.225749 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2085ced52fff46ebaa98324cbb2d8c11" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:05.226202 23723 leader_election.cc:341] T 19e4cd31e7774546a54704ccd405ffe6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.226650 24173 tablet_bootstrap.cc:492] T 91a3455e5931488da3134d6b5c18aef2 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:05.226783 24173 ts_tablet_manager.cc:1367] T 91a3455e5931488da3134d6b5c18aef2 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.131s user 0.015s sys 0.050s
W20240502 08:17:05.227020 23723 leader_election.cc:341] T 2085ced52fff46ebaa98324cbb2d8c11 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:17:05.227288 23723 leader_election.cc:341] T 1be95e0576c143e5a94c7aaf8d3e0625 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.227422 23723 leader_election.cc:302] T 1be95e0576c143e5a94c7aaf8d3e0625 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:05.227659 26011 raft_consensus.cc:2726] T 1be95e0576c143e5a94c7aaf8d3e0625 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:05.227574 24173 raft_consensus.cc:348] T 91a3455e5931488da3134d6b5c18aef2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.227751 24173 raft_consensus.cc:374] T 91a3455e5931488da3134d6b5c18aef2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.227800 24173 raft_consensus.cc:729] T 91a3455e5931488da3134d6b5c18aef2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
W20240502 08:17:05.228108 23723 leader_election.cc:341] T 19e4cd31e7774546a54704ccd405ffe6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.229617 23723 leader_election.cc:302] T 19e4cd31e7774546a54704ccd405ffe6 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:05.229869 26011 raft_consensus.cc:2726] T 19e4cd31e7774546a54704ccd405ffe6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:17:05.230239 23723 leader_election.cc:341] T 2085ced52fff46ebaa98324cbb2d8c11 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.231222 23723 leader_election.cc:302] T 2085ced52fff46ebaa98324cbb2d8c11 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:05.231483 26011 raft_consensus.cc:2726] T 2085ced52fff46ebaa98324cbb2d8c11 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:05.232097 24173 consensus_queue.cc:260] T 91a3455e5931488da3134d6b5c18aef2 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.233175 24173 ts_tablet_manager.cc:1398] T 91a3455e5931488da3134d6b5c18aef2 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:17:05.234102 24173 tablet_bootstrap.cc:492] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
W20240502 08:17:05.236136 23858 consensus_peers.cc:485] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a74a6eb6d214d858903b3307e09e9e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:05.237495 24180 ts_tablet_manager.cc:1888] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:05.237524 24173 tablet_bootstrap.cc:654] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:05.237690 24180 log.cc:1196] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/2ebf55c8024c483f9ef0afe6945a02ff
I20240502 08:17:05.238018 24180 ts_tablet_manager.cc:1909] T 2ebf55c8024c483f9ef0afe6945a02ff P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:05.239048 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 2ebf55c8024c483f9ef0afe6945a02ff (table default.Table_6cde1d82c6b1472a843c94cccec9ee6c [id=00551fb59aa04e23a8c7cb96f413164d]) successfully deleted
I20240502 08:17:05.239437 24180 tablet_replica.cc:330] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:05.239655 24180 raft_consensus.cc:2229] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.239846 24180 raft_consensus.cc:2258] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.240321 24180 ts_tablet_manager.cc:1875] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:05.242314 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_3e28eee4df6d488aa6bf8cab33b8a5a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:05.245826 23603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5904e9a9ca9f4e9eabf14ec7e39e9062" } modify_external_catalogs: true
I20240502 08:17:05.245999 23603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_5904e9a9ca9f4e9eabf14ec7e39e9062" } modify_external_catalogs: true
I20240502 08:17:05.246603 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7eb01c98aa404982a046162942c74422"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:05.249142 24175 tablet_bootstrap.cc:492] T 14075c65ed7a4aa78166943aa32f904f P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:05.249167 24173 tablet_bootstrap.cc:492] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:05.249264 24173 ts_tablet_manager.cc:1367] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240502 08:17:05.249302 24175 ts_tablet_manager.cc:1367] T 14075c65ed7a4aa78166943aa32f904f P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.001s
I20240502 08:17:05.250314 24175 raft_consensus.cc:348] T 14075c65ed7a4aa78166943aa32f904f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.250411 24173 raft_consensus.cc:348] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.250573 24173 raft_consensus.cc:374] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.250574 24175 raft_consensus.cc:374] T 14075c65ed7a4aa78166943aa32f904f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.250625 24173 raft_consensus.cc:729] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.250646 24175 raft_consensus.cc:729] T 14075c65ed7a4aa78166943aa32f904f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.250820 24173 consensus_queue.cc:260] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.251612 24173 ts_tablet_manager.cc:1398] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:05.252265 24173 tablet_bootstrap.cc:492] T 255f69f56d344ce7805eca0bfdf8a063 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:05.252260 24180 ts_tablet_manager.cc:1888] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:05.252511 24180 log.cc:1196] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/02e6a1ba23dd4f82bc721a60c0526871
I20240502 08:17:05.253140 24180 ts_tablet_manager.cc:1909] T 02e6a1ba23dd4f82bc721a60c0526871 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:05.253803 24173 tablet_bootstrap.cc:654] T 255f69f56d344ce7805eca0bfdf8a063 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:05.254546 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 02e6a1ba23dd4f82bc721a60c0526871 (table default.Table_91aaef5fd943406aa3548482115afc65 [id=4a26d823fb2e417f9211095db2b1e003]) successfully deleted
W20240502 08:17:05.255332 23594 catalog_manager.cc:4810] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): delete failed for tablet 121668ea2e2943c4b61354b8c165000e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 121668ea2e2943c4b61354b8c165000e already in progress: creating tablet
I20240502 08:17:05.255717 24180 tablet_replica.cc:330] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:05.256044 24180 raft_consensus.cc:2229] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.256299 24180 raft_consensus.cc:2258] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.257073 24180 ts_tablet_manager.cc:1875] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:05.257923 25012 ts_tablet_manager.cc:1888] T fddac683eda14ec1a0140926380f2c07 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:05.258253 24174 tablet_bootstrap.cc:492] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:05.258361 24174 ts_tablet_manager.cc:1367] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.020s
I20240502 08:17:05.258443 25012 log.cc:1196] T fddac683eda14ec1a0140926380f2c07 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/fddac683eda14ec1a0140926380f2c07
I20240502 08:17:05.258757 25012 ts_tablet_manager.cc:1909] T fddac683eda14ec1a0140926380f2c07 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:05.259065 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1318) took 1639 ms. Trace:
I20240502 08:17:05.259025 24174 raft_consensus.cc:348] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.259228 24174 raft_consensus.cc:374] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.259279 24174 raft_consensus.cc:729] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.259202 25012 rpcz_store.cc:276] 0502 08:17:03.619771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:03.621782 (+ 2011us) service_pool.cc:225] Handling call
0502 08:17:05.153866 (+1532084us) ts_tablet_manager.cc:1120] Deleting tablet fddac683eda14ec1a0140926380f2c07
0502 08:17:05.153874 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:05.205994 (+ 52120us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:05.257827 (+ 51833us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:05.259049 (+ 1222us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1531966}
I20240502 08:17:05.259475 24174 consensus_queue.cc:260] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.259940 24174 ts_tablet_manager.cc:1398] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:05.259969 25012 tablet_replica.cc:330] T b2eac3bde41c4415bdb97fc53a5b395f P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:05.260195 24174 tablet_bootstrap.cc:492] T 8e2c611d7af6428db289a7e2fbd586d4 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:05.260205 25012 raft_consensus.cc:2229] T b2eac3bde41c4415bdb97fc53a5b395f P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.260447 25012 raft_consensus.cc:2258] T b2eac3bde41c4415bdb97fc53a5b395f P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.261355 24174 tablet_bootstrap.cc:654] T 8e2c611d7af6428db289a7e2fbd586d4 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:05.265184 25012 ts_tablet_manager.cc:1875] T b2eac3bde41c4415bdb97fc53a5b395f P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:05.260082 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet fddac683eda14ec1a0140926380f2c07 (table default.Table_4b967ef2b2fc4e2caa4219afcae841a6 [id=c32fc96889554c32bdf1d4550633e91f]) successfully deleted
W20240502 08:17:05.265802 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7eb01c98aa404982a046162942c74422 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:05.265925 24173 tablet_bootstrap.cc:492] T 255f69f56d344ce7805eca0bfdf8a063 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:05.266034 24173 ts_tablet_manager.cc:1367] T 255f69f56d344ce7805eca0bfdf8a063 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
W20240502 08:17:05.265815 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e28eee4df6d488aa6bf8cab33b8a5a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:05.269937 24173 raft_consensus.cc:348] T 255f69f56d344ce7805eca0bfdf8a063 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.270318 24173 raft_consensus.cc:374] T 255f69f56d344ce7805eca0bfdf8a063 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.270476 24173 raft_consensus.cc:729] T 255f69f56d344ce7805eca0bfdf8a063 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.270752 24173 consensus_queue.cc:260] T 255f69f56d344ce7805eca0bfdf8a063 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.272221 24173 ts_tablet_manager.cc:1398] T 255f69f56d344ce7805eca0bfdf8a063 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:17:05.274351 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8331901741b64742a66ba773a1dd2a2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:05.277315 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_d871e4eda37844d49e6042631c03b1c3" } new_table_name: "default.Table_f6c16db7179f42efbd6b5979bdee926c" modify_external_catalogs: true
W20240502 08:17:05.283408 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb9f8c92b3d34caab0b9f28770ae9c88 consensus state has no leader
I20240502 08:17:05.283686 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_1ba31679300049d4839a333c354950c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:05.250936 24175 consensus_queue.cc:260] T 14075c65ed7a4aa78166943aa32f904f P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.286028 24175 ts_tablet_manager.cc:1398] T 14075c65ed7a4aa78166943aa32f904f P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.037s user 0.001s sys 0.000s
I20240502 08:17:05.288378 24180 ts_tablet_manager.cc:1888] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:05.288564 24180 log.cc:1196] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/14075c65ed7a4aa78166943aa32f904f
I20240502 08:17:05.289767 24181 ts_tablet_manager.cc:1909] T 9f3c00e5393c47df924bcb5dcf2a0f19 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:05.289896 24180 ts_tablet_manager.cc:1909] T 14075c65ed7a4aa78166943aa32f904f P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:05.290226 24175 tablet_bootstrap.cc:492] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:05.291597 24175 tablet_bootstrap.cc:654] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:05.297986 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 14075c65ed7a4aa78166943aa32f904f (table default.Table_40d78b21141944aea5fe3b77b41a7534 [id=3970bee0bcfc4cd6b0ab71113e1cf423]) successfully deleted
I20240502 08:17:05.298811 24180 tablet_replica.cc:330] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:05.299155 24180 raft_consensus.cc:2229] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:05.299568 24180 pending_rounds.cc:62] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2: Trying to abort 1 pending ops.
I20240502 08:17:05.299705 24180 pending_rounds.cc:68] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156531302924288 op_type: NO_OP noop_request { }
I20240502 08:17:05.300263 24180 raft_consensus.cc:2866] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:05.300595 24180 raft_consensus.cc:2258] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.301944 24180 ts_tablet_manager.cc:1875] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:05.303074 24173 tablet_bootstrap.cc:492] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:05.304699 24173 tablet_bootstrap.cc:654] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:05.306387 24181 tablet_replica.cc:330] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:05.306602 24181 raft_consensus.cc:2229] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.306759 24181 raft_consensus.cc:2258] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.307279 24181 ts_tablet_manager.cc:1875] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:05.309733 24174 tablet_bootstrap.cc:492] T 8e2c611d7af6428db289a7e2fbd586d4 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:05.309908 24174 ts_tablet_manager.cc:1367] T 8e2c611d7af6428db289a7e2fbd586d4 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.050s user 0.031s sys 0.006s
I20240502 08:17:05.310536 24174 raft_consensus.cc:348] T 8e2c611d7af6428db289a7e2fbd586d4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.310710 24174 raft_consensus.cc:374] T 8e2c611d7af6428db289a7e2fbd586d4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.310765 24174 raft_consensus.cc:729] T 8e2c611d7af6428db289a7e2fbd586d4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.310946 24174 consensus_queue.cc:260] T 8e2c611d7af6428db289a7e2fbd586d4 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.311065 24175 tablet_bootstrap.cc:492] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:05.311439 24174 ts_tablet_manager.cc:1398] T 8e2c611d7af6428db289a7e2fbd586d4 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 08:17:05.311478 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8331901741b64742a66ba773a1dd2a2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:05.311702 24174 tablet_bootstrap.cc:492] T f3b16f002d064b618760df1e77edaf17 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:05.306440 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 9f3c00e5393c47df924bcb5dcf2a0f19 (table default.Table_a44933d855bd48aa8865c11eaaa5cbe4 [id=7227b8f1e8164cc8aee8ded7c5917ac2]) successfully deleted
I20240502 08:17:05.312842 24174 tablet_bootstrap.cc:654] T f3b16f002d064b618760df1e77edaf17 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:17:05.312995 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ba31679300049d4839a333c354950c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:05.322696 23610 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_8ea1820cc857466bb10df672d16b6184"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:05.323225 23610 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ea1820cc857466bb10df672d16b6184 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:05.327387 23606 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b4905971ad3f4250a6b2e8f12c6ecd96" } new_table_name: "default.Table_880faad269284de2994b95b6ca55f347" modify_external_catalogs: true
I20240502 08:17:05.327946 24173 tablet_bootstrap.cc:492] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:05.328071 24173 ts_tablet_manager.cc:1367] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240502 08:17:05.329371 24173 raft_consensus.cc:348] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.332933 24173 raft_consensus.cc:374] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.335531 24173 raft_consensus.cc:729] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.335813 24173 consensus_queue.cc:260] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.336706 24181 ts_tablet_manager.cc:1888] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:05.337500 24181 log.cc:1196] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/bb4b4a4d6a364de8af3b54d99a0ae138
I20240502 08:17:05.337810 24181 ts_tablet_manager.cc:1909] T bb4b4a4d6a364de8af3b54d99a0ae138 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
W20240502 08:17:05.339166 23606 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d86d534538de4613bd148228a78e5e9a consensus state has no leader
I20240502 08:17:05.339378 23606 master_service.cc:946] ReplaceTablet: received request to replace tablet d5eead3736fb4b34b25850b248d0d62b from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:05.339713 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet bb4b4a4d6a364de8af3b54d99a0ae138 (table default.Table_86918f47dba74a9eb36be5145a017f5b [id=8dbcb2bdd1b14db192a3bd5dcc32fc9a]) successfully deleted
I20240502 08:17:05.339193 24181 tablet_replica.cc:330] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:05.339941 24174 tablet_bootstrap.cc:492] T f3b16f002d064b618760df1e77edaf17 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:05.340058 24174 ts_tablet_manager.cc:1367] T f3b16f002d064b618760df1e77edaf17 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.002s
I20240502 08:17:05.340102 24181 raft_consensus.cc:2229] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.340230 24181 raft_consensus.cc:2258] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.336412 23610 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_465187f6217440139197568c3a016f33" } new_table_name: "default.Table_76aea93e1bab4eecbe9a0835b7579028" modify_external_catalogs: true
I20240502 08:17:05.340693 24181 ts_tablet_manager.cc:1875] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:05.340643 24174 raft_consensus.cc:348] T f3b16f002d064b618760df1e77edaf17 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.340821 24174 raft_consensus.cc:374] T f3b16f002d064b618760df1e77edaf17 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.340880 24174 raft_consensus.cc:729] T f3b16f002d064b618760df1e77edaf17 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.341058 24174 consensus_queue.cc:260] T f3b16f002d064b618760df1e77edaf17 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.341542 24174 ts_tablet_manager.cc:1398] T f3b16f002d064b618760df1e77edaf17 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:05.342787 24173 ts_tablet_manager.cc:1398] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 08:17:05.345870 26001 raft_consensus.cc:482] T 8e2c611d7af6428db289a7e2fbd586d4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.345993 26001 raft_consensus.cc:504] T 8e2c611d7af6428db289a7e2fbd586d4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.346596 26001 leader_election.cc:288] T 8e2c611d7af6428db289a7e2fbd586d4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:05.347455 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2c611d7af6428db289a7e2fbd586d4" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:05.347949 23723 leader_election.cc:341] T 8e2c611d7af6428db289a7e2fbd586d4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.350204 24173 tablet_bootstrap.cc:492] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:05.352922 24181 ts_tablet_manager.cc:1888] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:05.353009 24181 log.cc:1196] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/0281c8123eab4c48bddc377f5695c6e0
I20240502 08:17:05.353261 24181 ts_tablet_manager.cc:1909] T 0281c8123eab4c48bddc377f5695c6e0 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:05.355243 26027 raft_consensus.cc:482] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.355240 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 0281c8123eab4c48bddc377f5695c6e0 (table default.Table_3e0123590a92473dabaacc9c158859e7 [id=d1a317371a5b43f185eb6453861e112a]) successfully deleted
I20240502 08:17:05.355356 26027 raft_consensus.cc:504] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.355851 26027 leader_election.cc:288] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:17:05.357179 23993 leader_election.cc:341] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 76e0f767c6534a608023e6e6d33ca7a8
I20240502 08:17:05.357718 24181 tablet_replica.cc:330] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:05.357797 24174 tablet_bootstrap.cc:492] T b054783710824dbc8f7bc3ea38f605b0 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:05.358143 24181 raft_consensus.cc:2229] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.358500 24181 raft_consensus.cc:2258] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.358947 24174 tablet_bootstrap.cc:654] T b054783710824dbc8f7bc3ea38f605b0 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
W20240502 08:17:05.359476 23610 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2d10ff285c724baeaf766ab2fe3585f3 consensus state has no leader
I20240502 08:17:05.359726 23610 master_service.cc:946] ReplaceTablet: received request to replace tablet 16a1c5276f014410b740c87b9971fb48 from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:05.360323 24173 tablet_bootstrap.cc:654] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:05.362288 24181 ts_tablet_manager.cc:1875] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:05.315747 25012 ts_tablet_manager.cc:1888] T b2eac3bde41c4415bdb97fc53a5b395f P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:05.365708 25012 log.cc:1196] T b2eac3bde41c4415bdb97fc53a5b395f P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/b2eac3bde41c4415bdb97fc53a5b395f
I20240502 08:17:05.366024 25012 ts_tablet_manager.cc:1909] T b2eac3bde41c4415bdb97fc53a5b395f P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:05.366328 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1329) took 1666 ms. Trace:
I20240502 08:17:05.366397 25012 rpcz_store.cc:276] 0502 08:17:03.699656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:03.699765 (+ 109us) service_pool.cc:225] Handling call
0502 08:17:05.259927 (+1560162us) ts_tablet_manager.cc:1120] Deleting tablet b2eac3bde41c4415bdb97fc53a5b395f
0502 08:17:05.259936 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:05.275195 (+ 15259us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:05.315648 (+ 40453us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:05.366308 (+ 50660us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1559975}
I20240502 08:17:05.366703 25012 tablet_replica.cc:330] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:05.366904 25012 raft_consensus.cc:2229] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:05.367151 25012 pending_rounds.cc:62] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b: Trying to abort 1 pending ops.
I20240502 08:17:05.367189 25012 pending_rounds.cc:68] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156528480813056 op_type: NO_OP noop_request { }
I20240502 08:17:05.367278 25012 raft_consensus.cc:2866] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:05.367344 25012 raft_consensus.cc:2258] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.347219 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2c611d7af6428db289a7e2fbd586d4" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:05.356458 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e0f767c6534a608023e6e6d33ca7a8" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:17:05.369027 23996 leader_election.cc:341] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.369112 23996 leader_election.cc:302] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:05.369398 26027 raft_consensus.cc:2726] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:17:05.369892 23723 leader_election.cc:341] T 8e2c611d7af6428db289a7e2fbd586d4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.370020 23723 leader_election.cc:302] T 8e2c611d7af6428db289a7e2fbd586d4 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:05.370429 26001 raft_consensus.cc:2726] T 8e2c611d7af6428db289a7e2fbd586d4 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:05.370781 23597 catalog_manager.cc:4830] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): tablet b2eac3bde41c4415bdb97fc53a5b395f (table default.Table_2d9b44d9d9d34aa184855169eb278e94 [id=f19f154c950e4a228766d5189c345eb6]) successfully deleted
I20240502 08:17:05.371645 25012 ts_tablet_manager.cc:1875] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:05.372033 24173 tablet_bootstrap.cc:492] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:05.372128 24173 ts_tablet_manager.cc:1367] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20240502 08:17:05.372783 24173 raft_consensus.cc:348] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.372936 24173 raft_consensus.cc:374] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.372982 24173 raft_consensus.cc:729] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.373159 24173 consensus_queue.cc:260] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.374776 24173 ts_tablet_manager.cc:1398] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:17:05.375974 24173 tablet_bootstrap.cc:492] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:05.378479 24173 tablet_bootstrap.cc:654] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:05.379499 25012 ts_tablet_manager.cc:1888] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:05.380487 25012 log.cc:1196] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/76e0f767c6534a608023e6e6d33ca7a8
I20240502 08:17:05.381095 25012 ts_tablet_manager.cc:1909] T 76e0f767c6534a608023e6e6d33ca7a8 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:05.382575 24181 ts_tablet_manager.cc:1888] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:05.379716 24175 ts_tablet_manager.cc:1367] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.090s user 0.003s sys 0.000s
I20240502 08:17:05.382668 24181 log.cc:1196] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/d16110e1b1664802a98182c73f1e41d3
I20240502 08:17:05.382938 24181 ts_tablet_manager.cc:1909] T d16110e1b1664802a98182c73f1e41d3 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:05.383443 24175 raft_consensus.cc:348] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.383721 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet d16110e1b1664802a98182c73f1e41d3 (table default.Table_9feed0e1d82c472aaf0e9b92cd9173d5 [id=0f45bbbc044d4e15b40e0a7f9edf6ef2]) successfully deleted
I20240502 08:17:05.383975 24175 raft_consensus.cc:374] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.384761 24175 raft_consensus.cc:729] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.385010 24173 tablet_bootstrap.cc:492] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:05.385212 24175 consensus_queue.cc:260] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.385747 24173 ts_tablet_manager.cc:1367] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20240502 08:17:05.386557 24173 raft_consensus.cc:348] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.387670 24173 raft_consensus.cc:374] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.387856 24173 raft_consensus.cc:729] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.388175 24173 consensus_queue.cc:260] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.387315 24181 tablet_replica.cc:330] T b2eac3bde41c4415bdb97fc53a5b395f P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:05.389699 24181 raft_consensus.cc:2229] T b2eac3bde41c4415bdb97fc53a5b395f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.389866 24181 raft_consensus.cc:2258] T b2eac3bde41c4415bdb97fc53a5b395f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.390334 24181 ts_tablet_manager.cc:1875] T b2eac3bde41c4415bdb97fc53a5b395f P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:05.391319 26027 raft_consensus.cc:482] T 91a3455e5931488da3134d6b5c18aef2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.391746 25649 raft_consensus.cc:482] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.391830 26027 raft_consensus.cc:504] T 91a3455e5931488da3134d6b5c18aef2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.391853 25649 raft_consensus.cc:504] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.392305 25649 leader_election.cc:288] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:05.392306 26027 leader_election.cc:288] T 91a3455e5931488da3134d6b5c18aef2 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:05.393018 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a3455e5931488da3134d6b5c18aef2" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:17:05.393057 23993 leader_election.cc:341] T 91a3455e5931488da3134d6b5c18aef2 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 91a3455e5931488da3134d6b5c18aef2
I20240502 08:17:05.393257 23916 raft_consensus.cc:2379] T 91a3455e5931488da3134d6b5c18aef2 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 149ce744feab49b5aa5bcaa14bd6aba5 in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:17:05.391646 24173 ts_tablet_manager.cc:1398] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:17:05.393678 23996 leader_election.cc:302] T 91a3455e5931488da3134d6b5c18aef2 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:05.394070 26027 raft_consensus.cc:3037] T 91a3455e5931488da3134d6b5c18aef2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.396021 24173 tablet_bootstrap.cc:492] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:05.396253 26027 raft_consensus.cc:2726] T 91a3455e5931488da3134d6b5c18aef2 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:05.397248 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e4cd31e7774546a54704ccd405ffe6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:05.397495 23787 raft_consensus.cc:2454] T 19e4cd31e7774546a54704ccd405ffe6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 0.
I20240502 08:17:05.398651 24173 tablet_bootstrap.cc:654] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:05.399570 23858 leader_election.cc:302] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:17:05.399912 25649 raft_consensus.cc:2781] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:05.399968 25649 raft_consensus.cc:482] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:05.400012 25649 raft_consensus.cc:3037] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.402340 25649 raft_consensus.cc:504] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.402806 25649 leader_election.cc:288] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:05.403359 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e4cd31e7774546a54704ccd405ffe6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"
W20240502 08:17:05.404042 23861 leader_election.cc:341] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.404305 24181 ts_tablet_manager.cc:1888] T b2eac3bde41c4415bdb97fc53a5b395f P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:05.404287 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e4cd31e7774546a54704ccd405ffe6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:05.404392 24181 log.cc:1196] T b2eac3bde41c4415bdb97fc53a5b395f P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/b2eac3bde41c4415bdb97fc53a5b395f
I20240502 08:17:05.404654 24181 ts_tablet_manager.cc:1909] T b2eac3bde41c4415bdb97fc53a5b395f P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
W20240502 08:17:05.404973 23861 leader_election.cc:341] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.405709 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet b2eac3bde41c4415bdb97fc53a5b395f (table default.Table_2d9b44d9d9d34aa184855169eb278e94 [id=f19f154c950e4a228766d5189c345eb6]) successfully deleted
I20240502 08:17:05.406989 23787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e4cd31e7774546a54704ccd405ffe6" candidate_uuid: "5b89a1f0485a44b7ac1710e2369a809b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:05.407192 23787 raft_consensus.cc:3037] T 19e4cd31e7774546a54704ccd405ffe6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.408499 24173 tablet_bootstrap.cc:492] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:05.408596 24173 ts_tablet_manager.cc:1367] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240502 08:17:05.409708 24173 raft_consensus.cc:348] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.410354 24173 raft_consensus.cc:374] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.410548 24173 raft_consensus.cc:729] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.410902 24173 consensus_queue.cc:260] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.411481 26029 raft_consensus.cc:482] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.411577 26029 raft_consensus.cc:504] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.411829 24059 raft_consensus.cc:3037] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.412060 26029 leader_election.cc:288] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:05.413245 24173 ts_tablet_manager.cc:1398] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:17:05.414336 24173 tablet_bootstrap.cc:492] T 6e35efdc0434429aa093fec7770782ef P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:05.414810 26052 consensus_queue.cc:1035] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:05.415323 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d8410ba7cc4b03bb64a138b1f9bb14" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:05.415503 23916 raft_consensus.cc:2379] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 149ce744feab49b5aa5bcaa14bd6aba5 in current term 1: Already voted for candidate a297a6c7c2504ca68e87b24f6691ffb2 in this term.
I20240502 08:17:05.415848 24173 tablet_bootstrap.cc:654] T 6e35efdc0434429aa093fec7770782ef P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
W20240502 08:17:05.416328 23993 leader_election.cc:341] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: b6d8410ba7cc4b03bb64a138b1f9bb14
I20240502 08:17:05.416435 23993 leader_election.cc:302] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
W20240502 08:17:05.417259 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 338fa14b1e004c36accccb3b671103c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 338fa14b1e004c36accccb3b671103c1 already in progress: creating tablet
W20240502 08:17:05.417636 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet ae2b20ae3a014c85af33d5ee51b704e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae2b20ae3a014c85af33d5ee51b704e1 already in progress: creating tablet
I20240502 08:17:05.418370 26029 raft_consensus.cc:3037] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.419381 24181 tablet_replica.cc:330] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
W20240502 08:17:05.419642 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet e34873a3f08043098f2740641b3dc41f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e34873a3f08043098f2740641b3dc41f already in progress: creating tablet
W20240502 08:17:05.419800 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 79a36b64095f445ba21a6304b887a112 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79a36b64095f445ba21a6304b887a112 already in progress: creating tablet
W20240502 08:17:05.419950 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet ed545e81a6934cf8972ff84181048156 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed545e81a6934cf8972ff84181048156 already in progress: creating tablet
W20240502 08:17:05.420086 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 121668ea2e2943c4b61354b8c165000e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 121668ea2e2943c4b61354b8c165000e already in progress: creating tablet
I20240502 08:17:05.421360 23609 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet b97b5ba571754aac8434d0b5ceb37bd7 (table default.Table_dfd4c11d83904abea317a3f01cf3c01a [id=8c2ca203bd9d4b01a1e9cab9546d0676]). Expected version 1 got 0
I20240502 08:17:05.422362 24181 raft_consensus.cc:2229] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.422374 23608 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: default.Table_851f246623304504aa9ddf2e6c5e9082 [id=4fecac7b0d9042b3b4f674f3fc1338ba] alter complete (version 1)
I20240502 08:17:05.422485 24181 raft_consensus.cc:2258] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.422885 24181 ts_tablet_manager.cc:1875] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:05.424733 23758 tablet_service.cc:1467] Processing CreateTablet for tablet fe2b5f10aea547c680e4697fd8ace724 (DEFAULT_TABLE table=default.Table_1fd5707963d04b938961fa6fa33e84c8 [id=d62f98cc6f204cacb5fad618bd784f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.425092 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2b5f10aea547c680e4697fd8ace724. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.425748 23787 raft_consensus.cc:2454] T 19e4cd31e7774546a54704ccd405ffe6 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b89a1f0485a44b7ac1710e2369a809b in term 1.
I20240502 08:17:05.427006 23858 leader_election.cc:302] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 149ce744feab49b5aa5bcaa14bd6aba5
I20240502 08:17:05.428501 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 4ad248497b474ea18b953cc269143c9b (DEFAULT_TABLE table=default.Table_d49f09ff91c349f1b72e94bf914ae862 [id=db9c8c61a69d4078b488b1f6a66b9d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.428804 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ad248497b474ea18b953cc269143c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.431622 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet fba8f6e017e848d59239ae98432b1837 (table default.Table_c7de500cdfa341798dbbd1cc9ab5e4fb [id=3deaa06a1e88491f9dda6abda2cbdd9f]). Expected version 1 got 0
I20240502 08:17:05.431836 23609 catalog_manager.cc:5576] TS 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687) does not have the latest schema for tablet 045b0882a40f4eaba94935d9bc90da0c (table default.Table_7c20ea5847184b48b0bd45fd5751b992 [id=abb72d2b04794e4aab7acee7a3a0925b]). Expected version 3 got 0
I20240502 08:17:05.433727 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 27b51a8324c14790989e4575798e2f7e (DEFAULT_TABLE table=default.Table_efc8673aacce43208f8c36e92f913555 [id=fd87c5ad01844a31a2cede7ae292097b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.433979 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b51a8324c14790989e4575798e2f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.434439 24031 tablet_service.cc:1514] Processing DeleteTablet for tablet 76e0f767c6534a608023e6e6d33ca7a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:05.434741 24023 tablet_service.cc:1467] Processing CreateTablet for tablet fe2b5f10aea547c680e4697fd8ace724 (DEFAULT_TABLE table=default.Table_1fd5707963d04b938961fa6fa33e84c8 [id=d62f98cc6f204cacb5fad618bd784f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.435031 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2b5f10aea547c680e4697fd8ace724. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.435128 24027 tablet_service.cc:1467] Processing CreateTablet for tablet c6e37ce12c2f4533b2fbacbe1a286a9e (DEFAULT_TABLE table=default.Table_986dd87cffc9409da955a904714d5e56 [id=4fa8c67f9770497d92887d9df0789fa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.435297 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 4756b76f86dd4da7bd5d552433d692ab (DEFAULT_TABLE table=default.Table_7a24f13034944055bb412fa1b79c70c0 [id=00cb9a3ff5a148c487af3f4fbab07f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.435429 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e37ce12c2f4533b2fbacbe1a286a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.435539 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4756b76f86dd4da7bd5d552433d692ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.437664 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ee52bcc61b4816acda5b8140b63504 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:41 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:05.437971 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 27b51a8324c14790989e4575798e2f7e (DEFAULT_TABLE table=default.Table_efc8673aacce43208f8c36e92f913555 [id=fd87c5ad01844a31a2cede7ae292097b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.438562 24029 tablet_service.cc:1467] Processing CreateTablet for tablet b718a7299eac4626a63d47d45773a25d (DEFAULT_TABLE table=default.Table_d7579e78c8b34bb199ee967a0fe0afa5 [id=89dff175eede42a49be8ec1686d20b36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.438752 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 41decbb38f7044419843bc5d68009c48 (DEFAULT_TABLE table=default.Table_ec9c41f8b19f478ba8524c3c13311bdb [id=6ecd7ea652eb4f1b8a12d707ab8fe501]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.438985 24024 tablet_service.cc:1467] Processing CreateTablet for tablet eeefe13c5aed49c09a144561626dbbd1 (DEFAULT_TABLE table=default.Table_a783dfa350f04101b7ead760676cb153 [id=14f3ba3f431b40d8905901b03893750e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.439000 23893 tablet_service.cc:1467] Processing CreateTablet for tablet 27b51a8324c14790989e4575798e2f7e (DEFAULT_TABLE table=default.Table_efc8673aacce43208f8c36e92f913555 [id=fd87c5ad01844a31a2cede7ae292097b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.439119 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 5b49f2bf26c24583b49cf2031570d21c (DEFAULT_TABLE table=default.Table_f5dcf3c3174742b0ad367dda4fb666bd [id=375751c3461c4d81beb79c5a7c7269e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.439318 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b51a8324c14790989e4575798e2f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.439383 24022 tablet_service.cc:1467] Processing CreateTablet for tablet 6e7372a4870749c6b702a940f051b305 (DEFAULT_TABLE table=default.Table_35a5601b06fc48a88a18e298ebc76964 [id=7e7be8c827054173922ac723ebd8bba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.439468 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 899a5ca7046e4e59a61b40288ffe9c3d (DEFAULT_TABLE table=default.Table_f59e6ff978214561a7e00a415c70835f [id=4182f29db8cc47b2b183aa397bd55f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.439504 23890 tablet_service.cc:1467] Processing CreateTablet for tablet fe2b5f10aea547c680e4697fd8ace724 (DEFAULT_TABLE table=default.Table_1fd5707963d04b938961fa6fa33e84c8 [id=d62f98cc6f204cacb5fad618bd784f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.439775 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2b5f10aea547c680e4697fd8ace724. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.439741 24040 tablet_service.cc:1467] Processing CreateTablet for tablet a69983eff7f1448cb8abeb3792585f2c (DEFAULT_TABLE table=default.Table_ce8849dfab4243758ff61836ba4a7070 [id=cec91495f3d64f34aaf9439db306c8fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.439810 24030 tablet_service.cc:1467] Processing CreateTablet for tablet a10ee8a9350d463f8b659f7f2e69b5b4 (DEFAULT_TABLE table=default.Table_c30eb6e23bfb485d8071c183499e1ac2 [id=7a4b434dccf448caa7793e047dc16b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.440114 23902 tablet_service.cc:1467] Processing CreateTablet for tablet 4ad248497b474ea18b953cc269143c9b (DEFAULT_TABLE table=default.Table_d49f09ff91c349f1b72e94bf914ae862 [id=db9c8c61a69d4078b488b1f6a66b9d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.440378 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ad248497b474ea18b953cc269143c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.441875 23898 tablet_service.cc:1467] Processing CreateTablet for tablet b718a7299eac4626a63d47d45773a25d (DEFAULT_TABLE table=default.Table_d7579e78c8b34bb199ee967a0fe0afa5 [id=89dff175eede42a49be8ec1686d20b36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.442221 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b718a7299eac4626a63d47d45773a25d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.443766 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 128c37802f0347518003a0fdea3798bb (DEFAULT_TABLE table=default.Table_39508ff6f1ec454dbbb39a729353140f [id=9ef741b58b6f472c82ccdae0e8a76e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.444197 24031 tablet_service.cc:1467] Processing CreateTablet for tablet 4ad248497b474ea18b953cc269143c9b (DEFAULT_TABLE table=default.Table_d49f09ff91c349f1b72e94bf914ae862 [id=db9c8c61a69d4078b488b1f6a66b9d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.446036 23901 tablet_service.cc:1467] Processing CreateTablet for tablet c6e37ce12c2f4533b2fbacbe1a286a9e (DEFAULT_TABLE table=default.Table_986dd87cffc9409da955a904714d5e56 [id=4fa8c67f9770497d92887d9df0789fa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.446430 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e37ce12c2f4533b2fbacbe1a286a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.446765 23760 tablet_service.cc:1467] Processing CreateTablet for tablet b718a7299eac4626a63d47d45773a25d (DEFAULT_TABLE table=default.Table_d7579e78c8b34bb199ee967a0fe0afa5 [id=89dff175eede42a49be8ec1686d20b36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.447105 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b718a7299eac4626a63d47d45773a25d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.449780 23761 tablet_service.cc:1467] Processing CreateTablet for tablet c6e37ce12c2f4533b2fbacbe1a286a9e (DEFAULT_TABLE table=default.Table_986dd87cffc9409da955a904714d5e56 [id=4fa8c67f9770497d92887d9df0789fa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.450110 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e37ce12c2f4533b2fbacbe1a286a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.451117 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a69983eff7f1448cb8abeb3792585f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.451306 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b49f2bf26c24583b49cf2031570d21c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.451459 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b718a7299eac4626a63d47d45773a25d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.451736 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a10ee8a9350d463f8b659f7f2e69b5b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.451869 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b51a8324c14790989e4575798e2f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.452173 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ad248497b474ea18b953cc269143c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.452348 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128c37802f0347518003a0fdea3798bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.453778 23766 tablet_service.cc:1467] Processing CreateTablet for tablet eeefe13c5aed49c09a144561626dbbd1 (DEFAULT_TABLE table=default.Table_a783dfa350f04101b7ead760676cb153 [id=14f3ba3f431b40d8905901b03893750e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.454078 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeefe13c5aed49c09a144561626dbbd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.454018 23903 tablet_service.cc:1467] Processing CreateTablet for tablet 4756b76f86dd4da7bd5d552433d692ab (DEFAULT_TABLE table=default.Table_7a24f13034944055bb412fa1b79c70c0 [id=00cb9a3ff5a148c487af3f4fbab07f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.454445 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 128c37802f0347518003a0fdea3798bb (DEFAULT_TABLE table=default.Table_39508ff6f1ec454dbbb39a729353140f [id=9ef741b58b6f472c82ccdae0e8a76e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.454874 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 41decbb38f7044419843bc5d68009c48 (DEFAULT_TABLE table=default.Table_ec9c41f8b19f478ba8524c3c13311bdb [id=6ecd7ea652eb4f1b8a12d707ab8fe501]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.455317 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 6e7372a4870749c6b702a940f051b305 (DEFAULT_TABLE table=default.Table_35a5601b06fc48a88a18e298ebc76964 [id=7e7be8c827054173922ac723ebd8bba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.455731 23889 tablet_service.cc:1467] Processing CreateTablet for tablet a69983eff7f1448cb8abeb3792585f2c (DEFAULT_TABLE table=default.Table_ce8849dfab4243758ff61836ba4a7070 [id=cec91495f3d64f34aaf9439db306c8fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.456151 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 5b49f2bf26c24583b49cf2031570d21c (DEFAULT_TABLE table=default.Table_f5dcf3c3174742b0ad367dda4fb666bd [id=375751c3461c4d81beb79c5a7c7269e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.456645 23891 tablet_service.cc:1467] Processing CreateTablet for tablet 899a5ca7046e4e59a61b40288ffe9c3d (DEFAULT_TABLE table=default.Table_f59e6ff978214561a7e00a415c70835f [id=4182f29db8cc47b2b183aa397bd55f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.457108 23888 tablet_service.cc:1514] Processing DeleteTablet for tablet 14075c65ed7a4aa78166943aa32f904f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:02 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:05.457191 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41decbb38f7044419843bc5d68009c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.457505 26052 raft_consensus.cc:2781] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:05.457670 26052 raft_consensus.cc:686] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Becoming Leader. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Running, Role: LEADER
I20240502 08:17:05.457693 23750 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e35efdc0434429aa093fec7770782ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:03 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:05.457895 26052 consensus_queue.cc:237] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.457981 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 4756b76f86dd4da7bd5d552433d692ab (DEFAULT_TABLE table=default.Table_7a24f13034944055bb412fa1b79c70c0 [id=00cb9a3ff5a148c487af3f4fbab07f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.458235 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4756b76f86dd4da7bd5d552433d692ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.461906 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899a5ca7046e4e59a61b40288ffe9c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.461841 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 128c37802f0347518003a0fdea3798bb (DEFAULT_TABLE table=default.Table_39508ff6f1ec454dbbb39a729353140f [id=9ef741b58b6f472c82ccdae0e8a76e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.462122 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128c37802f0347518003a0fdea3798bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.462109 23902 tablet_service.cc:1467] Processing CreateTablet for tablet eeefe13c5aed49c09a144561626dbbd1 (DEFAULT_TABLE table=default.Table_a783dfa350f04101b7ead760676cb153 [id=14f3ba3f431b40d8905901b03893750e]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:17:05.465718 23861 consensus_peers.cc:485] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:05.465826 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e7372a4870749c6b702a940f051b305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.465802 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 41decbb38f7044419843bc5d68009c48 (DEFAULT_TABLE table=default.Table_ec9c41f8b19f478ba8524c3c13311bdb [id=6ecd7ea652eb4f1b8a12d707ab8fe501]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.466076 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41decbb38f7044419843bc5d68009c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.466984 26027 raft_consensus.cc:482] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.467118 26027 raft_consensus.cc:504] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.467660 26027 leader_election.cc:288] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:05.468293 25649 consensus_queue.cc:1035] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:05.469156 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeefe13c5aed49c09a144561626dbbd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.469736 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 6e7372a4870749c6b702a940f051b305 (DEFAULT_TABLE table=default.Table_35a5601b06fc48a88a18e298ebc76964 [id=7e7be8c827054173922ac723ebd8bba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.469755 24033 tablet_service.cc:1467] Processing CreateTablet for tablet bd5d5f031609477fbac8c3854bcf3b22 (DEFAULT_TABLE table=default.Table_7e7adde66ddc4951a3af46dd52d0c3b2 [id=85fa9307f66e47d38299ea678e04c98f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.470007 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5d5f031609477fbac8c3854bcf3b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.470007 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e7372a4870749c6b702a940f051b305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.470189 23888 tablet_service.cc:1467] Processing CreateTablet for tablet a10ee8a9350d463f8b659f7f2e69b5b4 (DEFAULT_TABLE table=default.Table_c30eb6e23bfb485d8071c183499e1ac2 [id=7a4b434dccf448caa7793e047dc16b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.470680 23897 tablet_service.cc:1467] Processing CreateTablet for tablet bd5d5f031609477fbac8c3854bcf3b22 (DEFAULT_TABLE table=default.Table_7e7adde66ddc4951a3af46dd52d0c3b2 [id=85fa9307f66e47d38299ea678e04c98f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.473757 23757 tablet_service.cc:1467] Processing CreateTablet for tablet a69983eff7f1448cb8abeb3792585f2c (DEFAULT_TABLE table=default.Table_ce8849dfab4243758ff61836ba4a7070 [id=cec91495f3d64f34aaf9439db306c8fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.474033 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a69983eff7f1448cb8abeb3792585f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.473966 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2085ced52fff46ebaa98324cbb2d8c11" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:17:05.474397 23996 leader_election.cc:341] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.475111 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4756b76f86dd4da7bd5d552433d692ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.477821 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 5b49f2bf26c24583b49cf2031570d21c (DEFAULT_TABLE table=default.Table_f5dcf3c3174742b0ad367dda4fb666bd [id=375751c3461c4d81beb79c5a7c7269e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.477874 24040 tablet_service.cc:1467] Processing CreateTablet for tablet d12cfd9b0abe4997badd0c5f45e2a507 (DEFAULT_TABLE table=default.Table_1fd8c397b2b846f0b1a34f57047b988e [id=fe03743af10949c891296f5727ed7cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.478078 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b49f2bf26c24583b49cf2031570d21c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.478132 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d12cfd9b0abe4997badd0c5f45e2a507. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.478686 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeefe13c5aed49c09a144561626dbbd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.481794 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 899a5ca7046e4e59a61b40288ffe9c3d (DEFAULT_TABLE table=default.Table_f59e6ff978214561a7e00a415c70835f [id=4182f29db8cc47b2b183aa397bd55f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.482116 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899a5ca7046e4e59a61b40288ffe9c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.484170 24181 ts_tablet_manager.cc:1888] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:05.484257 24181 log.cc:1196] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/3a74a6eb6d214d858903b3307e09e9e6
I20240502 08:17:05.484529 24181 ts_tablet_manager.cc:1909] T 3a74a6eb6d214d858903b3307e09e9e6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:05.485415 24181 tablet_replica.cc:330] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:05.485781 23763 tablet_service.cc:1467] Processing CreateTablet for tablet a10ee8a9350d463f8b659f7f2e69b5b4 (DEFAULT_TABLE table=default.Table_c30eb6e23bfb485d8071c183499e1ac2 [id=7a4b434dccf448caa7793e047dc16b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.486079 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a10ee8a9350d463f8b659f7f2e69b5b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.490404 23895 tablet_service.cc:1467] Processing CreateTablet for tablet d12cfd9b0abe4997badd0c5f45e2a507 (DEFAULT_TABLE table=default.Table_1fd8c397b2b846f0b1a34f57047b988e [id=fe03743af10949c891296f5727ed7cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.490672 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d12cfd9b0abe4997badd0c5f45e2a507. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.492334 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128c37802f0347518003a0fdea3798bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.485688 24181 raft_consensus.cc:2229] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:05.495455 24181 raft_consensus.cc:2258] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.495625 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2085ced52fff46ebaa98324cbb2d8c11" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:05.495793 23784 raft_consensus.cc:2454] T 2085ced52fff46ebaa98324cbb2d8c11 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:17:05.495939 24181 ts_tablet_manager.cc:1875] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:05.497426 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5d5f031609477fbac8c3854bcf3b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.497432 23993 leader_election.cc:302] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:05.501791 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a69983eff7f1448cb8abeb3792585f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.505221 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e7372a4870749c6b702a940f051b305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.507720 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 7e45368ff2024c41bf04dd41dd02ca6d (DEFAULT_TABLE table=default.Table_773220dd4c35411ab8824a3f2368ca11 [id=b71eefd1ce0445849962c1a02ddd1ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.508062 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e45368ff2024c41bf04dd41dd02ca6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.508634 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 954e506168514097971698683884da29 (DEFAULT_TABLE table=default.Table_54324a5422aa4797a3b221716afe0cf3 [id=7bd5c55f0b1e4fec8b72eb97103aa3e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.508889 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 954e506168514097971698683884da29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.508967 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b49f2bf26c24583b49cf2031570d21c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.510854 24181 ts_tablet_manager.cc:1888] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:17:05.510938 24181 log.cc:1196] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/4c395a29c7a1485eab39ca107dd91fca
I20240502 08:17:05.511240 24181 ts_tablet_manager.cc:1909] T 4c395a29c7a1485eab39ca107dd91fca P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:05.511672 23764 tablet_service.cc:1467] Processing CreateTablet for tablet f68408a5db454ff3a1ab560925523160 (DEFAULT_TABLE table=default.Table_e46ef84488044afc9e7deb0fb0375871 [id=6eb7691f43814e4cbda0b50d89821d47]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.511900 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41decbb38f7044419843bc5d68009c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.511962 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68408a5db454ff3a1ab560925523160. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.515493 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899a5ca7046e4e59a61b40288ffe9c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.515861 23752 tablet_service.cc:1467] Processing CreateTablet for tablet f6a0a0d6e11e4914a19002d9f3a1c556 (DEFAULT_TABLE table=default.Table_273425ba746d4d198b31df2a796dcc35 [id=1ce4c8597c2f4f2889c9696d2b75bd09]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.516266 23752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a0a0d6e11e4914a19002d9f3a1c556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.517467 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 2ced35c52fc842a9b321c330cd1669ca (DEFAULT_TABLE table=default.Table_67fda5063740421fb45a174acee7c96a [id=844883bc8d424e89bb900906794e0af5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.517788 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ced35c52fc842a9b321c330cd1669ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.518144 24181 tablet_replica.cc:330] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:05.518303 24181 raft_consensus.cc:2229] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.518461 24181 raft_consensus.cc:2258] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.518767 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a10ee8a9350d463f8b659f7f2e69b5b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.518942 24181 ts_tablet_manager.cc:1875] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:05.521175 23750 tablet_service.cc:1467] Processing CreateTablet for tablet bd5d5f031609477fbac8c3854bcf3b22 (DEFAULT_TABLE table=default.Table_7e7adde66ddc4951a3af46dd52d0c3b2 [id=85fa9307f66e47d38299ea678e04c98f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.521441 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5d5f031609477fbac8c3854bcf3b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.524847 23756 tablet_service.cc:1467] Processing CreateTablet for tablet a7bb087d4e3e4faf99cfd36f4d8bdf3f (DEFAULT_TABLE table=default.Table_02e1ffb03a6543299adf3b5b8c6049fe [id=e1b0dc4442a14cd4b64b952afa3c9d6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.525174 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7bb087d4e3e4faf99cfd36f4d8bdf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.420648 26029 raft_consensus.cc:2726] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:05.526185 26029 raft_consensus.cc:2781] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:05.526254 26029 raft_consensus.cc:482] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:05.526329 26029 raft_consensus.cc:3037] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.528650 26029 raft_consensus.cc:504] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.529518 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2085ced52fff46ebaa98324cbb2d8c11" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:05.529709 23784 raft_consensus.cc:3037] T 2085ced52fff46ebaa98324cbb2d8c11 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.531092 24181 ts_tablet_manager.cc:1888] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240502 08:17:05.531189 24181 log.cc:1196] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/63afd6e4fee24edeaba1209105f1c3e6
I20240502 08:17:05.531586 24181 ts_tablet_manager.cc:1909] T 63afd6e4fee24edeaba1209105f1c3e6 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:05.531826 23784 raft_consensus.cc:2454] T 2085ced52fff46ebaa98324cbb2d8c11 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:17:05.532295 23993 leader_election.cc:302] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:17:05.538978 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 77a05bf7ba05410daee8448969dd6633 (DEFAULT_TABLE table=default.Table_7b2a8b4bd21542489d473a8b383f3e4a [id=21ba190413dc4444b5be39ef9c20f956]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.539543 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a05bf7ba05410daee8448969dd6633. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.542094 24181 tablet_replica.cc:330] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:05.542289 24181 raft_consensus.cc:2229] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.542449 24181 raft_consensus.cc:2258] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.542943 24181 ts_tablet_manager.cc:1875] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:05.547374 24051 raft_consensus.cc:3037] T e2792c45255c42adaeefb7aef4e25715 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.551973 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 93c2090a8c1f473385c0a311cdf8457d (DEFAULT_TABLE table=default.Table_6e85449b13c249ed9c64e3a26eec0212 [id=14941cbf871a470da8731e0de4364fb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.552315 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93c2090a8c1f473385c0a311cdf8457d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.553644 25649 consensus_queue.cc:1035] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:05.558676 24181 ts_tablet_manager.cc:1888] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:05.558776 24181 log.cc:1196] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/76e0f767c6534a608023e6e6d33ca7a8
I20240502 08:17:05.559089 24181 ts_tablet_manager.cc:1909] T 76e0f767c6534a608023e6e6d33ca7a8 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:05.560907 26027 raft_consensus.cc:2781] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:05.515861 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4d85b0ee3444b1906e63dae4428a60 (DEFAULT_TABLE table=default.Table_8e43140476184c94b144c88c14afe815 [id=3ad6bb6d4aea4ceb8b4d9fbb5f6d3530]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.562099 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4d85b0ee3444b1906e63dae4428a60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.565263 24181 tablet_replica.cc:330] T 72ee52bcc61b4816acda5b8140b63504 P 149ce744feab49b5aa5bcaa14bd6aba5: stopping tablet replica
I20240502 08:17:05.565495 24181 raft_consensus.cc:2229] T 72ee52bcc61b4816acda5b8140b63504 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.565686 24181 raft_consensus.cc:2258] T 72ee52bcc61b4816acda5b8140b63504 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.565838 23759 tablet_service.cc:1467] Processing CreateTablet for tablet c475d1be48a840bebcd7801d719e8d1f (DEFAULT_TABLE table=default.Table_3859b687d5484bfab843cd42a2f296c9 [id=d64fb2761fb34ccdbba875615555c1b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.566164 24181 ts_tablet_manager.cc:1875] T 72ee52bcc61b4816acda5b8140b63504 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:05.566200 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c475d1be48a840bebcd7801d719e8d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.569828 23753 tablet_service.cc:1467] Processing CreateTablet for tablet d12cfd9b0abe4997badd0c5f45e2a507 (DEFAULT_TABLE table=default.Table_1fd8c397b2b846f0b1a34f57047b988e [id=fe03743af10949c891296f5727ed7cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.570227 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d12cfd9b0abe4997badd0c5f45e2a507. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.572690 26029 leader_election.cc:288] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:05.573206 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2085ced52fff46ebaa98324cbb2d8c11" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
W20240502 08:17:05.573735 23996 leader_election.cc:341] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.583195 24181 ts_tablet_manager.cc:1888] T 72ee52bcc61b4816acda5b8140b63504 P 149ce744feab49b5aa5bcaa14bd6aba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:05.583313 24181 log.cc:1196] T 72ee52bcc61b4816acda5b8140b63504 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-2/wal/wals/72ee52bcc61b4816acda5b8140b63504
I20240502 08:17:05.583609 24181 ts_tablet_manager.cc:1909] T 72ee52bcc61b4816acda5b8140b63504 P 149ce744feab49b5aa5bcaa14bd6aba5: Deleting consensus metadata
I20240502 08:17:05.590171 26027 raft_consensus.cc:686] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:17:05.590487 26027 consensus_queue.cc:237] T 2085ced52fff46ebaa98324cbb2d8c11 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:17:05.590622 23858 consensus_peers.cc:485] T e2792c45255c42adaeefb7aef4e25715 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Couldn't send request to peer a297a6c7c2504ca68e87b24f6691ffb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e2792c45255c42adaeefb7aef4e25715. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:05.527606 23602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 0aababc690904da096423fc910fce2ab
I20240502 08:17:05.593827 23768 tablet_service.cc:1467] Processing CreateTablet for tablet f60d658c5c0c4cb2b205e031a3012059 (DEFAULT_TABLE table=default.Table_c15d8047d6fd438dbdb3f1961b549f4c [id=a5da193c209e46aeb9f2dfbba13fb9c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.528163 23603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 586fc16fa81f40c1b8554f9894be4092
I20240502 08:17:05.594256 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f60d658c5c0c4cb2b205e031a3012059. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.529654 23607 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 353db059d276404ca9f338ebbca021ac
I20240502 08:17:05.529783 23605 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 5edadeb53a354c39a33ff68ffbde5ad1
I20240502 08:17:05.595248 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 586fc16fa81f40c1b8554f9894be4092 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:05 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:05.595263 23751 tablet_service.cc:1514] Processing DeleteTablet for tablet 353db059d276404ca9f338ebbca021ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:05 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:05.530391 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet 2a6881b68c76417bba71c8426abc4768
I20240502 08:17:05.489715 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 3a74a6eb6d214d858903b3307e09e9e6 (table default.Table_c22209bdcbfd4dfa9e643ab9c98dc005 [id=232529911c8d43e2860c52d5527597d1]) successfully deleted
W20240502 08:17:05.596412 23597 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.225.67:44163, user_credentials={real_user=slave}} blocked reactor thread for 106697us
I20240502 08:17:05.596536 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aababc690904da096423fc910fce2ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:05 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:05.545857 23606 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet d5eead3736fb4b34b25850b248d0d62b
I20240502 08:17:05.598073 23606 catalog_manager.cc:6554] ReplaceTablet: tablet d5eead3736fb4b34b25850b248d0d62b deleted and replaced by tablet e41354d638544d40ae5134dd6a0bf868
I20240502 08:17:05.598043 24033 tablet_service.cc:1467] Processing CreateTablet for tablet a7bb087d4e3e4faf99cfd36f4d8bdf3f (DEFAULT_TABLE table=default.Table_02e1ffb03a6543299adf3b5b8c6049fe [id=e1b0dc4442a14cd4b64b952afa3c9d6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.598428 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7bb087d4e3e4faf99cfd36f4d8bdf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.598394 24030 tablet_service.cc:1467] Processing CreateTablet for tablet f6a0a0d6e11e4914a19002d9f3a1c556 (DEFAULT_TABLE table=default.Table_273425ba746d4d198b31df2a796dcc35 [id=1ce4c8597c2f4f2889c9696d2b75bd09]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.598814 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a0a0d6e11e4914a19002d9f3a1c556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.599985 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 77a05bf7ba05410daee8448969dd6633 (DEFAULT_TABLE table=default.Table_7b2a8b4bd21542489d473a8b383f3e4a [id=21ba190413dc4444b5be39ef9c20f956]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.600396 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a05bf7ba05410daee8448969dd6633. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.601330 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 7e45368ff2024c41bf04dd41dd02ca6d (DEFAULT_TABLE table=default.Table_773220dd4c35411ab8824a3f2368ca11 [id=b71eefd1ce0445849962c1a02ddd1ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.601747 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 954e506168514097971698683884da29 (DEFAULT_TABLE table=default.Table_54324a5422aa4797a3b221716afe0cf3 [id=7bd5c55f0b1e4fec8b72eb97103aa3e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.601780 24040 tablet_service.cc:1467] Processing CreateTablet for tablet 2ced35c52fc842a9b321c330cd1669ca (DEFAULT_TABLE table=default.Table_67fda5063740421fb45a174acee7c96a [id=844883bc8d424e89bb900906794e0af5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.602080 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e45368ff2024c41bf04dd41dd02ca6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.602268 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 77a05bf7ba05410daee8448969dd6633 (DEFAULT_TABLE table=default.Table_7b2a8b4bd21542489d473a8b383f3e4a [id=21ba190413dc4444b5be39ef9c20f956]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.602458 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 954e506168514097971698683884da29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.598066 24039 tablet_service.cc:1467] Processing CreateTablet for tablet f68408a5db454ff3a1ab560925523160 (DEFAULT_TABLE table=default.Table_e46ef84488044afc9e7deb0fb0375871 [id=6eb7691f43814e4cbda0b50d89821d47]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.602725 23899 tablet_service.cc:1467] Processing CreateTablet for tablet f6a0a0d6e11e4914a19002d9f3a1c556 (DEFAULT_TABLE table=default.Table_273425ba746d4d198b31df2a796dcc35 [id=1ce4c8597c2f4f2889c9696d2b75bd09]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.602834 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68408a5db454ff3a1ab560925523160. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.603055 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a0a0d6e11e4914a19002d9f3a1c556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.605646 24040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ced35c52fc842a9b321c330cd1669ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.605727 23888 tablet_service.cc:1467] Processing CreateTablet for tablet f68408a5db454ff3a1ab560925523160 (DEFAULT_TABLE table=default.Table_e46ef84488044afc9e7deb0fb0375871 [id=6eb7691f43814e4cbda0b50d89821d47]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.606036 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68408a5db454ff3a1ab560925523160. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.608434 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a05bf7ba05410daee8448969dd6633. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.606923 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 2ced35c52fc842a9b321c330cd1669ca (DEFAULT_TABLE table=default.Table_67fda5063740421fb45a174acee7c96a [id=844883bc8d424e89bb900906794e0af5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.607353 23898 tablet_service.cc:1467] Processing CreateTablet for tablet a7bb087d4e3e4faf99cfd36f4d8bdf3f (DEFAULT_TABLE table=default.Table_02e1ffb03a6543299adf3b5b8c6049fe [id=e1b0dc4442a14cd4b64b952afa3c9d6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.607698 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 954e506168514097971698683884da29 (DEFAULT_TABLE table=default.Table_54324a5422aa4797a3b221716afe0cf3 [id=7bd5c55f0b1e4fec8b72eb97103aa3e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.609675 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4d85b0ee3444b1906e63dae4428a60 (DEFAULT_TABLE table=default.Table_8e43140476184c94b144c88c14afe815 [id=3ad6bb6d4aea4ceb8b4d9fbb5f6d3530]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.612815 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7bb087d4e3e4faf99cfd36f4d8bdf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.613943 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ced35c52fc842a9b321c330cd1669ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.615363 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 954e506168514097971698683884da29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.619262 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4d85b0ee3444b1906e63dae4428a60 (DEFAULT_TABLE table=default.Table_8e43140476184c94b144c88c14afe815 [id=3ad6bb6d4aea4ceb8b4d9fbb5f6d3530]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.608232 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 7e45368ff2024c41bf04dd41dd02ca6d (DEFAULT_TABLE table=default.Table_773220dd4c35411ab8824a3f2368ca11 [id=b71eefd1ce0445849962c1a02ddd1ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.619592 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4d85b0ee3444b1906e63dae4428a60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.619863 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e45368ff2024c41bf04dd41dd02ca6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.624620 23753 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a6881b68c76417bba71c8426abc4768 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:05.624888 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet d5eead3736fb4b34b25850b248d0d62b with delete_type TABLET_DATA_DELETED (replaced by tablet e41354d638544d40ae5134dd6a0bf868) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:05.624987 23892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4d85b0ee3444b1906e63dae4428a60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.627195 23756 tablet_service.cc:1467] Processing CreateTablet for tablet c4a343c44863403faea8acc65262cb34 (DEFAULT_TABLE table=default.Table_375988d4437044bc890604e6d5ab8af2 [id=6c14f79d3913425e9c0a0d0b524c1c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.627583 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4a343c44863403faea8acc65262cb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.630371 23754 tablet_service.cc:1467] Processing CreateTablet for tablet f3ae926889814758813d19a5fd44adf4 (DEFAULT_TABLE table=default.Table_d03c41e82d684920af0c093caa7cd1e6 [id=1a0ddcb26d794ee99068fa11c3b345e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.630683 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3ae926889814758813d19a5fd44adf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.631261 23763 tablet_service.cc:1467] Processing CreateTablet for tablet d2c3323e86ca4505b8d561317c4b83a7 (DEFAULT_TABLE table=default.Table_8b41e4c4860d4f0882c1a60a9404a218 [id=794fb7d878f84a179b11a3b388582f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.631527 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2c3323e86ca4505b8d561317c4b83a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.633899 23753 tablet_service.cc:1467] Processing CreateTablet for tablet ea9a1db2fe564f7995260b8c88a51a6c (DEFAULT_TABLE table=default.Table_e7d36f8ce9ad4afeaf61010b5b2513bb [id=6d37cf4e64944b32977ffbe0a51b62e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.634163 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9a1db2fe564f7995260b8c88a51a6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.593302 23609 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 1be95e0576c143e5a94c7aaf8d3e0625 (table default.Table_2dd52fe14c3a4a18926430f365b64763 [id=c817273ee3b84c21b43137b220ba13b3]). Expected version 3 got 0
I20240502 08:17:05.637847 23751 tablet_service.cc:1467] Processing CreateTablet for tablet ef2b9cd2560a44c9bcdb30fde7bacc04 (DEFAULT_TABLE table=default.Table_e88b8a2ebdba4f77922330a6da96ec51 [id=e3e48555c0904377b1b3de4fc7f9aaed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.638077 23609 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 90d4e2f23c9649a4b1dfb6a179ca7be5 (table default.Table_b2e1c08bfc404c7ebb2720631ad10495 [id=9480d59a76fc401e959a1e41caafc845]). Expected version 3 got 0
I20240502 08:17:05.638236 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef2b9cd2560a44c9bcdb30fde7bacc04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.641206 23763 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e2c611d7af6428db289a7e2fbd586d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:05.641384 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 948402db25e849a4ac2a8e3555e91512 (DEFAULT_TABLE table=default.Table_597cd56369f34933bb8b8d05e8bc6c0f [id=4eb444fc5d814c9586becf6a5ebd37c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.641548 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9bee58d8d94c30bf52d8d2b944fa32 (DEFAULT_TABLE table=default.Table_cffafed96e01457cb67da1cd5033964e [id=554569ab168e4086b693b2e3151d718a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.641661 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948402db25e849a4ac2a8e3555e91512. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.641806 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9bee58d8d94c30bf52d8d2b944fa32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.641206 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 2085ced52fff46ebaa98324cbb2d8c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:51 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:05.645920 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 888fbe7359d54bce87048b277ac7a5b4 (DEFAULT_TABLE table=default.Table_c646218554fc4d108a83a3db92322f1c [id=078840bee2584c39879baee5fd0ed376]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.646198 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888fbe7359d54bce87048b277ac7a5b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.651196 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 93c2090a8c1f473385c0a311cdf8457d (DEFAULT_TABLE table=default.Table_6e85449b13c249ed9c64e3a26eec0212 [id=14941cbf871a470da8731e0de4364fb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.651566 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93c2090a8c1f473385c0a311cdf8457d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.651755 24027 tablet_service.cc:1467] Processing CreateTablet for tablet c475d1be48a840bebcd7801d719e8d1f (DEFAULT_TABLE table=default.Table_3859b687d5484bfab843cd42a2f296c9 [id=d64fb2761fb34ccdbba875615555c1b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.652070 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c475d1be48a840bebcd7801d719e8d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.652992 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 4c395a29c7a1485eab39ca107dd91fca (table default.Table_1159b00d82f4483c8209c879e2d3c06a [id=272acbb9a6ca4f2dbcbfea6aacbe9cfc]) successfully deleted
I20240502 08:17:05.653364 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 63afd6e4fee24edeaba1209105f1c3e6 (table default.Table_9f3ea2fdcdc2477b89d509e868ffe4de [id=84e36915119b4125a1b0d64b2d1647fa]) successfully deleted
I20240502 08:17:05.653493 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 76e0f767c6534a608023e6e6d33ca7a8 (table default.Table_7929569b25404ea08efa4875a3c241a3 [id=51db0a1374374028806594b722a42fcc]) successfully deleted
I20240502 08:17:05.653620 23597 catalog_manager.cc:4830] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): tablet 72ee52bcc61b4816acda5b8140b63504 (table default.Table_d035b24dfb644b7aa77b160ee28d36e5 [id=19ef62dadf8943bdb06f82727634f764]) successfully deleted
I20240502 08:17:05.656605 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a6881b68c76417bba71c8426abc4768 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:05.656795 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 353db059d276404ca9f338ebbca021ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:05 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:05.657359 24030 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9bee58d8d94c30bf52d8d2b944fa32 (DEFAULT_TABLE table=default.Table_cffafed96e01457cb67da1cd5033964e [id=554569ab168e4086b693b2e3151d718a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.657656 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9bee58d8d94c30bf52d8d2b944fa32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.661302 23897 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aababc690904da096423fc910fce2ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:05 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:05.661346 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9bee58d8d94c30bf52d8d2b944fa32 (DEFAULT_TABLE table=default.Table_cffafed96e01457cb67da1cd5033964e [id=554569ab168e4086b693b2e3151d718a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.661489 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 353db059d276404ca9f338ebbca021ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:05 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:05.661656 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aababc690904da096423fc910fce2ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:05 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:05.661746 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet 5edadeb53a354c39a33ff68ffbde5ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:05.661772 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9bee58d8d94c30bf52d8d2b944fa32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.661825 24033 tablet_service.cc:1514] Processing DeleteTablet for tablet 586fc16fa81f40c1b8554f9894be4092 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:05 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:05.661933 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet 586fc16fa81f40c1b8554f9894be4092 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:05 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:05.661976 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet 5edadeb53a354c39a33ff68ffbde5ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:05.662276 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a6881b68c76417bba71c8426abc4768 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:05.663575 24029 tablet_service.cc:1467] Processing CreateTablet for tablet ea9a1db2fe564f7995260b8c88a51a6c (DEFAULT_TABLE table=default.Table_e7d36f8ce9ad4afeaf61010b5b2513bb [id=6d37cf4e64944b32977ffbe0a51b62e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.663839 24034 tablet_service.cc:1467] Processing CreateTablet for tablet f3ae926889814758813d19a5fd44adf4 (DEFAULT_TABLE table=default.Table_d03c41e82d684920af0c093caa7cd1e6 [id=1a0ddcb26d794ee99068fa11c3b345e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.663935 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9a1db2fe564f7995260b8c88a51a6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.664251 24022 tablet_service.cc:1467] Processing CreateTablet for tablet d2c3323e86ca4505b8d561317c4b83a7 (DEFAULT_TABLE table=default.Table_8b41e4c4860d4f0882c1a60a9404a218 [id=794fb7d878f84a179b11a3b388582f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.664530 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2c3323e86ca4505b8d561317c4b83a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.665769 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 888fbe7359d54bce87048b277ac7a5b4 (DEFAULT_TABLE table=default.Table_c646218554fc4d108a83a3db92322f1c [id=078840bee2584c39879baee5fd0ed376]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.666054 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888fbe7359d54bce87048b277ac7a5b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.668891 26143 raft_consensus.cc:482] T 981b7a7292884b1fb48af99db6e762c0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.668998 26143 raft_consensus.cc:504] T 981b7a7292884b1fb48af99db6e762c0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.669181 24032 tablet_service.cc:1467] Processing CreateTablet for tablet ef2b9cd2560a44c9bcdb30fde7bacc04 (DEFAULT_TABLE table=default.Table_e88b8a2ebdba4f77922330a6da96ec51 [id=e3e48555c0904377b1b3de4fc7f9aaed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.669477 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef2b9cd2560a44c9bcdb30fde7bacc04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.669823 24040 tablet_service.cc:1514] Processing DeleteTablet for tablet d5eead3736fb4b34b25850b248d0d62b with delete_type TABLET_DATA_DELETED (replaced by tablet e41354d638544d40ae5134dd6a0bf868) from {username='slave'} at 127.0.0.1:51782
I20240502 08:17:05.670003 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981b7a7292884b1fb48af99db6e762c0" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:05.670148 24033 tablet_service.cc:1467] Processing CreateTablet for tablet f60d658c5c0c4cb2b205e031a3012059 (DEFAULT_TABLE table=default.Table_c15d8047d6fd438dbdb3f1961b549f4c [id=a5da193c209e46aeb9f2dfbba13fb9c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.670442 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f60d658c5c0c4cb2b205e031a3012059. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.670866 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 0a32fa1d6a7f4b6e85f94b5c668a64a9 (DEFAULT_TABLE table=default.Table_86bd849a645b4a2b923b48963b9f51f6 [id=3f4615ddecfd4e38b8386d7664914c87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.671169 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a32fa1d6a7f4b6e85f94b5c668a64a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.671137 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981b7a7292884b1fb48af99db6e762c0" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:05.671559 23996 leader_election.cc:302] T 981b7a7292884b1fb48af99db6e762c0 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:05.673764 24039 tablet_service.cc:1467] Processing CreateTablet for tablet c4a343c44863403faea8acc65262cb34 (DEFAULT_TABLE table=default.Table_375988d4437044bc890604e6d5ab8af2 [id=6c14f79d3913425e9c0a0d0b524c1c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.674019 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4a343c44863403faea8acc65262cb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.674474 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 948402db25e849a4ac2a8e3555e91512 (DEFAULT_TABLE table=default.Table_597cd56369f34933bb8b8d05e8bc6c0f [id=4eb444fc5d814c9586becf6a5ebd37c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.674734 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948402db25e849a4ac2a8e3555e91512. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.676131 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb9cb0164164b43bf08c2a8532b766b (DEFAULT_TABLE table=default.Table_e00515baeaae4ce9ad93000143f2a53a [id=13293a9ab7004e8a965e004f2fb5a14a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.676518 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb9cb0164164b43bf08c2a8532b766b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.676460 23891 tablet_service.cc:1514] Processing DeleteTablet for tablet d5eead3736fb4b34b25850b248d0d62b with delete_type TABLET_DATA_DELETED (replaced by tablet e41354d638544d40ae5134dd6a0bf868) from {username='slave'} at 127.0.0.1:33620
I20240502 08:17:05.677148 23891 tablet_service.cc:1467] Processing CreateTablet for tablet f60d658c5c0c4cb2b205e031a3012059 (DEFAULT_TABLE table=default.Table_c15d8047d6fd438dbdb3f1961b549f4c [id=a5da193c209e46aeb9f2dfbba13fb9c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.677485 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f60d658c5c0c4cb2b205e031a3012059. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.677861 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3ae926889814758813d19a5fd44adf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.683329 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd4d8d8db7441d690878b65edf5d07d (DEFAULT_TABLE table=default.Table_1d4afdf8dc3e4bf39090955f6cca35e3 [id=fdba0b44fcc84c95967895b20014afc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.683802 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd4d8d8db7441d690878b65edf5d07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.689796 23895 tablet_service.cc:1467] Processing CreateTablet for tablet ea9a1db2fe564f7995260b8c88a51a6c (DEFAULT_TABLE table=default.Table_e7d36f8ce9ad4afeaf61010b5b2513bb [id=6d37cf4e64944b32977ffbe0a51b62e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.690522 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9a1db2fe564f7995260b8c88a51a6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.692507 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 948402db25e849a4ac2a8e3555e91512 (DEFAULT_TABLE table=default.Table_597cd56369f34933bb8b8d05e8bc6c0f [id=4eb444fc5d814c9586becf6a5ebd37c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.693040 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948402db25e849a4ac2a8e3555e91512. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.641396 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet 5edadeb53a354c39a33ff68ffbde5ad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:17:04 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:05.699481 23753 tablet_service.cc:1467] Processing CreateTablet for tablet b30e5c761e0040ad8c32a25a0d2906e3 (DEFAULT_TABLE table=default.Table_224b3429692a4ba1a1ac71d94089886b [id=1b4287236fcc4d4abf9d009d14bf8dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.699811 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b30e5c761e0040ad8c32a25a0d2906e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.702572 26143 leader_election.cc:288] T 981b7a7292884b1fb48af99db6e762c0 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:05.702908 26143 raft_consensus.cc:2726] T 981b7a7292884b1fb48af99db6e762c0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:05.703600 26143 raft_consensus.cc:482] T 255f69f56d344ce7805eca0bfdf8a063 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.703774 26143 raft_consensus.cc:504] T 255f69f56d344ce7805eca0bfdf8a063 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.705529 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "255f69f56d344ce7805eca0bfdf8a063" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:17:05.706123 23996 leader_election.cc:341] T 255f69f56d344ce7805eca0bfdf8a063 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.707237 23749 tablet_service.cc:1467] Processing CreateTablet for tablet ddc83cb73f504390abd1c87001ac9154 (DEFAULT_TABLE table=default.Table_9fc71ad3aa2d407a9db02e57623dc854 [id=cd9918bb1e0947ac82049f3739a5ee65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.707872 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc83cb73f504390abd1c87001ac9154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.708748 23764 tablet_service.cc:1467] Processing CreateTablet for tablet c4664d782e3c48a099afbb27dce5c18d (DEFAULT_TABLE table=default.Table_24fea7b9f9a44b11a31e7eb63eafb557 [id=fa6e9e5b517d4135b887790639aaa625]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.709555 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4664d782e3c48a099afbb27dce5c18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.711026 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet a17fb4f2f02e43dc9f7cfdb7c73f7774 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a23c06cb3704a7b83143a02efe2a8b3) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:05.711233 23759 tablet_service.cc:1467] Processing CreateTablet for tablet c6e5e4b9599f4588829c69c097bf2ba3 (DEFAULT_TABLE table=default.Table_6a09f0e331ba4a51bbecca6ddaf0bf51 [id=748a3493656247acb75317ea19ed5795]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.711488 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e5e4b9599f4588829c69c097bf2ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.712175 23895 tablet_service.cc:1467] Processing CreateTablet for tablet d2c3323e86ca4505b8d561317c4b83a7 (DEFAULT_TABLE table=default.Table_8b41e4c4860d4f0882c1a60a9404a218 [id=794fb7d878f84a179b11a3b388582f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.712539 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2c3323e86ca4505b8d561317c4b83a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.720685 24175 ts_tablet_manager.cc:1398] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.338s user 0.001s sys 0.000s
I20240502 08:17:05.721813 24175 tablet_bootstrap.cc:492] T 02e6a1ba23dd4f82bc721a60c0526871 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:05.723646 24175 tablet_bootstrap.cc:654] T 02e6a1ba23dd4f82bc721a60c0526871 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:05.727500 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 98875622ea184b379e2a0acabd01ab65 (DEFAULT_TABLE table=default.Table_f1fdf44a6fa14967a3599402a901e520 [id=0f34bbb405f14c139bc55a93cb284dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.727838 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98875622ea184b379e2a0acabd01ab65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.729498 24180 ts_tablet_manager.cc:1888] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240502 08:17:05.729750 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1331) took 2029 ms. Trace:
I20240502 08:17:05.729830 25012 rpcz_store.cc:276] 0502 08:17:03.700464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:03.702222 (+ 1758us) service_pool.cc:225] Handling call
0502 08:17:05.366656 (+1664434us) ts_tablet_manager.cc:1120] Deleting tablet 76e0f767c6534a608023e6e6d33ca7a8
0502 08:17:05.366665 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:05.375942 (+ 9277us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:05.379396 (+ 3454us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:05.729728 (+350332us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1664266}
I20240502 08:17:05.730337 25012 tablet_replica.cc:330] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:05.730597 25012 raft_consensus.cc:2229] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.730825 25012 raft_consensus.cc:2258] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.731499 25012 ts_tablet_manager.cc:1875] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:05.739114 23902 tablet_service.cc:1467] Processing CreateTablet for tablet c475d1be48a840bebcd7801d719e8d1f (DEFAULT_TABLE table=default.Table_3859b687d5484bfab843cd42a2f296c9 [id=d64fb2761fb34ccdbba875615555c1b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.739771 23902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c475d1be48a840bebcd7801d719e8d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.743408 24052 raft_consensus.cc:3037] T 981b7a7292884b1fb48af99db6e762c0 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.745486 26143 leader_election.cc:288] T 255f69f56d344ce7805eca0bfdf8a063 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:05.747056 25649 consensus_queue.cc:1035] T 981b7a7292884b1fb48af99db6e762c0 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:05.757966 23609 catalog_manager.cc:5484] T 19e4cd31e7774546a54704ccd405ffe6 P 5b89a1f0485a44b7ac1710e2369a809b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66). New cstate: current_term: 1 leader_uuid: "5b89a1f0485a44b7ac1710e2369a809b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:05.774055 23606 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 981b7a7292884b1fb48af99db6e762c0 (table default.Table_2911b4ae0383471eafe8761c4674beed [id=a681d30fb3e5469aad1bc27dfdd56ef7]). Expected version 3 got 0
I20240502 08:17:05.776106 23888 tablet_service.cc:1467] Processing CreateTablet for tablet c4a343c44863403faea8acc65262cb34 (DEFAULT_TABLE table=default.Table_375988d4437044bc890604e6d5ab8af2 [id=6c14f79d3913425e9c0a0d0b524c1c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.776686 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4a343c44863403faea8acc65262cb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.779392 23606 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet 981b7a7292884b1fb48af99db6e762c0 (table default.Table_2911b4ae0383471eafe8761c4674beed [id=a681d30fb3e5469aad1bc27dfdd56ef7]). Expected version 3 got 0
I20240502 08:17:05.786805 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 93c2090a8c1f473385c0a311cdf8457d (DEFAULT_TABLE table=default.Table_6e85449b13c249ed9c64e3a26eec0212 [id=14941cbf871a470da8731e0de4364fb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.787344 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93c2090a8c1f473385c0a311cdf8457d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.796419 23889 tablet_service.cc:1467] Processing CreateTablet for tablet f3ae926889814758813d19a5fd44adf4 (DEFAULT_TABLE table=default.Table_d03c41e82d684920af0c093caa7cd1e6 [id=1a0ddcb26d794ee99068fa11c3b345e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.796996 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3ae926889814758813d19a5fd44adf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.802157 23901 tablet_service.cc:1467] Processing CreateTablet for tablet ef2b9cd2560a44c9bcdb30fde7bacc04 (DEFAULT_TABLE table=default.Table_e88b8a2ebdba4f77922330a6da96ec51 [id=e3e48555c0904377b1b3de4fc7f9aaed]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:17:05.802220 23993 leader_election.cc:341] T 255f69f56d344ce7805eca0bfdf8a063 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 255f69f56d344ce7805eca0bfdf8a063
I20240502 08:17:05.802326 23993 leader_election.cc:302] T 255f69f56d344ce7805eca0bfdf8a063 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:05.803264 26142 raft_consensus.cc:482] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.803390 26142 raft_consensus.cc:504] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.803968 26142 leader_election.cc:288] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:05.804086 26142 raft_consensus.cc:2726] T 255f69f56d344ce7805eca0bfdf8a063 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:05.804710 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a17fb4f2f02e43dc9f7cfdb7c73f7774" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
W20240502 08:17:05.805164 23996 leader_election.cc:341] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.814477 26147 raft_consensus.cc:482] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.814690 26147 raft_consensus.cc:504] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.815742 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae9c817c7be42da8c689267c7eb5a06" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:05.815949 23916 raft_consensus.cc:2454] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:17:05.816452 23996 leader_election.cc:302] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:17:05.816749 26142 raft_consensus.cc:2781] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:05.816820 26142 raft_consensus.cc:482] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:05.816901 26142 raft_consensus.cc:3037] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.819764 26142 raft_consensus.cc:504] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:05.820374 26142 leader_election.cc:288] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:17:05.820904 23993 leader_election.cc:341] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 7ae9c817c7be42da8c689267c7eb5a06
I20240502 08:17:05.821012 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae9c817c7be42da8c689267c7eb5a06" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:05.821157 23916 raft_consensus.cc:3037] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.824015 23916 raft_consensus.cc:2454] T 7ae9c817c7be42da8c689267c7eb5a06 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:17:05.824522 23996 leader_election.cc:302] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:05.825026 26142 raft_consensus.cc:2781] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:05.825192 26142 raft_consensus.cc:686] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:17:05.826395 26142 consensus_queue.cc:237] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
W20240502 08:17:05.833123 23993 leader_election.cc:341] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 7ae9c817c7be42da8c689267c7eb5a06
I20240502 08:17:05.834654 26143 raft_consensus.cc:482] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.834784 26143 raft_consensus.cc:504] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.835475 26143 leader_election.cc:288] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:17:05.836694 23993 leader_election.cc:341] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 5e56af7b42cb481a8363d60f49ee3f61
I20240502 08:17:05.836360 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e56af7b42cb481a8363d60f49ee3f61" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:05.837054 23916 raft_consensus.cc:2454] T 5e56af7b42cb481a8363d60f49ee3f61 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:17:05.837455 23996 leader_election.cc:302] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:05.837662 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb9cb0164164b43bf08c2a8532b766b (DEFAULT_TABLE table=default.Table_e00515baeaae4ce9ad93000143f2a53a [id=13293a9ab7004e8a965e004f2fb5a14a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.837870 26143 raft_consensus.cc:2781] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:05.837935 26143 raft_consensus.cc:482] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:05.837987 26143 raft_consensus.cc:3037] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.838006 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb9cb0164164b43bf08c2a8532b766b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.840154 26143 raft_consensus.cc:504] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.840654 26143 leader_election.cc:288] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:05.841213 23917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e56af7b42cb481a8363d60f49ee3f61" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:05.841349 23917 raft_consensus.cc:3037] T 5e56af7b42cb481a8363d60f49ee3f61 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.843842 23917 raft_consensus.cc:2454] T 5e56af7b42cb481a8363d60f49ee3f61 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
W20240502 08:17:05.844157 23595 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.126:43265, user_credentials={real_user=slave}} blocked reactor thread for 58108.1us
I20240502 08:17:05.844405 23996 leader_election.cc:302] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:17:05.844988 26143 raft_consensus.cc:2781] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:05.845152 26143 raft_consensus.cc:686] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:17:05.845258 23606 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_4846d672262146a1a1ce9debdbbb71b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:05.845423 26143 consensus_queue.cc:237] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:17:05.845844 23606 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4846d672262146a1a1ce9debdbbb71b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:17:05.870012 23993 leader_election.cc:341] T 5e56af7b42cb481a8363d60f49ee3f61 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 5e56af7b42cb481a8363d60f49ee3f61
I20240502 08:17:05.871253 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 0a32fa1d6a7f4b6e85f94b5c668a64a9 (DEFAULT_TABLE table=default.Table_86bd849a645b4a2b923b48963b9f51f6 [id=3f4615ddecfd4e38b8386d7664914c87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.871768 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a32fa1d6a7f4b6e85f94b5c668a64a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.880964 26147 leader_election.cc:288] T 7ae9c817c7be42da8c689267c7eb5a06 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:05.882349 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd4d8d8db7441d690878b65edf5d07d (DEFAULT_TABLE table=default.Table_1d4afdf8dc3e4bf39090955f6cca35e3 [id=fdba0b44fcc84c95967895b20014afc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.882673 24024 tablet_service.cc:1467] Processing CreateTablet for tablet b30e5c761e0040ad8c32a25a0d2906e3 (DEFAULT_TABLE table=default.Table_224b3429692a4ba1a1ac71d94089886b [id=1b4287236fcc4d4abf9d009d14bf8dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.883203 24022 tablet_service.cc:1467] Processing CreateTablet for tablet ddc83cb73f504390abd1c87001ac9154 (DEFAULT_TABLE table=default.Table_9fc71ad3aa2d407a9db02e57623dc854 [id=cd9918bb1e0947ac82049f3739a5ee65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.883555 24034 tablet_service.cc:1467] Processing CreateTablet for tablet c6e5e4b9599f4588829c69c097bf2ba3 (DEFAULT_TABLE table=default.Table_6a09f0e331ba4a51bbecca6ddaf0bf51 [id=748a3493656247acb75317ea19ed5795]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.883597 24030 tablet_service.cc:1467] Processing CreateTablet for tablet c4664d782e3c48a099afbb27dce5c18d (DEFAULT_TABLE table=default.Table_24fea7b9f9a44b11a31e7eb63eafb557 [id=fa6e9e5b517d4135b887790639aaa625]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.882688 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd4d8d8db7441d690878b65edf5d07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.884116 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e5e4b9599f4588829c69c097bf2ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.887125 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4664d782e3c48a099afbb27dce5c18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.889705 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc83cb73f504390abd1c87001ac9154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.737233 23753 tablet_service.cc:1467] Processing CreateTablet for tablet bf00c20573fd4dc19be526acb277363b (DEFAULT_TABLE table=default.Table_6689e6f205644f8091743ad64544240a [id=f6f095bcf58146879a857da5079b2073]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.739111 23768 tablet_service.cc:1467] Processing CreateTablet for tablet e5748e14c816443ca97474c1ecfd546c (DEFAULT_TABLE table=default.Table_ac7aaf3a56b1446482a72525630c9947 [id=62a518bd057b4144b761149488f70de8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.891029 24180 log.cc:1196] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-0/wal/wals/23ba091d8efb44858bf90a104c0f442f
I20240502 08:17:05.891127 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf00c20573fd4dc19be526acb277363b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.891372 24180 ts_tablet_manager.cc:1909] T 23ba091d8efb44858bf90a104c0f442f P a297a6c7c2504ca68e87b24f6691ffb2: Deleting consensus metadata
I20240502 08:17:05.892361 24180 tablet_replica.cc:330] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2: stopping tablet replica
I20240502 08:17:05.892525 24180 raft_consensus.cc:2229] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:05.892493 23594 catalog_manager.cc:4830] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): tablet 23ba091d8efb44858bf90a104c0f442f (table default.Table_7f8b1dce548042b7886ff9f5e150c3f1 [id=39e09ba050be418293957c566ee53c15]) successfully deleted
I20240502 08:17:05.892642 24180 raft_consensus.cc:2258] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:05.893092 24180 ts_tablet_manager.cc:1875] T d4b03d759a354a39aea44fd5d4eeb089 P a297a6c7c2504ca68e87b24f6691ffb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:17:05.893908 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b30e5c761e0040ad8c32a25a0d2906e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.895529 26147 raft_consensus.cc:482] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.895637 26147 raft_consensus.cc:504] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.744740 23749 tablet_service.cc:1467] Processing CreateTablet for tablet a165732a666542d3a5c92ee4e303af77 (DEFAULT_TABLE table=default.Table_ff3ffdf15eba464b8f3d89be9f35a38f [id=cd3d264c45e14e29804a8d3a46daf6af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.746598 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 7bee464bfca6432fa74b03bed0855884 (DEFAULT_TABLE table=default.Table_5805b20a437a4a3784a19f373a194bc6 [id=d1ee3941e5ba411dbc987014863dc123]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.896626 26147 leader_election.cc:288] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:05.896703 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a165732a666542d3a5c92ee4e303af77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.750166 24174 tablet_bootstrap.cc:492] T b054783710824dbc8f7bc3ea38f605b0 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:05.896884 24174 ts_tablet_manager.cc:1367] T b054783710824dbc8f7bc3ea38f605b0 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.539s user 0.009s sys 0.027s
I20240502 08:17:05.897053 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14075c65ed7a4aa78166943aa32f904f" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:05.897260 23916 raft_consensus.cc:2454] T 14075c65ed7a4aa78166943aa32f904f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
W20240502 08:17:05.897370 23993 leader_election.cc:341] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 14075c65ed7a4aa78166943aa32f904f
I20240502 08:17:05.897553 24174 raft_consensus.cc:348] T b054783710824dbc8f7bc3ea38f605b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.897734 24174 raft_consensus.cc:374] T b054783710824dbc8f7bc3ea38f605b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.897775 24174 raft_consensus.cc:729] T b054783710824dbc8f7bc3ea38f605b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.897758 23996 leader_election.cc:302] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:05.897927 24174 consensus_queue.cc:260] T b054783710824dbc8f7bc3ea38f605b0 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.898348 24174 ts_tablet_manager.cc:1398] T b054783710824dbc8f7bc3ea38f605b0 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 08:17:05.898593 24174 tablet_bootstrap.cc:492] T ed428ef1be8a49c19e4b6b48025b9cac P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:05.899711 24174 tablet_bootstrap.cc:654] T ed428ef1be8a49c19e4b6b48025b9cac P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:05.753247 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 2a68d4921605419d81a581a68adbe5b4 (DEFAULT_TABLE table=default.Table_3209c07b69e64b7a9f32f74fa58aef11 [id=dfb9dce29b174be7a541a1e6cb814efc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.900416 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a68d4921605419d81a581a68adbe5b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.903628 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 0a32fa1d6a7f4b6e85f94b5c668a64a9 (DEFAULT_TABLE table=default.Table_86bd849a645b4a2b923b48963b9f51f6 [id=3f4615ddecfd4e38b8386d7664914c87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.904018 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5748e14c816443ca97474c1ecfd546c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.904126 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb9cb0164164b43bf08c2a8532b766b (DEFAULT_TABLE table=default.Table_e00515baeaae4ce9ad93000143f2a53a [id=13293a9ab7004e8a965e004f2fb5a14a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.904613 23892 tablet_service.cc:1467] Processing CreateTablet for tablet 888fbe7359d54bce87048b277ac7a5b4 (DEFAULT_TABLE table=default.Table_c646218554fc4d108a83a3db92322f1c [id=078840bee2584c39879baee5fd0ed376]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.760918 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa944ce9e9d451895d03ebcceb63786 (DEFAULT_TABLE table=default.Table_40d7974e43e447dea4a527c9468becd9 [id=7bb1055636a14adb812803e8334f8dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.905030 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa944ce9e9d451895d03ebcceb63786. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.905771 26143 raft_consensus.cc:2781] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:05.905835 26143 raft_consensus.cc:482] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:05.905882 26143 raft_consensus.cc:3037] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.908242 26143 raft_consensus.cc:504] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.766654 23751 tablet_service.cc:1467] Processing CreateTablet for tablet e8f78713c7d14703a0874bcb664ae655 (DEFAULT_TABLE table=default.Table_1f74b893a9be40b6919d054cc22dbf98 [id=2aa334c9a83c4254893864c0f9f0e3be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:05.908831 26143 leader_election.cc:288] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:05.908890 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8f78713c7d14703a0874bcb664ae655. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.910076 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14075c65ed7a4aa78166943aa32f904f" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:05.910248 23916 raft_consensus.cc:3037] T 14075c65ed7a4aa78166943aa32f904f P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.794682 26011 consensus_queue.cc:1035] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:05.808533 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a17fb4f2f02e43dc9f7cfdb7c73f7774" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:05.910900 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bee464bfca6432fa74b03bed0855884. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:05.912858 23916 raft_consensus.cc:2454] T 14075c65ed7a4aa78166943aa32f904f P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:17:05.913162 26001 consensus_queue.cc:1035] T 6ee764398b364d029ae7f0f575a30d64 P a297a6c7c2504ca68e87b24f6691ffb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:05.845247 26162 raft_consensus.cc:482] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.913611 26162 raft_consensus.cc:504] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.914294 26162 leader_election.cc:288] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:05.914294 23996 leader_election.cc:302] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters:
I20240502 08:17:05.914561 26147 raft_consensus.cc:2781] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:05.914687 26147 raft_consensus.cc:686] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:17:05.914767 23784 raft_consensus.cc:2454] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:17:05.915508 23993 leader_election.cc:302] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters: 5b89a1f0485a44b7ac1710e2369a809b
I20240502 08:17:05.915942 26027 raft_consensus.cc:2781] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:05.916018 26027 raft_consensus.cc:482] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:05.916098 26027 raft_consensus.cc:3037] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.919026 26027 raft_consensus.cc:504] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.920111 26027 leader_election.cc:288] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:05.920464 26027 raft_consensus.cc:482] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.920626 26027 raft_consensus.cc:504] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.921140 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a17fb4f2f02e43dc9f7cfdb7c73f7774" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:05.921336 23784 raft_consensus.cc:3037] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.924319 23784 raft_consensus.cc:2454] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P a297a6c7c2504ca68e87b24f6691ffb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:17:05.924638 24175 tablet_bootstrap.cc:492] T 02e6a1ba23dd4f82bc721a60c0526871 P 5b89a1f0485a44b7ac1710e2369a809b: No bootstrap required, opened a new log
I20240502 08:17:05.924773 24175 ts_tablet_manager.cc:1367] T 02e6a1ba23dd4f82bc721a60c0526871 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent bootstrapping tablet: real 0.204s user 0.001s sys 0.001s
I20240502 08:17:05.928323 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet b054783710824dbc8f7bc3ea38f605b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:56 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:05.929344 24175 raft_consensus.cc:348] T 02e6a1ba23dd4f82bc721a60c0526871 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.929522 24175 raft_consensus.cc:374] T 02e6a1ba23dd4f82bc721a60c0526871 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.929589 24175 raft_consensus.cc:729] T 02e6a1ba23dd4f82bc721a60c0526871 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89a1f0485a44b7ac1710e2369a809b, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.929739 24175 consensus_queue.cc:260] T 02e6a1ba23dd4f82bc721a60c0526871 P 5b89a1f0485a44b7ac1710e2369a809b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.929805 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a17fb4f2f02e43dc9f7cfdb7c73f7774" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:05.930205 24175 ts_tablet_manager.cc:1398] T 02e6a1ba23dd4f82bc721a60c0526871 P 5b89a1f0485a44b7ac1710e2369a809b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:17:05.933676 24175 tablet_bootstrap.cc:492] T 255f69f56d344ce7805eca0bfdf8a063 P 5b89a1f0485a44b7ac1710e2369a809b: Bootstrap starting.
I20240502 08:17:05.935079 24175 tablet_bootstrap.cc:654] T 255f69f56d344ce7805eca0bfdf8a063 P 5b89a1f0485a44b7ac1710e2369a809b: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:05.935547 24174 tablet_bootstrap.cc:492] T ed428ef1be8a49c19e4b6b48025b9cac P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:05.935695 24174 ts_tablet_manager.cc:1367] T ed428ef1be8a49c19e4b6b48025b9cac P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.007s
I20240502 08:17:05.936218 24174 raft_consensus.cc:348] T ed428ef1be8a49c19e4b6b48025b9cac P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.936470 24174 raft_consensus.cc:374] T ed428ef1be8a49c19e4b6b48025b9cac P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.936563 24174 raft_consensus.cc:729] T ed428ef1be8a49c19e4b6b48025b9cac P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.936801 24174 consensus_queue.cc:260] T ed428ef1be8a49c19e4b6b48025b9cac P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.937273 24174 ts_tablet_manager.cc:1398] T ed428ef1be8a49c19e4b6b48025b9cac P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 08:17:05.925339 23993 leader_election.cc:302] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
W20240502 08:17:05.937731 23996 leader_election.cc:341] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:05.938454 26027 leader_election.cc:288] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:05.938647 26027 raft_consensus.cc:2781] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:05.938764 26027 raft_consensus.cc:686] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:17:05.939015 26027 consensus_queue.cc:237] T a17fb4f2f02e43dc9f7cfdb7c73f7774 P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:17:05.942119 23993 leader_election.cc:341] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 14075c65ed7a4aa78166943aa32f904f
W20240502 08:17:05.943835 23993 leader_election.cc:341] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 2ebf55c8024c483f9ef0afe6945a02ff
I20240502 08:17:05.944180 23609 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet ed428ef1be8a49c19e4b6b48025b9cac (table default.Table_4bb4046caa9e462a92c3af8f5f3042d0 [id=43202b0bc3234eec8809f7a88275f464]). Expected version 2 got 0
I20240502 08:17:05.950125 23601 master_service.cc:946] ReplaceTablet: received request to replace tablet f239cbee895542ed9fb5431ef13af0fa from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:05.915045 26147 consensus_queue.cc:237] T 14075c65ed7a4aa78166943aa32f904f P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.951977 24174 tablet_bootstrap.cc:492] T 04e9485d3f3d4b2eaeb5f3dabcce9d58 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:05.956420 23601 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f87df6a45ad94428a9aa6b2796a0fd25: Sending DeleteTablet for 3 replicas of tablet f239cbee895542ed9fb5431ef13af0fa
I20240502 08:17:05.956790 23601 catalog_manager.cc:6554] ReplaceTablet: tablet f239cbee895542ed9fb5431ef13af0fa deleted and replaced by tablet 30559fe2067a4221a1fec980ee73c0fa
I20240502 08:17:05.958235 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet f239cbee895542ed9fb5431ef13af0fa with delete_type TABLET_DATA_DELETED (replaced by tablet 30559fe2067a4221a1fec980ee73c0fa) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:05.958917 24174 tablet_bootstrap.cc:654] T 04e9485d3f3d4b2eaeb5f3dabcce9d58 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:05.959081 23609 catalog_manager.cc:5576] TS a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709) does not have the latest schema for tablet b0cf1a6de3d649b984c50f45393ec0f1 (table default.Table_d416b40fe3d24dafa8db5ee61d13acc0 [id=fa5100e581a7430ab8f0c19b2daae994]). Expected version 1 got 0
I20240502 08:17:05.961494 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_fe3d811fe8df4dbaaa21870da8bc557a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:05.962179 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe3d811fe8df4dbaaa21870da8bc557a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:05.970481 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_79d776656fab45d9aca336ade99acfff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:05.970800 24174 tablet_bootstrap.cc:492] T 04e9485d3f3d4b2eaeb5f3dabcce9d58 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:05.970959 24174 ts_tablet_manager.cc:1367] T 04e9485d3f3d4b2eaeb5f3dabcce9d58 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
W20240502 08:17:05.971129 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79d776656fab45d9aca336ade99acfff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:05.971634 24174 raft_consensus.cc:348] T 04e9485d3f3d4b2eaeb5f3dabcce9d58 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.971845 24174 raft_consensus.cc:374] T 04e9485d3f3d4b2eaeb5f3dabcce9d58 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.971911 24174 raft_consensus.cc:729] T 04e9485d3f3d4b2eaeb5f3dabcce9d58 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.972174 24174 consensus_queue.cc:260] T 04e9485d3f3d4b2eaeb5f3dabcce9d58 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:05.977928 24174 ts_tablet_manager.cc:1398] T 04e9485d3f3d4b2eaeb5f3dabcce9d58 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
W20240502 08:17:05.979514 23594 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet eaea443ae9044b76b48b3b5f92a9ae7e (table default.Table_dd18d96e98ec4c1ca5e651b6d9e5a4dc [id=6fd6e744b27b4a6fbfc1fab435a68c7f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet eaea443ae9044b76b48b3b5f92a9ae7e consensus state has no leader
I20240502 08:17:05.980145 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e9485d3f3d4b2eaeb5f3dabcce9d58 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:48 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:05.988188 23609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_073501c40de24668a6de56fb524cd8f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:05.988361 23996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 58004.7us
I20240502 08:17:05.994385 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebf55c8024c483f9ef0afe6945a02ff" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:05.994591 23916 raft_consensus.cc:2454] T 2ebf55c8024c483f9ef0afe6945a02ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:17:06.003666 24174 tablet_bootstrap.cc:492] T 0a2ff8af162d40cc87d99ec4fe91229d P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:06.005887 24174 tablet_bootstrap.cc:654] T 0a2ff8af162d40cc87d99ec4fe91229d P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:06.008427 26052 consensus_queue.cc:1035] T a91e2b50321e4241847d97a558b419f2 P 5b89a1f0485a44b7ac1710e2369a809b [LEADER]: Connected to new peer: Peer: permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:06.009205 23597 catalog_manager.cc:4810] TS 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): delete failed for tablet 2a6881b68c76417bba71c8426abc4768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a6881b68c76417bba71c8426abc4768 already in progress: creating tablet
I20240502 08:17:06.010131 24027 tablet_service.cc:1467] Processing CreateTablet for tablet bf00c20573fd4dc19be526acb277363b (DEFAULT_TABLE table=default.Table_6689e6f205644f8091743ad64544240a [id=f6f095bcf58146879a857da5079b2073]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.010504 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf00c20573fd4dc19be526acb277363b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.010617 23895 tablet_service.cc:1467] Processing CreateTablet for tablet c4664d782e3c48a099afbb27dce5c18d (DEFAULT_TABLE table=default.Table_24fea7b9f9a44b11a31e7eb63eafb557 [id=fa6e9e5b517d4135b887790639aaa625]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.010825 23897 tablet_service.cc:1467] Processing CreateTablet for tablet bf00c20573fd4dc19be526acb277363b (DEFAULT_TABLE table=default.Table_6689e6f205644f8091743ad64544240a [id=f6f095bcf58146879a857da5079b2073]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.011433 23904 tablet_service.cc:1467] Processing CreateTablet for tablet 98875622ea184b379e2a0acabd01ab65 (DEFAULT_TABLE table=default.Table_f1fdf44a6fa14967a3599402a901e520 [id=0f34bbb405f14c139bc55a93cb284dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.011977 23903 tablet_service.cc:1467] Processing CreateTablet for tablet e5748e14c816443ca97474c1ecfd546c (DEFAULT_TABLE table=default.Table_ac7aaf3a56b1446482a72525630c9947 [id=62a518bd057b4144b761149488f70de8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.012524 23893 tablet_service.cc:1467] Processing CreateTablet for tablet a165732a666542d3a5c92ee4e303af77 (DEFAULT_TABLE table=default.Table_ff3ffdf15eba464b8f3d89be9f35a38f [id=cd3d264c45e14e29804a8d3a46daf6af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.013034 23905 tablet_service.cc:1467] Processing CreateTablet for tablet 7bee464bfca6432fa74b03bed0855884 (DEFAULT_TABLE table=default.Table_5805b20a437a4a3784a19f373a194bc6 [id=d1ee3941e5ba411dbc987014863dc123]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.013672 23887 tablet_service.cc:1467] Processing CreateTablet for tablet c6e5e4b9599f4588829c69c097bf2ba3 (DEFAULT_TABLE table=default.Table_6a09f0e331ba4a51bbecca6ddaf0bf51 [id=748a3493656247acb75317ea19ed5795]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.013772 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 98875622ea184b379e2a0acabd01ab65 (DEFAULT_TABLE table=default.Table_f1fdf44a6fa14967a3599402a901e520 [id=0f34bbb405f14c139bc55a93cb284dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.014057 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98875622ea184b379e2a0acabd01ab65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.014104 23891 tablet_service.cc:1467] Processing CreateTablet for tablet ddc83cb73f504390abd1c87001ac9154 (DEFAULT_TABLE table=default.Table_9fc71ad3aa2d407a9db02e57623dc854 [id=cd9918bb1e0947ac82049f3739a5ee65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.014302 23890 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd4d8d8db7441d690878b65edf5d07d (DEFAULT_TABLE table=default.Table_1d4afdf8dc3e4bf39090955f6cca35e3 [id=fdba0b44fcc84c95967895b20014afc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.014678 23888 tablet_service.cc:1467] Processing CreateTablet for tablet b30e5c761e0040ad8c32a25a0d2906e3 (DEFAULT_TABLE table=default.Table_224b3429692a4ba1a1ac71d94089886b [id=1b4287236fcc4d4abf9d009d14bf8dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.014689 23601 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_907037a5ad7346a195559d77b1d691d1" } new_table_name: "default.Table_7dcbe7c25224443ea7c48d3d937575ef" modify_external_catalogs: true
I20240502 08:17:06.015082 24022 tablet_service.cc:1467] Processing CreateTablet for tablet e5748e14c816443ca97474c1ecfd546c (DEFAULT_TABLE table=default.Table_ac7aaf3a56b1446482a72525630c9947 [id=62a518bd057b4144b761149488f70de8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.015362 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5748e14c816443ca97474c1ecfd546c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.017764 24030 tablet_service.cc:1467] Processing CreateTablet for tablet a165732a666542d3a5c92ee4e303af77 (DEFAULT_TABLE table=default.Table_ff3ffdf15eba464b8f3d89be9f35a38f [id=cd3d264c45e14e29804a8d3a46daf6af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.018045 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a165732a666542d3a5c92ee4e303af77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.018297 24024 tablet_service.cc:1467] Processing CreateTablet for tablet 7bee464bfca6432fa74b03bed0855884 (DEFAULT_TABLE table=default.Table_5805b20a437a4a3784a19f373a194bc6 [id=d1ee3941e5ba411dbc987014863dc123]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.018610 24024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bee464bfca6432fa74b03bed0855884. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:06.022784 23601 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90d44be09f284f3d9042038c92f3b28d consensus state has no leader
I20240502 08:17:06.023532 26029 raft_consensus.cc:482] T e2792c45255c42adaeefb7aef4e25715 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5b89a1f0485a44b7ac1710e2369a809b)
I20240502 08:17:06.023599 24034 tablet_service.cc:1467] Processing CreateTablet for tablet 2a68d4921605419d81a581a68adbe5b4 (DEFAULT_TABLE table=default.Table_3209c07b69e64b7a9f32f74fa58aef11 [id=dfb9dce29b174be7a541a1e6cb814efc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.023639 26029 raft_consensus.cc:504] T e2792c45255c42adaeefb7aef4e25715 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } } peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } }
I20240502 08:17:06.024008 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a68d4921605419d81a581a68adbe5b4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:06.024755 23723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 218603us
I20240502 08:17:06.024844 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa944ce9e9d451895d03ebcceb63786 (DEFAULT_TABLE table=default.Table_40d7974e43e447dea4a527c9468becd9 [id=7bb1055636a14adb812803e8334f8dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.025182 24033 tablet_service.cc:1467] Processing CreateTablet for tablet e8f78713c7d14703a0874bcb664ae655 (DEFAULT_TABLE table=default.Table_1f74b893a9be40b6919d054cc22dbf98 [id=2aa334c9a83c4254893864c0f9f0e3be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.025635 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa944ce9e9d451895d03ebcceb63786. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:06.025753 23609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_073501c40de24668a6de56fb524cd8f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:06.030987 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8f78713c7d14703a0874bcb664ae655. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.033726 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_b13bfe8c518a4f27b9c2006d6f35b5c7" } new_table_name: "default.Table_e45186d178974475a1e24cd40492e74c" modify_external_catalogs: true
W20240502 08:17:06.040664 23605 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f6e5de9033944979c30e3bd4d1e24fd consensus state has no leader
I20240502 08:17:06.041162 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_4cb3cbaa86204a86b93170efb0f0738d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:06.043190 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4cb3cbaa86204a86b93170efb0f0738d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:17:06.052779 24173 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.636s user 0.022s sys 0.082s
W20240502 08:17:06.053330 24173 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.637s user 0.022s sys 0.082s
I20240502 08:17:06.053409 26029 leader_election.cc:288] T e2792c45255c42adaeefb7aef4e25715 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:06.054649 24173 tablet_bootstrap.cc:492] T 6e35efdc0434429aa093fec7770782ef P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:06.054917 24173 ts_tablet_manager.cc:1367] T 6e35efdc0434429aa093fec7770782ef P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.641s user 0.023s sys 0.082s
W20240502 08:17:06.055016 23993 leader_election.cc:341] T e2792c45255c42adaeefb7aef4e25715 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: e2792c45255c42adaeefb7aef4e25715
I20240502 08:17:06.055382 23605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_3f2ff9f792db4a6db22cc74a223e986e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:06.055895 23605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f2ff9f792db4a6db22cc74a223e986e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:06.068487 24174 tablet_bootstrap.cc:492] T 0a2ff8af162d40cc87d99ec4fe91229d P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:06.068645 24174 ts_tablet_manager.cc:1367] T 0a2ff8af162d40cc87d99ec4fe91229d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.004s
I20240502 08:17:06.069792 24174 raft_consensus.cc:348] T 0a2ff8af162d40cc87d99ec4fe91229d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.069994 24174 raft_consensus.cc:374] T 0a2ff8af162d40cc87d99ec4fe91229d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 08:17:06.081686 23996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.225.66:44687, user_credentials={real_user=slave}} blocked reactor thread for 87918.6us
I20240502 08:17:06.081933 24173 raft_consensus.cc:348] T 6e35efdc0434429aa093fec7770782ef P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.070040 24174 raft_consensus.cc:729] T 0a2ff8af162d40cc87d99ec4fe91229d P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:06.082293 24173 raft_consensus.cc:374] T 6e35efdc0434429aa093fec7770782ef P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:06.082401 24173 raft_consensus.cc:729] T 6e35efdc0434429aa093fec7770782ef P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:06.082420 24174 consensus_queue.cc:260] T 0a2ff8af162d40cc87d99ec4fe91229d P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.085734 24174 ts_tablet_manager.cc:1398] T 0a2ff8af162d40cc87d99ec4fe91229d P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240502 08:17:06.086025 24174 tablet_bootstrap.cc:492] T 7d48d1f30fb34ee6a9a68908a3442d43 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:06.087533 24174 tablet_bootstrap.cc:654] T 7d48d1f30fb34ee6a9a68908a3442d43 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:06.082055 23996 leader_election.cc:302] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:06.088433 26143 raft_consensus.cc:2781] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:06.088508 26143 raft_consensus.cc:482] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:06.088569 26143 raft_consensus.cc:3037] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:06.089197 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2792c45255c42adaeefb7aef4e25715" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:06.089733 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a2ff8af162d40cc87d99ec4fe91229d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:16:42 UTC) from {username='slave'} at 127.0.0.1:43544
I20240502 08:17:06.089854 23996 leader_election.cc:302] T e2792c45255c42adaeefb7aef4e25715 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5; no voters: 5b89a1f0485a44b7ac1710e2369a809b, a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:06.090417 23601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_6e169e5548c241aaa47804ab20e12a2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:06.090961 23601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e169e5548c241aaa47804ab20e12a2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:06.090902 26143 raft_consensus.cc:504] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.091456 26143 leader_election.cc:288] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
W20240502 08:17:06.092504 23993 leader_election.cc:341] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 2ebf55c8024c483f9ef0afe6945a02ff
I20240502 08:17:06.092612 26143 raft_consensus.cc:2726] T e2792c45255c42adaeefb7aef4e25715 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:17:06.093283 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ebf55c8024c483f9ef0afe6945a02ff" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:06.093479 23916 raft_consensus.cc:3037] T 2ebf55c8024c483f9ef0afe6945a02ff P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:06.095631 24173 consensus_queue.cc:260] T 6e35efdc0434429aa093fec7770782ef P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.096055 23916 raft_consensus.cc:2454] T 2ebf55c8024c483f9ef0afe6945a02ff P 5b89a1f0485a44b7ac1710e2369a809b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 1.
I20240502 08:17:06.096359 24173 ts_tablet_manager.cc:1398] T 6e35efdc0434429aa093fec7770782ef P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20240502 08:17:06.096545 23996 leader_election.cc:302] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:06.096858 26143 raft_consensus.cc:2781] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:06.096987 26143 raft_consensus.cc:686] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 LEADER]: Becoming Leader. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Running, Role: LEADER
I20240502 08:17:06.097249 26143 consensus_queue.cc:237] T 2ebf55c8024c483f9ef0afe6945a02ff P 149ce744feab49b5aa5bcaa14bd6aba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.101714 24173 tablet_bootstrap.cc:492] T 1be95e0576c143e5a94c7aaf8d3e0625 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:06.103581 24173 tablet_bootstrap.cc:654] T 1be95e0576c143e5a94c7aaf8d3e0625 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:06.108384 23764 tablet_service.cc:1467] Processing CreateTablet for tablet 305536315f9f4dbf9827c2199d7d7a43 (DEFAULT_TABLE table=default.Table_8ea1820cc857466bb10df672d16b6184 [id=ecc918a958a04e54bcd458b2fc88d5dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.108855 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 305536315f9f4dbf9827c2199d7d7a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.112797 24173 tablet_bootstrap.cc:492] T 1be95e0576c143e5a94c7aaf8d3e0625 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:06.112926 24173 ts_tablet_manager.cc:1367] T 1be95e0576c143e5a94c7aaf8d3e0625 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20240502 08:17:06.113623 24173 raft_consensus.cc:348] T 1be95e0576c143e5a94c7aaf8d3e0625 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.113806 24173 raft_consensus.cc:374] T 1be95e0576c143e5a94c7aaf8d3e0625 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:06.113862 24173 raft_consensus.cc:729] T 1be95e0576c143e5a94c7aaf8d3e0625 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:06.118327 24173 consensus_queue.cc:260] T 1be95e0576c143e5a94c7aaf8d3e0625 P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.118793 23607 master_service.cc:946] ReplaceTablet: received request to replace tablet 484c4aeb0f9b4f5aa495c229ad8a9bb6 from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:06.120740 23756 tablet_service.cc:1467] Processing CreateTablet for tablet b950c358ff3b45b2b1c33336bfa1a9a3 (DEFAULT_TABLE table=default.Table_7eb01c98aa404982a046162942c74422 [id=232c71ff8f964d2ca331f05238940306]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.121194 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b950c358ff3b45b2b1c33336bfa1a9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.121834 24173 ts_tablet_manager.cc:1398] T 1be95e0576c143e5a94c7aaf8d3e0625 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 08:17:06.122143 24173 tablet_bootstrap.cc:492] T 23ba091d8efb44858bf90a104c0f442f P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:06.122685 24174 tablet_bootstrap.cc:492] T 7d48d1f30fb34ee6a9a68908a3442d43 P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:06.122820 24174 ts_tablet_manager.cc:1367] T 7d48d1f30fb34ee6a9a68908a3442d43 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.001s
I20240502 08:17:06.123481 24174 raft_consensus.cc:348] T 7d48d1f30fb34ee6a9a68908a3442d43 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.123625 24173 tablet_bootstrap.cc:654] T 23ba091d8efb44858bf90a104c0f442f P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:06.126684 24174 raft_consensus.cc:374] T 7d48d1f30fb34ee6a9a68908a3442d43 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:06.126782 24174 raft_consensus.cc:729] T 7d48d1f30fb34ee6a9a68908a3442d43 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:06.126758 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd08087675b4bd582a79a09189a1099 (DEFAULT_TABLE table=default.Table_e63bcdd98eae4bef904eb59804455320 [id=f5a485627ba84eb79e59f0be95a73f30]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.126935 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 5f28dce603c84ba0a908f964be9b39d3 (DEFAULT_TABLE table=default.Table_71806954494040a98a8f1a388a869a66 [id=0e850566a9154bd58f2a105042142e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.127183 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd08087675b4bd582a79a09189a1099. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.127032 24174 consensus_queue.cc:260] T 7d48d1f30fb34ee6a9a68908a3442d43 P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.127405 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f28dce603c84ba0a908f964be9b39d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.130059 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 06ea1da7983346cba24d13ad7da4c5ee (DEFAULT_TABLE table=default.Table_cb97a20720ec4b2384ff5373d1629576 [id=bc334d15d8ca423e8a1d547fe061040d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.130429 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ea1da7983346cba24d13ad7da4c5ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.139072 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce40f6a3fad4cb2881c58adc2a9e2a9 (DEFAULT_TABLE table=default.Table_79d6fd82fcf8426b8b0eb1a3fe0d00f0 [id=c4ca8a78b63e4c9084696fca7adf0dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.139214 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 94da846295394d1caa759af3f5b158ee (DEFAULT_TABLE table=default.Table_aed335510eae46a89e254b954ca8ad08 [id=d13aa05e234c46a7a1f0246418749657]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.139431 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce40f6a3fad4cb2881c58adc2a9e2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.139639 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94da846295394d1caa759af3f5b158ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.143621 24174 ts_tablet_manager.cc:1398] T 7d48d1f30fb34ee6a9a68908a3442d43 P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20240502 08:17:06.146894 24174 tablet_bootstrap.cc:492] T 57df4bde56a44809ad43d563baeefecb P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:06.148563 24174 tablet_bootstrap.cc:654] T 57df4bde56a44809ad43d563baeefecb P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:06.153792 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 89d54711adf14aadbd2ee65b6fae9eaa (DEFAULT_TABLE table=default.Table_c888f6a56a0f4bda8415ba9c572747e0 [id=b801d3b8308547fc85e6c11b5e9c3c53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.154162 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89d54711adf14aadbd2ee65b6fae9eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.153548 23759 tablet_service.cc:1467] Processing CreateTablet for tablet fbdf2274323a4715ae11bd8a63c04b66 (DEFAULT_TABLE table=default.Table_0632d2e7212e481daaf9b7195c423de8 [id=7707afec47794de59855903f314d0c92]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.159168 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbdf2274323a4715ae11bd8a63c04b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.159387 23754 tablet_service.cc:1467] Processing CreateTablet for tablet 4add03824d0f4c4094da7a6877e18b5b (DEFAULT_TABLE table=default.Table_e73e0921e2154d34aef2a2f3061b8b0a [id=0a6c8a7a685c4283866efd982e7a3c93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.159669 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4add03824d0f4c4094da7a6877e18b5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.160691 23768 tablet_service.cc:1467] Processing CreateTablet for tablet dcfb16dd97d242769a23c474555be685 (DEFAULT_TABLE table=default.Table_edd6a0d33e854c2487fc0df22cda4524 [id=16d9c9f3d0e74e7b85f0bfac6ad06839]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.153548 23749 tablet_service.cc:1467] Processing CreateTablet for tablet ec3991b99cfd4465925887a53955867f (DEFAULT_TABLE table=default.Table_6e21621360404c8aaa08ac5646edcc95 [id=09a1692ac57d459e96352c3037c0d6ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.161140 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec3991b99cfd4465925887a53955867f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.161449 23764 tablet_service.cc:1467] Processing CreateTablet for tablet de332e23b36547c8aa6c912f13dd35bf (DEFAULT_TABLE table=default.Table_3e28eee4df6d488aa6bf8cab33b8a5a6 [id=5b2d42634db74dc0a62ce9bb53335509]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.161895 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcfb16dd97d242769a23c474555be685. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.164626 23605 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_70dac5e6bd8546ec9021c0142d578880" } new_table_name: "default.Table_a0223094db824dce8d841051a31a54c9" modify_external_catalogs: true
I20240502 08:17:06.179348 26143 raft_consensus.cc:482] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:06.179528 26143 raft_consensus.cc:504] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.180294 26143 leader_election.cc:288] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:06.180351 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 079f0dc351f04c60a231eb6759e67eeb (DEFAULT_TABLE table=default.Table_ee3de52e0dcb4bfea556a83fee836104 [id=82c354fa7f8f4564b6897965451ed328]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.180518 23759 tablet_service.cc:1467] Processing CreateTablet for tablet e651270d7e324dabb670eaeaf1fcc8a9 (DEFAULT_TABLE table=default.Table_99cb7bc53f4a44ea93503add4cddf5d0 [id=02d01ca51f1c40eca62c8a0c3bba14ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.180706 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 079f0dc351f04c60a231eb6759e67eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.180930 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e651270d7e324dabb670eaeaf1fcc8a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.183856 23756 tablet_service.cc:1467] Processing CreateTablet for tablet b607406f3dd442cdb99daa3cfdf5e8a3 (DEFAULT_TABLE table=default.Table_699d9f15dcfc42cf9b943d1b3967106c [id=0bfcd47761f24d8a9686b5c1b4ac6590]), partition=RANGE (key) PARTITION UNBOUNDED
W20240502 08:17:06.184155 23993 leader_election.cc:341] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 02e6a1ba23dd4f82bc721a60c0526871
I20240502 08:17:06.184257 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b607406f3dd442cdb99daa3cfdf5e8a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.184911 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e6a1ba23dd4f82bc721a60c0526871" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:06.185142 23916 raft_consensus.cc:2454] T 02e6a1ba23dd4f82bc721a60c0526871 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:17:06.185911 23763 tablet_service.cc:1467] Processing CreateTablet for tablet fa17356f0460460d8aedf6b3fbc7508e (DEFAULT_TABLE table=default.Table_ae752acc232641abb68b65f3530c58d0 [id=6c58868df2f74fb0ac024e6e041a3f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.186205 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa17356f0460460d8aedf6b3fbc7508e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.188223 23753 tablet_service.cc:1467] Processing CreateTablet for tablet 09504b73d50b4e909d5648e5947c1f14 (DEFAULT_TABLE table=default.Table_b92c1042c74e4e16b0298b65923e32d3 [id=b782cff97d77479bb0c1e66636c33492]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.188616 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09504b73d50b4e909d5648e5947c1f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.191651 23751 tablet_service.cc:1467] Processing CreateTablet for tablet c11d99d23c3646dfaec839ea71b38dfc (DEFAULT_TABLE table=default.Table_92101faa794b4c918cf635eb3e135147 [id=d401de28b59e4d918daa6ea433e4f5fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.192010 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11d99d23c3646dfaec839ea71b38dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.193809 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de332e23b36547c8aa6c912f13dd35bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.194478 24174 tablet_bootstrap.cc:492] T 57df4bde56a44809ad43d563baeefecb P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:06.194615 24174 ts_tablet_manager.cc:1367] T 57df4bde56a44809ad43d563baeefecb P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.002s
I20240502 08:17:06.195307 24174 raft_consensus.cc:348] T 57df4bde56a44809ad43d563baeefecb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.195494 24174 raft_consensus.cc:374] T 57df4bde56a44809ad43d563baeefecb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:06.195549 24174 raft_consensus.cc:729] T 57df4bde56a44809ad43d563baeefecb P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:06.195762 24174 consensus_queue.cc:260] T 57df4bde56a44809ad43d563baeefecb P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.195856 23754 tablet_service.cc:1467] Processing CreateTablet for tablet b630c13bc0d9433180bf38e1d0a5b1ba (DEFAULT_TABLE table=default.Table_cc84cf8bfcbb448fb07e0025acdddb22 [id=c3744e1a5bb54f9783ac1e4193f60ba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.196368 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b630c13bc0d9433180bf38e1d0a5b1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.197870 24174 ts_tablet_manager.cc:1398] T 57df4bde56a44809ad43d563baeefecb P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:06.198956 24174 tablet_bootstrap.cc:492] T 3449d7d1e8044b6cbf20766a5597e5ab P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:06.204455 23996 leader_election.cc:302] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, 5b89a1f0485a44b7ac1710e2369a809b; no voters: a297a6c7c2504ca68e87b24f6691ffb2
I20240502 08:17:06.205540 24174 tablet_bootstrap.cc:654] T 3449d7d1e8044b6cbf20766a5597e5ab P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:06.207448 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a17fb4f2f02e43dc9f7cfdb7c73f7774" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:06.207805 26143 raft_consensus.cc:2781] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:06.208348 26143 raft_consensus.cc:482] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:06.208688 26143 raft_consensus.cc:3037] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:06.209854 23757 tablet_service.cc:1467] Processing CreateTablet for tablet e8bc6b11b8be42ad99fd6fb25a8c7b22 (DEFAULT_TABLE table=default.Table_921ba2dcfb7a4c51bc2f12ac85328ef9 [id=65995dd8abfa4beb9cbebb47dea43cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.210197 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8bc6b11b8be42ad99fd6fb25a8c7b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.212172 26143 raft_consensus.cc:504] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:17:06.213467 23993 leader_election.cc:341] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709): Not found: Tablet not found: 02e6a1ba23dd4f82bc721a60c0526871
I20240502 08:17:06.214277 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e6a1ba23dd4f82bc721a60c0526871" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b"
I20240502 08:17:06.214475 23916 raft_consensus.cc:3037] T 02e6a1ba23dd4f82bc721a60c0526871 P 5b89a1f0485a44b7ac1710e2369a809b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:06.216022 26143 leader_election.cc:288] T 02e6a1ba23dd4f82bc721a60c0526871 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:06.216527 23763 tablet_service.cc:1467] Processing CreateTablet for tablet a35b18b970c14ce383026524321089fe (DEFAULT_TABLE table=default.Table_36bbcbb0bb934222b8c149b2da3c7957 [id=f7af85a5dec4486798fc7e7740f75daa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.217053 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35b18b970c14ce383026524321089fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.221673 26157 raft_consensus.cc:482] T f3b16f002d064b618760df1e77edaf17 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:06.221886 26157 raft_consensus.cc:504] T f3b16f002d064b618760df1e77edaf17 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.222682 26157 leader_election.cc:288] T f3b16f002d064b618760df1e77edaf17 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
I20240502 08:17:06.223901 24173 tablet_bootstrap.cc:492] T 23ba091d8efb44858bf90a104c0f442f P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:06.224123 24173 ts_tablet_manager.cc:1367] T 23ba091d8efb44858bf90a104c0f442f P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.102s user 0.002s sys 0.004s
I20240502 08:17:06.224959 24173 raft_consensus.cc:348] T 23ba091d8efb44858bf90a104c0f442f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.225232 24173 raft_consensus.cc:374] T 23ba091d8efb44858bf90a104c0f442f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:06.225351 24173 raft_consensus.cc:729] T 23ba091d8efb44858bf90a104c0f442f P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149ce744feab49b5aa5bcaa14bd6aba5, State: Initialized, Role: FOLLOWER
I20240502 08:17:06.225687 24173 consensus_queue.cc:260] T 23ba091d8efb44858bf90a104c0f442f P 149ce744feab49b5aa5bcaa14bd6aba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.225670 23607 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_016fe5ca17dc4523baadc8fc8ec8a843"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 08:17:06.226442 23607 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_016fe5ca17dc4523baadc8fc8ec8a843 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:06.227164 24173 ts_tablet_manager.cc:1398] T 23ba091d8efb44858bf90a104c0f442f P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:06.227573 24173 tablet_bootstrap.cc:492] T b97b5ba571754aac8434d0b5ceb37bd7 P 149ce744feab49b5aa5bcaa14bd6aba5: Bootstrap starting.
I20240502 08:17:06.229826 24054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b16f002d064b618760df1e77edaf17" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
I20240502 08:17:06.230242 26001 raft_consensus.cc:482] T b054783710824dbc8f7bc3ea38f605b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:06.230379 26001 raft_consensus.cc:504] T b054783710824dbc8f7bc3ea38f605b0 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.230644 23603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_85790535db304ad1a64db38d894171be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:06.230953 26001 leader_election.cc:288] T b054783710824dbc8f7bc3ea38f605b0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687), 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163)
W20240502 08:17:06.231318 23603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85790535db304ad1a64db38d894171be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240502 08:17:06.232699 23861 consensus_peers.cc:485] T 63afd6e4fee24edeaba1209105f1c3e6 P 5b89a1f0485a44b7ac1710e2369a809b -> Peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Couldn't send request to peer 149ce744feab49b5aa5bcaa14bd6aba5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63afd6e4fee24edeaba1209105f1c3e6. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:06.232916 23723 leader_election.cc:341] T f3b16f002d064b618760df1e77edaf17 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:06.233990 23602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36668:
name: "default.Table_7f2080708f3444a79678bcde4b0003a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240502 08:17:06.234318 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b054783710824dbc8f7bc3ea38f605b0" candidate_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" is_pre_election: true
W20240502 08:17:06.234594 23602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f2080708f3444a79678bcde4b0003a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:17:06.236356 23749 tablet_service.cc:1467] Processing CreateTablet for tablet d612b67dbd98445fb992b964d628aa65 (DEFAULT_TABLE table=default.Table_d95b1a6e19ef46958e311e6c87b06bb4 [id=44ff7f9000c4467b881ac6bf75b607b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.236821 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d612b67dbd98445fb992b964d628aa65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.236685 23604 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36668:
table { table_name: "default.Table_3a88ce68a50f4471b706d37631957df0" } new_table_name: "default.Table_3252ab646cd34396a7e27e9af1502050" modify_external_catalogs: true
I20240502 08:17:06.238404 23608 master_service.cc:946] ReplaceTablet: received request to replace tablet 9a9676602d1c4b37b3f4126a29f5456f from {username='slave'} at 127.0.0.1:36668
I20240502 08:17:06.241441 24173 tablet_bootstrap.cc:654] T b97b5ba571754aac8434d0b5ceb37bd7 P 149ce744feab49b5aa5bcaa14bd6aba5: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:06.241919 23757 tablet_service.cc:1467] Processing CreateTablet for tablet 5b44d97e380d4ee1ab353c69424df036 (DEFAULT_TABLE table=default.Table_5dc37698aa0741cc8f5292288f5e4532 [id=cc901cc13f1e4b8dbe350ca4555051de]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.242393 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b44d97e380d4ee1ab353c69424df036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.245896 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf00c20573fd4dc19be526acb277363b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.246112 23888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b30e5c761e0040ad8c32a25a0d2906e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.246515 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bee464bfca6432fa74b03bed0855884. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.246667 23890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd4d8d8db7441d690878b65edf5d07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.246882 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98875622ea184b379e2a0acabd01ab65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.247026 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4664d782e3c48a099afbb27dce5c18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.247511 23903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5748e14c816443ca97474c1ecfd546c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.247696 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a32fa1d6a7f4b6e85f94b5c668a64a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.247984 23893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a165732a666542d3a5c92ee4e303af77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.248163 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef2b9cd2560a44c9bcdb30fde7bacc04. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:17:06.249151 23723 leader_election.cc:341] T b054783710824dbc8f7bc3ea38f605b0 P a297a6c7c2504ca68e87b24f6691ffb2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 149ce744feab49b5aa5bcaa14bd6aba5 (127.22.225.67:44163): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:06.249491 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e5e4b9599f4588829c69c097bf2ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.250054 23891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc83cb73f504390abd1c87001ac9154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.250411 24174 tablet_bootstrap.cc:492] T 3449d7d1e8044b6cbf20766a5597e5ab P a297a6c7c2504ca68e87b24f6691ffb2: No bootstrap required, opened a new log
I20240502 08:17:06.250562 24174 ts_tablet_manager.cc:1367] T 3449d7d1e8044b6cbf20766a5597e5ab P a297a6c7c2504ca68e87b24f6691ffb2: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.000s
I20240502 08:17:06.251197 24174 raft_consensus.cc:348] T 3449d7d1e8044b6cbf20766a5597e5ab P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.251394 24174 raft_consensus.cc:374] T 3449d7d1e8044b6cbf20766a5597e5ab P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:06.251449 24174 raft_consensus.cc:729] T 3449d7d1e8044b6cbf20766a5597e5ab P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a297a6c7c2504ca68e87b24f6691ffb2, State: Initialized, Role: FOLLOWER
I20240502 08:17:06.251682 24174 consensus_queue.cc:260] T 3449d7d1e8044b6cbf20766a5597e5ab P a297a6c7c2504ca68e87b24f6691ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.253702 24174 ts_tablet_manager.cc:1398] T 3449d7d1e8044b6cbf20766a5597e5ab P a297a6c7c2504ca68e87b24f6691ffb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:06.254734 24174 tablet_bootstrap.cc:492] T 29f82ad5860843f1adfbb40cdb21a4a3 P a297a6c7c2504ca68e87b24f6691ffb2: Bootstrap starting.
I20240502 08:17:06.255920 24174 tablet_bootstrap.cc:654] T 29f82ad5860843f1adfbb40cdb21a4a3 P a297a6c7c2504ca68e87b24f6691ffb2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:06.256026 25012 ts_tablet_manager.cc:1888] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240502 08:17:06.256124 25012 log.cc:1196] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting WAL directory at /tmp/dist-test-task2vVFYb/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714637789994300-23429-0/minicluster-data/ts-1/wal/wals/b6d8410ba7cc4b03bb64a138b1f9bb14
I20240502 08:17:06.256453 25012 ts_tablet_manager.cc:1909] T b6d8410ba7cc4b03bb64a138b1f9bb14 P 5b89a1f0485a44b7ac1710e2369a809b: Deleting consensus metadata
I20240502 08:17:06.256774 25012 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33620 (request call id 1332) took 2555 ms. Trace:
I20240502 08:17:06.256850 25012 rpcz_store.cc:276] 0502 08:17:03.700838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:03.702425 (+ 1587us) service_pool.cc:225] Handling call
0502 08:17:05.730286 (+2027861us) ts_tablet_manager.cc:1120] Deleting tablet b6d8410ba7cc4b03bb64a138b1f9bb14
0502 08:17:05.730297 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:17:06.250190 (+519893us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:06.255944 (+ 5754us) tablet_metadata.cc:641] Metadata flushed
0502 08:17:06.256751 (+ 807us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2027743}
I20240502 08:17:06.257248 25012 tablet_replica.cc:330] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b: stopping tablet replica
I20240502 08:17:06.257464 25012 raft_consensus.cc:2229] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:17:06.258028 23763 tablet_service.cc:1467] Processing CreateTablet for tablet 374439069702476c9563675ceae97481 (DEFAULT_TABLE table=default.Table_8331901741b64742a66ba773a1dd2a2b [id=adb3e1eb68ae4240a43a6f4e29deb125]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.258386 23763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 374439069702476c9563675ceae97481. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.258695 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce7f3c381704fc0937bcea120d7b3d7 (DEFAULT_TABLE table=default.Table_5b4382ee4fc14029864a8f4b11589da4 [id=ee7160db9d5f445faa9814e1f39b2315]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.258983 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce7f3c381704fc0937bcea120d7b3d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.264282 24173 tablet_bootstrap.cc:492] T b97b5ba571754aac8434d0b5ceb37bd7 P 149ce744feab49b5aa5bcaa14bd6aba5: No bootstrap required, opened a new log
I20240502 08:17:06.264441 24173 ts_tablet_manager.cc:1367] T b97b5ba571754aac8434d0b5ceb37bd7 P 149ce744feab49b5aa5bcaa14bd6aba5: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20240502 08:17:06.264653 25012 pending_rounds.cc:62] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b: Trying to abort 1 pending ops.
I20240502 08:17:06.264704 25012 pending_rounds.cc:68] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7023156528973623296 op_type: NO_OP noop_request { }
I20240502 08:17:06.264809 25012 raft_consensus.cc:2866] T 3a74a6eb6d214d858903b3307e09e9e6 P 5b89a1f0485a44b7ac1710e2369a809b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:06.265167 26143 raft_consensus.cc:482] T 1be95e0576c143e5a94c7aaf8d3e0625 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:06.265298 26143 raft_consensus.cc:504] T 1be95e0576c143e5a94c7aaf8d3e0625 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.265937 26143 leader_election.cc:288] T 1be95e0576c143e5a94c7aaf8d3e0625 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:06.265879 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 7573ee42671644b49071cfdce8c0d556 (DEFAULT_TABLE table=default.Table_88f82d9d71d2401da70f910b87546fe2 [id=d77b4b5710b34fbca9e7062e580ff801]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:06.266216 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7573ee42671644b49071cfdce8c0d556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.266630 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be95e0576c143e5a94c7aaf8d3e0625" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" is_pre_election: true
I20240502 08:17:06.266804 23784 raft_consensus.cc:2454] T 1be95e0576c143e5a94c7aaf8d3e0625 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 149ce744feab49b5aa5bcaa14bd6aba5 in term 0.
I20240502 08:17:06.267318 23993 leader_election.cc:302] T 1be95e0576c143e5a94c7aaf8d3e0625 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149ce744feab49b5aa5bcaa14bd6aba5, a297a6c7c2504ca68e87b24f6691ffb2; no voters:
I20240502 08:17:06.267587 26143 raft_consensus.cc:2781] T 1be95e0576c143e5a94c7aaf8d3e0625 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:06.267663 26143 raft_consensus.cc:482] T 1be95e0576c143e5a94c7aaf8d3e0625 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:06.267732 26143 raft_consensus.cc:3037] T 1be95e0576c143e5a94c7aaf8d3e0625 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:06.269889 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be95e0576c143e5a94c7aaf8d3e0625" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89a1f0485a44b7ac1710e2369a809b" is_pre_election: true
I20240502 08:17:06.270123 26143 raft_consensus.cc:504] T 1be95e0576c143e5a94c7aaf8d3e0625 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
W20240502 08:17:06.270370 23996 leader_election.cc:341] T 1be95e0576c143e5a94c7aaf8d3e0625 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:06.270920 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be95e0576c143e5a94c7aaf8d3e0625" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a297a6c7c2504ca68e87b24f6691ffb2"
I20240502 08:17:06.271067 23784 raft_consensus.cc:3037] T 1be95e0576c143e5a94c7aaf8d3e0625 P a297a6c7c2504ca68e87b24f6691ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:06.271396 26143 leader_election.cc:288] T 1be95e0576c143e5a94c7aaf8d3e0625 P 149ce744feab49b5aa5bcaa14bd6aba5 [CANDIDATE]: Term 1 election: Requested vote from peers a297a6c7c2504ca68e87b24f6691ffb2 (127.22.225.65:38709), 5b89a1f0485a44b7ac1710e2369a809b (127.22.225.66:44687)
I20240502 08:17:06.271618 24173 raft_consensus.cc:348] T b97b5ba571754aac8434d0b5ceb37bd7 P 149ce744feab49b5aa5bcaa14bd6aba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a297a6c7c2504ca68e87b24f6691ffb2" member_type: VOTER last_known_addr { host: "127.22.225.65" port: 38709 } } peers { permanent_uuid: "5b89a1f0485a44b7ac1710e2369a809b" member_type: VOTER last_known_addr { host: "127.22.225.66" port: 44687 } } peers { permanent_uuid: "149ce744feab49b5aa5bcaa14bd6aba5" member_type: VOTER last_known_addr { host: "127.22.225.67" port: 44163 } }
I20240502 08:17:06.271739 23914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be95e0576c143e5a94c7aaf8d3e0625" candidate_uuid: "149ce744feab49b5aa5bcaa14bd6aba5"